From 7c75b157ac5323ef44bd0eb20c17fe72461b9b2a Mon Sep 17 00:00:00 2001 From: RA489 Date: Tue, 6 Apr 2021 16:52:44 +0530 Subject: [PATCH 001/249] Improvement for other tools --- content/en/docs/reference/tools/_index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/tools/_index.md b/content/en/docs/reference/tools/_index.md index 7194ab83bd..48a9b20f41 100644 --- a/content/en/docs/reference/tools/_index.md +++ b/content/en/docs/reference/tools/_index.md @@ -8,14 +8,14 @@ no_list: true --- -Kubernetes contains several built-in tools to help you work with the Kubernetes system. +Kubernetes contains several built-in as well as third party tools to help you work with the Kubernetes system. ## Minikube -[`minikube`](https://minikube.sigs.k8s.io/docs/) is a tool that +[`minikube`](https://minikube.sigs.k8s.io/docs/) is a built-in tool that runs a single-node Kubernetes cluster locally on your workstation for development and testing purposes. @@ -26,7 +26,7 @@ to a Kubernetes cluster, troubleshoot them, and manage the cluster and its resou ## Helm -[`Kubernetes Helm`](https://github.com/kubernetes/helm) is a tool for managing packages of pre-configured +[`Kubernetes Helm`](https://github.com/kubernetes/helm) is a third party managed tool for managing packages of pre-configured Kubernetes resources, aka Kubernetes charts. Use Helm to: @@ -39,7 +39,7 @@ Use Helm to: ## Kompose -[`Kompose`](https://github.com/kubernetes/kompose) is a tool to help Docker Compose users move to Kubernetes. +[`Kompose`](https://github.com/kubernetes/kompose) is a built-in tool to help Docker Compose users move to Kubernetes. Use Kompose to: From f334e9e23d4002eb43c72f31285c6b17a87b0454 Mon Sep 17 00:00:00 2001 From: Jihoon Seo Date: Tue, 22 Jun 2021 18:08:32 +0900 Subject: [PATCH 002/249] [ru] Remove exec permission on markdown files --- content/ru/docs/concepts/architecture/_index.md | 0 content/ru/docs/concepts/overview/working-with-objects/_index.md | 0 content/ru/docs/reference/glossary/cloud-provider.md | 0 content/ru/docs/reference/glossary/container.md | 0 content/ru/docs/reference/glossary/controller.md | 0 content/ru/docs/reference/glossary/daemonset.md | 0 content/ru/docs/reference/glossary/deployment.md | 0 content/ru/docs/reference/glossary/docker.md | 0 content/ru/docs/reference/glossary/etcd.md | 0 content/ru/docs/reference/glossary/kube-apiserver.md | 0 content/ru/docs/reference/glossary/kube-controller-manager.md | 0 content/ru/docs/reference/glossary/kube-proxy.md | 0 content/ru/docs/reference/glossary/kube-scheduler.md | 0 content/ru/docs/reference/glossary/kubelet.md | 0 content/ru/docs/reference/glossary/label.md | 0 content/ru/docs/reference/glossary/name.md | 0 content/ru/docs/reference/glossary/node.md | 0 content/ru/docs/reference/glossary/pod.md | 0 content/ru/docs/reference/glossary/selector.md | 0 content/ru/docs/reference/glossary/service.md | 0 content/ru/docs/reference/glossary/uid.md | 0 content/ru/docs/reference/kubectl/_index.md | 0 content/ru/docs/tasks/configure-pod-container/_index.md | 0 content/ru/docs/tasks/tools/_index.md | 0 24 files changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 content/ru/docs/concepts/architecture/_index.md mode change 100755 => 100644 content/ru/docs/concepts/overview/working-with-objects/_index.md mode change 100755 => 100644 content/ru/docs/reference/glossary/cloud-provider.md mode change 100755 => 100644 content/ru/docs/reference/glossary/container.md mode change 100755 => 100644 content/ru/docs/reference/glossary/controller.md mode change 100755 => 100644 content/ru/docs/reference/glossary/daemonset.md mode change 100755 => 100644 content/ru/docs/reference/glossary/deployment.md mode change 100755 => 100644 content/ru/docs/reference/glossary/docker.md mode change 100755 => 100644 content/ru/docs/reference/glossary/etcd.md mode change 100755 => 100644 content/ru/docs/reference/glossary/kube-apiserver.md mode change 100755 => 100644 content/ru/docs/reference/glossary/kube-controller-manager.md mode change 100755 => 100644 content/ru/docs/reference/glossary/kube-proxy.md mode change 100755 => 100644 content/ru/docs/reference/glossary/kube-scheduler.md mode change 100755 => 100644 content/ru/docs/reference/glossary/kubelet.md mode change 100755 => 100644 content/ru/docs/reference/glossary/label.md mode change 100755 => 100644 content/ru/docs/reference/glossary/name.md mode change 100755 => 100644 content/ru/docs/reference/glossary/node.md mode change 100755 => 100644 content/ru/docs/reference/glossary/pod.md mode change 100755 => 100644 content/ru/docs/reference/glossary/selector.md mode change 100755 => 100644 content/ru/docs/reference/glossary/service.md mode change 100755 => 100644 content/ru/docs/reference/glossary/uid.md mode change 100755 => 100644 content/ru/docs/reference/kubectl/_index.md mode change 100755 => 100644 content/ru/docs/tasks/configure-pod-container/_index.md mode change 100755 => 100644 content/ru/docs/tasks/tools/_index.md diff --git a/content/ru/docs/concepts/architecture/_index.md b/content/ru/docs/concepts/architecture/_index.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/concepts/overview/working-with-objects/_index.md b/content/ru/docs/concepts/overview/working-with-objects/_index.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/cloud-provider.md b/content/ru/docs/reference/glossary/cloud-provider.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/container.md b/content/ru/docs/reference/glossary/container.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/controller.md b/content/ru/docs/reference/glossary/controller.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/daemonset.md b/content/ru/docs/reference/glossary/daemonset.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/deployment.md b/content/ru/docs/reference/glossary/deployment.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/docker.md b/content/ru/docs/reference/glossary/docker.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/etcd.md b/content/ru/docs/reference/glossary/etcd.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/kube-apiserver.md b/content/ru/docs/reference/glossary/kube-apiserver.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/kube-controller-manager.md b/content/ru/docs/reference/glossary/kube-controller-manager.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/kube-proxy.md b/content/ru/docs/reference/glossary/kube-proxy.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/kube-scheduler.md b/content/ru/docs/reference/glossary/kube-scheduler.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/kubelet.md b/content/ru/docs/reference/glossary/kubelet.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/label.md b/content/ru/docs/reference/glossary/label.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/name.md b/content/ru/docs/reference/glossary/name.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/node.md b/content/ru/docs/reference/glossary/node.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/pod.md b/content/ru/docs/reference/glossary/pod.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/selector.md b/content/ru/docs/reference/glossary/selector.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/service.md b/content/ru/docs/reference/glossary/service.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/glossary/uid.md b/content/ru/docs/reference/glossary/uid.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/reference/kubectl/_index.md b/content/ru/docs/reference/kubectl/_index.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/tasks/configure-pod-container/_index.md b/content/ru/docs/tasks/configure-pod-container/_index.md old mode 100755 new mode 100644 diff --git a/content/ru/docs/tasks/tools/_index.md b/content/ru/docs/tasks/tools/_index.md old mode 100755 new mode 100644 From bee4cb77c2b6c12425794c84de503a8a59f3bf3b Mon Sep 17 00:00:00 2001 From: Jihoon Seo Date: Tue, 22 Jun 2021 18:12:05 +0900 Subject: [PATCH 003/249] [de] Remove exec permission on markdown files --- content/de/docs/concepts/cluster-administration/_index.md | 0 content/de/docs/concepts/configuration/_index.md | 0 content/de/docs/concepts/containers/_index.md | 0 content/de/docs/concepts/overview/_index.md | 0 content/de/docs/concepts/policy/_index.md | 0 content/de/docs/concepts/services-networking/_index.md | 0 content/de/docs/concepts/storage/_index.md | 0 content/de/docs/reference/glossary/etcd.md | 0 content/de/docs/reference/glossary/index.md | 0 content/de/docs/reference/glossary/kube-apiserver.md | 0 content/de/docs/reference/glossary/kube-controller-manager.md | 0 content/de/docs/reference/glossary/kube-scheduler.md | 0 content/de/docs/reference/glossary/kubelet.md | 0 content/de/docs/reference/kubectl/_index.md | 0 content/de/docs/setup/release/_index.md | 0 content/de/docs/tasks/access-application-cluster/_index.md | 0 content/de/docs/tasks/access-kubernetes-api/_index.md | 0 content/de/docs/tasks/administer-cluster/_index.md | 0 content/de/docs/tasks/configure-pod-container/_index.md | 0 content/de/docs/tasks/debug-application-cluster/_index.md | 0 content/de/docs/tasks/federation/_index.md | 0 content/de/docs/tasks/inject-data-application/_index.md | 0 content/de/docs/tasks/manage-daemon/_index.md | 0 content/de/docs/tasks/run-application/_index.md | 0 content/de/docs/tasks/service-catalog/_index.md | 0 content/de/docs/tasks/tls/_index.md | 0 content/de/docs/tasks/tools/_index.md | 0 27 files changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 content/de/docs/concepts/cluster-administration/_index.md mode change 100755 => 100644 content/de/docs/concepts/configuration/_index.md mode change 100755 => 100644 content/de/docs/concepts/containers/_index.md mode change 100755 => 100644 content/de/docs/concepts/overview/_index.md mode change 100755 => 100644 content/de/docs/concepts/policy/_index.md mode change 100755 => 100644 content/de/docs/concepts/services-networking/_index.md mode change 100755 => 100644 content/de/docs/concepts/storage/_index.md mode change 100755 => 100644 content/de/docs/reference/glossary/etcd.md mode change 100755 => 100644 content/de/docs/reference/glossary/index.md mode change 100755 => 100644 content/de/docs/reference/glossary/kube-apiserver.md mode change 100755 => 100644 content/de/docs/reference/glossary/kube-controller-manager.md mode change 100755 => 100644 content/de/docs/reference/glossary/kube-scheduler.md mode change 100755 => 100644 content/de/docs/reference/glossary/kubelet.md mode change 100755 => 100644 content/de/docs/reference/kubectl/_index.md mode change 100755 => 100644 content/de/docs/setup/release/_index.md mode change 100755 => 100644 content/de/docs/tasks/access-application-cluster/_index.md mode change 100755 => 100644 content/de/docs/tasks/access-kubernetes-api/_index.md mode change 100755 => 100644 content/de/docs/tasks/administer-cluster/_index.md mode change 100755 => 100644 content/de/docs/tasks/configure-pod-container/_index.md mode change 100755 => 100644 content/de/docs/tasks/debug-application-cluster/_index.md mode change 100755 => 100644 content/de/docs/tasks/federation/_index.md mode change 100755 => 100644 content/de/docs/tasks/inject-data-application/_index.md mode change 100755 => 100644 content/de/docs/tasks/manage-daemon/_index.md mode change 100755 => 100644 content/de/docs/tasks/run-application/_index.md mode change 100755 => 100644 content/de/docs/tasks/service-catalog/_index.md mode change 100755 => 100644 content/de/docs/tasks/tls/_index.md mode change 100755 => 100644 content/de/docs/tasks/tools/_index.md diff --git a/content/de/docs/concepts/cluster-administration/_index.md b/content/de/docs/concepts/cluster-administration/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/configuration/_index.md b/content/de/docs/concepts/configuration/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/containers/_index.md b/content/de/docs/concepts/containers/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/overview/_index.md b/content/de/docs/concepts/overview/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/policy/_index.md b/content/de/docs/concepts/policy/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/services-networking/_index.md b/content/de/docs/concepts/services-networking/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/concepts/storage/_index.md b/content/de/docs/concepts/storage/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/etcd.md b/content/de/docs/reference/glossary/etcd.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/index.md b/content/de/docs/reference/glossary/index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-apiserver.md b/content/de/docs/reference/glossary/kube-apiserver.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-controller-manager.md b/content/de/docs/reference/glossary/kube-controller-manager.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kube-scheduler.md b/content/de/docs/reference/glossary/kube-scheduler.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/glossary/kubelet.md b/content/de/docs/reference/glossary/kubelet.md old mode 100755 new mode 100644 diff --git a/content/de/docs/reference/kubectl/_index.md b/content/de/docs/reference/kubectl/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/setup/release/_index.md b/content/de/docs/setup/release/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/access-application-cluster/_index.md b/content/de/docs/tasks/access-application-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/access-kubernetes-api/_index.md b/content/de/docs/tasks/access-kubernetes-api/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/administer-cluster/_index.md b/content/de/docs/tasks/administer-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/configure-pod-container/_index.md b/content/de/docs/tasks/configure-pod-container/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/debug-application-cluster/_index.md b/content/de/docs/tasks/debug-application-cluster/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/federation/_index.md b/content/de/docs/tasks/federation/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/inject-data-application/_index.md b/content/de/docs/tasks/inject-data-application/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/manage-daemon/_index.md b/content/de/docs/tasks/manage-daemon/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/run-application/_index.md b/content/de/docs/tasks/run-application/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/service-catalog/_index.md b/content/de/docs/tasks/service-catalog/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/tls/_index.md b/content/de/docs/tasks/tls/_index.md old mode 100755 new mode 100644 diff --git a/content/de/docs/tasks/tools/_index.md b/content/de/docs/tasks/tools/_index.md old mode 100755 new mode 100644 From c1dc2b074f04a5b124de019c82df932820bcc148 Mon Sep 17 00:00:00 2001 From: kartik494 Date: Thu, 1 Jul 2021 13:05:52 +0530 Subject: [PATCH 004/249] Add docker server registry --- content/en/docs/concepts/configuration/secret.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/configuration/secret.md b/content/en/docs/concepts/configuration/secret.md index 48ac53ed47..2fab83d67a 100644 --- a/content/en/docs/concepts/configuration/secret.md +++ b/content/en/docs/concepts/configuration/secret.md @@ -201,7 +201,8 @@ to create a Docker registry Secret, you can do: kubectl create secret docker-registry secret-tiger-docker \ --docker-username=tiger \ --docker-password=pass113 \ - --docker-email=tiger@acme.com + --docker-email=tiger@acme.com \ + --docker-server=my-registry:5000 ``` This command creates a Secret of type `kubernetes.io/dockerconfigjson`. From 6158a7f9260acc508e8829ccf0a42264e8459c9a Mon Sep 17 00:00:00 2001 From: sgpinkus Date: Fri, 9 Jul 2021 00:00:03 +1000 Subject: [PATCH 005/249] Update persistent-volumes.md Makes option 3 for "reclaiming" a released PV a bit clearer. Used to be: > Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition. But the 2nd part applies more to reclamation option 1 and is kind of contradictory with option 3 (which isn't really "reclaiming" anything AFAICT). So just move to it's own stand alone sentence. --- content/en/docs/concepts/storage/persistent-volumes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index f45d17ff54..b41fdcd069 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -131,7 +131,9 @@ The `Retain` reclaim policy allows for manual reclamation of the resource. When 1. Delete the PersistentVolume. The associated storage asset in external infrastructure (such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume) still exists after the PV is deleted. 1. Manually clean up the data on the associated storage asset accordingly. -1. Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the storage asset definition. +1. Manually delete the associated storage asset. + +If you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition. #### Delete From cb712cdcd190a354f6dd055aa2a1ae7a01e3214b Mon Sep 17 00:00:00 2001 From: kartik494 Date: Fri, 2 Jul 2021 10:43:58 +0530 Subject: [PATCH 006/249] Add commit for hostname --- .../en/docs/concepts/configuration/secret.md | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/content/en/docs/concepts/configuration/secret.md b/content/en/docs/concepts/configuration/secret.md index 2fab83d67a..ba12fea864 100644 --- a/content/en/docs/concepts/configuration/secret.md +++ b/content/en/docs/concepts/configuration/secret.md @@ -202,7 +202,7 @@ kubectl create secret docker-registry secret-tiger-docker \ --docker-username=tiger \ --docker-password=pass113 \ --docker-email=tiger@acme.com \ - --docker-server=my-registry:5000 + --docker-server=my-registry.example:5000 ``` This command creates a Secret of type `kubernetes.io/dockerconfigjson`. @@ -212,15 +212,21 @@ on the fly: ```json { - "auths": { - "https://index.docker.io/v1/": { - "username": "tiger", - "password": "pass113", - "email": "tiger@acme.com", - "auth": "dGlnZXI6cGFzczExMw==" - } - } + "apiVersion": "v1", + "data": { + ".dockerconfigjson": "eyJhdXRocyI6eyJteS1yZWdpc3RyeTo1MDAwIjp7InVzZXJuYW1lIjoidGlnZXIiLCJwYXNzd29yZCI6InBhc3MxMTMiLCJlbWFpbCI6InRpZ2VyQGFjbWUuY29tIiwiYXV0aCI6ImRHbG5aWEk2Y0dGemN6RXhNdz09In19fQ==" + }, + "kind": "Secret", + "metadata": { + "creationTimestamp": "2021-07-01T07:30:59Z", + "name": "secret-tiger-docker", + "namespace": "default", + "resourceVersion": "566718", + "uid": "e15c1d7b-9071-4100-8681-f3a7a2ce89ca" + }, + "type": "kubernetes.io/dockerconfigjson" } + ``` ### Basic authentication Secret From 25b6493dd8b0ae6bdd7597c3b116cf18f6c2151c Mon Sep 17 00:00:00 2001 From: Romain Guichard Date: Tue, 20 Jul 2021 18:36:19 +0200 Subject: [PATCH 007/249] fix(fr): typo and trailing spaces --- .../services-networking/endpoint-slices.md | 43 ++++++++----------- 1 file changed, 19 insertions(+), 24 deletions(-) diff --git a/content/fr/docs/concepts/services-networking/endpoint-slices.md b/content/fr/docs/concepts/services-networking/endpoint-slices.md index f5b157bc01..7771b7fe19 100644 --- a/content/fr/docs/concepts/services-networking/endpoint-slices.md +++ b/content/fr/docs/concepts/services-networking/endpoint-slices.md @@ -1,5 +1,5 @@ --- -reviewers: +reviewers: title: EndpointSlices feature: title: EndpointSlices @@ -21,14 +21,14 @@ _EndpointSlices_ offrent une méthode simple pour suivre les Endpoints d'un rés -## Resource pour EndpointSlice {#endpointslice-resource} +## Ressource pour EndpointSlice {#endpointslice-resource} -Dans Kubernetes, un EndpointSlice contient des reférences à un ensemble de Endpoints. -Le controleur d'EndpointSlice crée automatiquement des EndpointSlices pour un Service quand un {{< glossary_tooltip text="sélecteur" term_id="selector" >}} est spécifié. -Ces EnpointSlices vont inclure des références à n'importe quels Pods qui correspondent aux selecteurs de Service. +Dans Kubernetes, un EndpointSlice contient des références à un ensemble de Endpoints. +Le controleur d'EndpointSlice crée automatiquement des EndpointSlices pour un Service quand un {{< glossary_tooltip text="sélecteur" term_id="selector" >}} est spécifié. +Ces EndpointSlices vont inclure des références à n'importe quels Pods qui correspondent aux selecteurs de Service. EndpointSlices groupent ensemble les Endpoints d'un réseau par combinaisons uniques de Services et de Ports. -Par exemple, voici un échantillon d'une resource EndpointSlice pour le Kubernetes Service `exemple`. +Par exemple, voici un échantillon d'une ressource EndpointSlice pour le Kubernetes Service `exemple`. ```yaml apiVersion: discovery.k8s.io/v1beta1 @@ -53,15 +53,15 @@ endpoints: topology.kubernetes.io/zone: us-west2-a ``` -Les EndpointSlices gérés par le contrôleur d'EndpointSlice n'auront, par défaut, pas plus de 100 Endpoints chacun. -En dessous de cette échelle, EndpointSlices devraient mapper 1:1 les Endpoints et les Services et devraientt avoir une performance similaire. +Les EndpointSlices gérés par le contrôleur d'EndpointSlice n'auront, par défaut, pas plus de 100 Endpoints chacun. +En dessous de cette échelle, EndpointSlices devraient mapper 1:1 les Endpoints et les Services et devraient avoir une performance similaire. -EndpointSlices peuvent agir en tant que source de vérité pour kube-proxy quand il s'agit du routage d'un trafic interne. +EndpointSlices peuvent agir en tant que source de vérité pour kube-proxy quand il s'agit du routage d'un trafic interne. Lorsqu'ils sont activés, ils devraient offrir une amélioration de performance pour les services qui ont une grand quantité d'Endpoints. ### Types d'addresses -Les EndpointSlices supportent 3 types d'addresses: +Les EndpointSlices supportent 3 types d'addresses : * IPv4 * IPv6 @@ -69,14 +69,14 @@ Les EndpointSlices supportent 3 types d'addresses: ### Topologie -Chaque Endpoint dans un EnpointSlice peut contenir des informations de topologie pertinentes. +Chaque Endpoint dans un EndpointSlice peut contenir des informations de topologie pertinentes. Ceci est utilisé pour indiquer où se trouve un Endpoint, qui contient les informations sur le Node, zone et région correspondantes. Lorsque les valeurs sont disponibles, les labels de Topologies suivants seront définis par le contrôleur EndpointSlice: * `kubernetes.io/hostname` - Nom du Node sur lequel l'Endpoint se situe. * `topology.kubernetes.io/zone` - Zone dans laquelle l'Endpoint se situe. * `topology.kubernetes.io/region` - Région dans laquelle l'Endpoint se situe. -Le contrôleur EndpointSlice surveille les Services et les Pods pour assurer que leurs correspondances avec les EndpointSlices sont à jour. +Le contrôleur EndpointSlice surveille les Services et les Pods pour assurer que leurs correspondances avec les EndpointSlices sont à jour. Le contrôleur gère les EndpointSlices pour tous les Services qui ont un sélecteur - [référence: {{< glossary_tooltip text="sélecteur" term_id="selector" >}}] - specifié. Celles-ci représenteront les IPs des Pods qui correspondent au sélecteur. ### Capacité d'EndpointSlices @@ -85,32 +85,27 @@ Les EndpointSlices sont limités à une capacité de 100 Endpoints chacun, par d ### Distribution d'EndpointSlices -Chaque EndpointSlice a un ensemble de ports qui s'applique à tous les Endpoints dans la resource. -Lorsque les ports nommés sont utilisés pour un Service, les Pods peuvent se retrouver avec différents ports cibles pour le même port nommé, nécessitant différents EndpointSlices. +Chaque EndpointSlice a un ensemble de ports qui s'applique à tous les Endpoints dans la ressource. +Lorsque les ports nommés sont utilisés pour un Service, les Pods peuvent se retrouver avec différents ports cibles pour le même port nommé, nécessitant différents EndpointSlices. Le contrôleur essaie de remplir les EndpointSlices aussi complètement que possible, mais ne les rééquilibre pas activement. La logique du contrôleur est assez simple: -1. Itérer à travers les EnpointSlices existants, retirer les Endpoints qui ne sont plus voulus et mettre à jour les Endpoints qui ont changé. +1. Itérer à travers les EndpointSlices existants, retirer les Endpoints qui ne sont plus voulus et mettre à jour les Endpoints qui ont changé. 2. Itérer à travers les EndpointSlices qui ont été modifiés dans la première étape et les remplir avec n'importe quel Endpoint nécéssaire. 3. S'il reste encore des Endpoints nouveaux à ajouter, essayez de les mettre dans une slice qui n'a pas été changée et/ou en créer une nouvelle. -Par-dessus tout, la troisième étape priorise la limitation de mises à jour d'EnpointSlice sur une distribution complètement pleine d'EndpointSlices. Par exemple, s'il y avait 10 nouveaux Endpoints à ajouter et 2 EndpointSlices qui peuvent contenir 5 Endpoints en plus chacun; cette approche créera un nouveau EndpointSlice au lieu de remplir les EndpointSlice existants. -C'est à dire, une seule création EndpointSlice est préférable à plusieurs mises à jour d'EndpointSlices. +Par-dessus tout, la troisième étape priorise la limitation de mises à jour d'EndpointSlice sur une distribution complètement pleine d'EndpointSlices. Par exemple, s'il y avait 10 nouveaux Endpoints à ajouter et 2 EndpointSlices qui peuvent contenir 5 Endpoints en plus chacun; cette approche créera un nouveau EndpointSlice au lieu de remplir les EndpointSlice existants. C'est à dire, une seule création EndpointSlice est préférable à plusieurs mises à jour d'EndpointSlices. -Avec kube-proxy exécuté sur chaque Node et surveillant EndpointSlices, chaque changement d'un EndpointSlice devient relativement coûteux puisqu'ils seront transmis à chaque Node du cluster. +Avec kube-proxy exécuté sur chaque Node et surveillant EndpointSlices, chaque changement d'un EndpointSlice devient relativement coûteux puisqu'ils seront transmis à chaque Node du cluster. Cette approche vise à limiter le nombre de modifications qui doivent être envoyées à chaque Node, même si ça peut causer plusieurs EndpointSlices non remplis. En pratique, cette distribution bien peu idéale devrait être rare. La plupart des changements traités par le contrôleur EndpointSlice seront suffisamment petits pour tenir dans un EndpointSlice existant, et sinon, un nouveau EndpointSlice aurait probablement été bientôt nécessaire de toute façon. Les mises à jour continues des déploiements fournissent également une compaction naturelle des EndpointSlices avec tous leurs pods et les Endpoints correspondants qui se feront remplacer. ## Motivation -L'API des Endpoints fournit une méthode simple et facile à suivre pour les Endpoints dans Kubernetes. -Malheureusement, comme les clusters Kubernetes et Services sont devenus plus grands, les limitations de cette API sont devenues plus visibles. -Plus particulièrement, celles-ci comprennent des limitations liées au dimensionnement vers un plus grand nombre d'Endpoints d'un réseau. +L'API des Endpoints fournit une méthode simple et facile à suivre pour les Endpoints dans Kubernetes. Malheureusement, comme les clusters Kubernetes et Services sont devenus plus grands, les limitations de cette API sont devenues plus visibles. Plus particulièrement, celles-ci comprennent des limitations liées au dimensionnement vers un plus grand nombre d'Endpoints d'un réseau. -Puisque tous les Endpoints d'un réseau pour un Service ont été stockés dans une seule ressource Endpoints, ces ressources pourraient devenir assez lourdes. -Cela affecte les performances des composants Kubernetes (notamment le plan de contrôle) et cause une grande quantité de trafic réseau et de traitements lorsque les Endpoints changent. -Les EndpointSlices aident à atténuer ces problèmes ainsi qu'à fournir une plate-forme extensible pour des fonctionnalités supplémentaires telles que le routage topologique. +Puisque tous les Endpoints d'un réseau pour un Service ont été stockés dans une seule ressource Endpoints, ces ressources pourraient devenir assez lourdes. Cela affecte les performances des composants Kubernetes (notamment le plan de contrôle) et cause une grande quantité de trafic réseau et de traitements lorsque les Endpoints changent. Les EndpointSlices aident à atténuer ces problèmes ainsi qu'à fournir une plate-forme extensible pour des fonctionnalités supplémentaires telles que le routage topologique. From 570757a47a64b61780d1fdc337ea0703d1d7f271 Mon Sep 17 00:00:00 2001 From: Ritikaa96 Date: Wed, 21 Jul 2021 16:46:18 +0530 Subject: [PATCH 008/249] updatnig flannel link address Signed-off-by: Ritikaa96 --- content/en/docs/concepts/cluster-administration/networking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md index c517b13175..05a4a7e952 100644 --- a/content/en/docs/concepts/cluster-administration/networking.md +++ b/content/en/docs/concepts/cluster-administration/networking.md @@ -269,7 +269,7 @@ Lars Kellogg-Stedman. [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes. -Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes. +Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/cni.dev/blob/main/content/plugins/v0.9/meta/flannel.md), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes. ### OVN4NFV-K8s-Plugin (OVN based CNI controller & plugin) From ff636956661b848ba2919bb8e2fc6de6df8dab99 Mon Sep 17 00:00:00 2001 From: Abhijit Hoskeri Date: Fri, 23 Jul 2021 13:03:33 -0700 Subject: [PATCH 009/249] encrypt-data: Don't recommend AES-CBC CBC is not recommended any more due to vulnerability to padding oracle attacks. Promote secretbox instead. --- content/en/docs/tasks/administer-cluster/encrypt-data.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md index 8499855bb0..36d1517f85 100644 --- a/content/en/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md @@ -58,8 +58,7 @@ resources: Each `resources` array item is a separate config and contains a complete configuration. The `resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`) that should be encrypted. The `providers` array is an ordered list of the possible encryption -providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, -but not both in the same item). +providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, but not both in the same item). The first provider in the list is used to encrypt resources going into storage. When reading resources from storage each provider that matches the stored data attempts to decrypt the data in @@ -78,9 +77,9 @@ read that resource will fail until it is deleted or a valid decryption key is pr Name | Encryption | Strength | Speed | Key Length | Other Considerations -----|------------|----------|-------|------------|--------------------- `identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. -`aescbc` | AES-CBC with PKCS#7 padding | Strongest | Fast | 32-byte | The recommended choice for encryption at rest but may be slightly slower than `secretbox`. `secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review. `aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented. +`aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. `kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/) Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider @@ -215,5 +214,3 @@ and restart all `kube-apiserver` processes. Then run: kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` to force all secrets to be decrypted. - - From 0032199ed69d08825bbbbdd0773b41b425694baa Mon Sep 17 00:00:00 2001 From: Sanu Satyadarshi Date: Mon, 2 Aug 2021 12:55:33 +0530 Subject: [PATCH 010/249] Add missing ServiceAccount Added missing ServiceAccount, ClusterRole and ClusterRoleBinding. https://github.com/kubernetes/website/issues/15280 https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml --- .../admin/dns/dns-horizontal-autoscaler.yaml | 95 +++++++++++++++---- 1 file changed, 78 insertions(+), 17 deletions(-) diff --git a/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml b/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml index dff87cf851..ada91c1a35 100644 --- a/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml +++ b/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml @@ -1,33 +1,94 @@ +kind: ServiceAccount +apiVersion: v1 +metadata: + name: kube-dns-autoscaler + namespace: kube-system + labels: + addonmanager.kubernetes.io/mode: Reconcile +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:kube-dns-autoscaler + labels: + addonmanager.kubernetes.io/mode: Reconcile +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["replicationcontrollers/scale"] + verbs: ["get", "update"] + - apiGroups: ["apps"] + resources: ["deployments/scale", "replicasets/scale"] + verbs: ["get", "update"] +# Remove the configmaps rule once below issue is fixed: +# kubernetes-incubator/cluster-proportional-autoscaler#16 + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:kube-dns-autoscaler + labels: + addonmanager.kubernetes.io/mode: Reconcile +subjects: + - kind: ServiceAccount + name: kube-dns-autoscaler + namespace: kube-system +roleRef: + kind: ClusterRole + name: system:kube-dns-autoscaler + apiGroup: rbac.authorization.k8s.io + +--- apiVersion: apps/v1 kind: Deployment metadata: - name: dns-autoscaler + name: kube-dns-autoscaler namespace: kube-system labels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler + kubernetes.io/cluster-service: "true" + addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler template: metadata: labels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler spec: + priorityClassName: system-cluster-critical + securityContext: + seccompProfile: + type: RuntimeDefault + supplementalGroups: [ 65534 ] + fsGroup: 65534 + nodeSelector: + kubernetes.io/os: linux containers: - name: autoscaler - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.6.0 + image: k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.4 resources: - requests: - cpu: 20m - memory: 10Mi + requests: + cpu: "20m" + memory: "10Mi" command: - - /cluster-proportional-autoscaler - - --namespace=kube-system - - --configmap=dns-autoscaler - - --target= - # When cluster is using large nodes(with more cores), "coresPerReplica" should dominate. - # If using small nodes, "nodesPerReplica" should dominate. - - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"min":1}} - - --logtostderr=true - - --v=2 + - /cluster-proportional-autoscaler + - --namespace=kube-system + - --configmap=kube-dns-autoscaler + # Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base + - --target= + # When cluster is using large nodes(with more cores), "coresPerReplica" should dominate. + # If using small nodes, "nodesPerReplica" should dominate. + - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true,"includeUnschedulableNodes":true}} + - --logtostderr=true + - --v=2 + tolerations: + - key: "CriticalAddonsOnly" + operator: "Exists" + serviceAccountName: kube-dns-autoscaler From 9bd532fc6728d81ead12ad3aa51526f8f1e91db7 Mon Sep 17 00:00:00 2001 From: "Claudia J. Kang" Date: Thu, 5 Aug 2021 15:04:31 +0900 Subject: [PATCH 011/249] [ko] Enhance docs/tasks/tools/install-kubectl-windows.md --- .../ko/docs/tasks/tools/install-kubectl-windows.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/ko/docs/tasks/tools/install-kubectl-windows.md b/content/ko/docs/tasks/tools/install-kubectl-windows.md index bb5b45831e..b05bd7ab02 100644 --- a/content/ko/docs/tasks/tools/install-kubectl-windows.md +++ b/content/ko/docs/tasks/tools/install-kubectl-windows.md @@ -47,20 +47,20 @@ card: kubectl 바이너리를 체크섬 파일을 통해 검증한다. - - 수동으로 `CertUtil` 의 출력과 다운로드한 체크섬 파일을 비교하기 위해서 커맨드 프롬프트를 사용한다. + - 커맨드 프롬프트를 사용하는 경우, `CertUtil` 의 출력과 다운로드한 체크섬 파일을 수동으로 비교한다. ```cmd CertUtil -hashfile kubectl.exe SHA256 type kubectl.exe.sha256 ``` - - `-eq` 연산자를 통해 `True` 또는 `False` 결과를 얻는 자동 검증을 위해서 PowerShell을 사용한다. + - PowerShell을 사용하는 경우, `-eq` 연산자를 통해 `True` 또는 `False` 결과가 출력되는 자동 검증을 수행한다. ```powershell $($(CertUtil -hashfile .\kubectl.exe SHA256)[1] -replace " ", "") -eq $(type .\kubectl.exe.sha256) ``` -1. 바이너리를 `PATH` 가 설정된 디렉터리에 추가한다. +1. `PATH`로 설정된 디렉터리 중 하나에 kubectl 바이너리를 추가한다. 1. `kubectl` 의 버전이 다운로드한 버전과 같은지 확인한다. @@ -160,20 +160,20 @@ kubectl은 Bash 및 Zsh에 대한 자동 완성 지원을 제공하므로 입력 kubectl-convert 바이너리를 체크섬 파일을 통해 검증한다. - - 수동으로 `CertUtil` 의 출력과 다운로드한 체크섬 파일을 비교하기 위해서 커맨드 프롬프트를 사용한다. + - 커맨드 프롬프트를 사용하는 경우, `CertUtil` 의 출력과 다운로드한 체크섬 파일을 수동으로 비교한다. ```cmd CertUtil -hashfile kubectl-convert.exe SHA256 type kubectl-convert.exe.sha256 ``` - - `-eq` 연산자를 통해 `True` 또는 `False` 결과를 얻는 자동 검증을 위해서 PowerShell을 사용한다. + - PowerShell을 사용하는 경우, `-eq` 연산자를 통해 `True` 또는 `False` 결과가 출력되는 자동 검증을 수행한다. ```powershell $($(CertUtil -hashfile .\kubectl-convert.exe SHA256)[1] -replace " ", "") -eq $(type .\kubectl-convert.exe.sha256) ``` -1. 바이너리를 `PATH` 가 설정된 디렉터리에 추가한다. +1. `PATH`로 설정된 디렉터리 중 하나에 kubectl-convert 바이너리를 추가한다. 1. 플러그인이 정상적으로 설치되었는지 확인한다. From 3571016b3c3a27da5a404f2275873e2505f6680f Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Fri, 13 Aug 2021 11:47:50 -0300 Subject: [PATCH 012/249] v0.1 for sig node spotlight --- .../2021-08-13-SIG-Node-Spotlight/index.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md new file mode 100644 index 0000000000..a3bb3367ad --- /dev/null +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -0,0 +1,76 @@ +--- +layout: blog +title: "Spotlight on SIG Node" +date: 2021-08-13 +slug: sig-node-spotlight-2021 +--- + +**Author:** Dewan Ahmed, Red Hat + +## Introduction + +Are you interested in learning about what [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) does and how you can get involved? Well, you're at the right place. SIG Node is responsible for the components that support the controlled interactions between pods and host resources. In this blog, we have summarized our conversation with [Elana Hashman](https://twitter.com/ehashdn) & [Sergey Kanzhelev](https://twitter.com/SergeyKanzhelev), who walks us through the various aspects of being a part of the SIG and shares some insights about how others can get involved. + + +## A summary of our conversation + +### DA. Could you tell us a little about what SIG Node does? + +SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIGs focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. + + + +### DA. "SIG Node is a vertical SIG" could you explain a bit more? + +EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code. + +Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector. + +### DA. How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG? + +SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority. + +### DA. As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work? + +EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result. + +In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues. + +### DA. Why should new and existing contributors consider joining SIG Node? + +SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster. + +The benefits of SIG Node is also this SIG challenge. SIG maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. + +### DA. What do you do to help new contributors get started? + +EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start. + +I always encourage new contributors to work on things that they have some investment in already. In SIG Node, that might mean volunteering to help fix a bug that you have personally been affected by, or helping to triage bugs you care about by priority. + +To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver. + +[Davanum Srinivas](https://twitter.com/@dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. + + +### DA. Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? + +SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. + +The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms. + +Finally, we are always looking for feedback on how best to run your workload. Come and explain specifics of it, and what features in SIG Node components may help to run it better. + +### DA. What are you getting positive feedback on, and what’s coming up next for SIG Node? + +EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on. + +We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies. + +### DA: Any closing thoughts/resources you’d like to share? + +SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info. + +## Wrap Up + +SIG Node hosted a [KubeCon + CloudNativeCon Europe 2021 talk](https://www.youtube.com/watch?v=z5aY4e2RENA) with an intro and deep dive to their awesome SIG. Join the SIG's meetings to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream Node team as a contributor! \ No newline at end of file From ad05f090e7f2c572236fbc84a1c994bf6b61c5b8 Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Fri, 13 Aug 2021 11:50:13 -0300 Subject: [PATCH 013/249] Removing interviewer initials --- .../2021-08-13-SIG-Node-Spotlight/index.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index a3bb3367ad..51c1e5486f 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -14,35 +14,35 @@ Are you interested in learning about what [SIG Node](https://github.com/kubernet ## A summary of our conversation -### DA. Could you tell us a little about what SIG Node does? +### Could you tell us a little about what SIG Node does? SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIGs focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. -### DA. "SIG Node is a vertical SIG" could you explain a bit more? +### "SIG Node is a vertical SIG" could you explain a bit more? EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code. Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector. -### DA. How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG? +### How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG? SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority. -### DA. As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work? +### As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work? EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result. In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues. -### DA. Why should new and existing contributors consider joining SIG Node? +### Why should new and existing contributors consider joining SIG Node? SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster. The benefits of SIG Node is also this SIG challenge. SIG maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. -### DA. What do you do to help new contributors get started? +### What do you do to help new contributors get started? EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start. @@ -53,7 +53,7 @@ To come up to speed on any open source code base, there are two strategies you c [Davanum Srinivas](https://twitter.com/@dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. -### DA. Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? +### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. @@ -61,13 +61,13 @@ The other way of contribution is to help document SIG node features. This type o Finally, we are always looking for feedback on how best to run your workload. Come and explain specifics of it, and what features in SIG Node components may help to run it better. -### DA. What are you getting positive feedback on, and what’s coming up next for SIG Node? +### What are you getting positive feedback on, and what’s coming up next for SIG Node? EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on. We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies. -### DA: Any closing thoughts/resources you’d like to share? +### Any closing thoughts/resources you’d like to share? SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info. From df8f30b1209a6d7b3a69cee8ef61c9ba30781af3 Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Fri, 13 Aug 2021 12:26:24 -0300 Subject: [PATCH 014/249] Fixing minor typos. --- .../en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 51c1e5486f..335344f9e6 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -16,7 +16,7 @@ Are you interested in learning about what [SIG Node](https://github.com/kubernet ### Could you tell us a little about what SIG Node does? -SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIGs focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. +SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. @@ -38,9 +38,9 @@ In June, we ran our first bug scrub event to work through the backlog of issues ### Why should new and existing contributors consider joining SIG Node? -SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster. +SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you the foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster. -The benefits of SIG Node is also this SIG challenge. SIG maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. +SIG Node maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. ### What do you do to help new contributors get started? @@ -50,7 +50,7 @@ I always encourage new contributors to work on things that they have some invest To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver. -[Davanum Srinivas](https://twitter.com/@dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. +[Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. ### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? From 5b34d70e92a10347180bf5da761d687e757ede5f Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Fri, 13 Aug 2021 14:24:53 -0300 Subject: [PATCH 015/249] Post-PR grammatical fixes --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 335344f9e6..1dbf20eed7 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -9,7 +9,7 @@ slug: sig-node-spotlight-2021 ## Introduction -Are you interested in learning about what [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) does and how you can get involved? Well, you're at the right place. SIG Node is responsible for the components that support the controlled interactions between pods and host resources. In this blog, we have summarized our conversation with [Elana Hashman](https://twitter.com/ehashdn) & [Sergey Kanzhelev](https://twitter.com/SergeyKanzhelev), who walks us through the various aspects of being a part of the SIG and shares some insights about how others can get involved. +Are you interested in learning about what [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) does and how you can get involved? Well, you're at the right place. SIG Node is responsible for the components that support the controlled interactions between pods and host resources. In this blog, we have summarized our conversation with [Elana Hashman](https://twitter.com/ehashdn) & [Sergey Kanzhelev](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved. ## A summary of our conversation From 583e4d50e40924cc5e3cdb18d6bacf0cf56375b7 Mon Sep 17 00:00:00 2001 From: "Claudia J. Kang" Date: Thu, 5 Aug 2021 14:30:23 +0900 Subject: [PATCH 016/249] [ko] Translate /tasks/debug-application-cluster/debug-running-pod.md --- .../debug-running-pod.md | 340 ++++++++++++++++++ 1 file changed, 340 insertions(+) create mode 100644 content/ko/docs/tasks/debug-application-cluster/debug-running-pod.md diff --git a/content/ko/docs/tasks/debug-application-cluster/debug-running-pod.md b/content/ko/docs/tasks/debug-application-cluster/debug-running-pod.md new file mode 100644 index 0000000000..0145967dd8 --- /dev/null +++ b/content/ko/docs/tasks/debug-application-cluster/debug-running-pod.md @@ -0,0 +1,340 @@ +--- + + + +title: 동작 중인 파드 디버그 +content_type: task +--- + + + +이 페이지는 노드에서 동작 중인(혹은 크래시된) 파드를 디버그하는 방법에 대해 설명한다. + + + +## {{% heading "prerequisites" %}} + + +* 여러분의 {{< glossary_tooltip text="파드" term_id="pod" >}}는 이미 스케줄링 되어 + 동작하고 있을 것이다. 만약 파드가 아직 동작중이지 않다면, [애플리케이션 + 트러블슈팅](/docs/tasks/debug-application-cluster/debug-application/)을 참고한다. +* 일부 고급 디버깅 과정에서는 해당 파드가 어떤 노드에서 동작하고 있는지 + 알아야 하고, 해당 노드에서 쉘 명령어를 실행시킬 수 있어야 한다. + `kubectl`을 사용하는 일반적인 디버깅 과정에서는 이러한 접근 권한이 필요하지 않다. + + + + + +## 파드의 로그 확인하기 {#examine-pod-logs} + +먼저, 확인하고자 하는 컨테이너의 로그를 확인한다. + +```shell +kubectl logs ${POD_NAME} ${CONTAINER_NAME} +``` + +만약 컨테이너가 이전에 크래시 되었다면, 다음의 명령을 통해 컨테이너의 크래시 로그를 살펴볼 수 있다. + +```shell +kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME} +``` + +## exec를 통해 컨테이너 디버깅하기 {#container-exec} + +만약 {{< glossary_tooltip text="컨테이너 이미지" term_id="image" >}}에 +디버깅 도구가 포함되어 있다면, `kubectl exec`을 통해 특정 컨테이너에서 해당 명령들을 +실행할 수 있다. (리눅스나 윈도우 OS를 기반으로 만들어진 이미지에는 대부분 디버깅 도구를 포함하고 +있다.) + +```shell +kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN} +``` + +{{< note >}} +`-c ${CONTAINER_NAME}` 인자는 선택적이다. 만약 하나의 컨테이너만 포함된 파드라면 해당 옵션을 생략할 수 있다. +{{< /note >}} + +예를 들어, 동작 중인 카산드라 파드의 로그를 살펴보기 위해서는 다음과 같은 명령을 실행할 수 있다. + +```shell +kubectl exec cassandra -- cat /var/log/cassandra/system.log +``` + +`kubectl exec`에 `-i`와 `-t` 옵션을 사용해서 터미널에서 접근할 수 있는 쉘을 실행시킬 수도 있다. +예를 들면 다음과 같다. + +```shell +kubectl exec -it cassandra -- sh +``` + +더욱 상세한 내용은 다음 [동작중인 컨테이너의 쉘에 접근하기]( +/docs/tasks/debug-application-cluster/get-shell-running-container/)를 참고하라. + +## 임시(ephemeral) 디버그 컨테이너를 사용해서 디버깅하기 {#ephemeral-container} + +{{< feature-state state="alpha" for_k8s_version="v1.18" >}} + +컨테이너가 크래시 됐거나 [distroless 이미지](https://github.com/GoogleContainerTools/distroless)처럼 +컨테이너 이미지에 디버깅 도구를 포함하고 있지 않아 +`kubectl exec`가 충분하지 않을 경우에는 +{{< glossary_tooltip text="임시(Ephemeral) 컨테이너" term_id="ephemeral-container" >}}를 사용하는 것이 +인터랙티브한 트러블슈팅에 유용하다. `kubectl` `v1.18` +버전부터는 임시 컨테이너를 생성할 수 있는 알파 단계의 +명령어가 있다. + +### 임시 컨테이너를 사용한 디버깅 예시 {#ephemeral-container-example} + +{{< note >}} +이 섹션에서 소개하는 예시를 사용하기 위해서는 +여러분의 클러스터에 `EphemeralContainers` [기능 게이트](/docs/reference/command-line-tools-reference/feature-gates/)가 +활성화되어 있어야 하고 `kubectl`의 버전이 v1.18 이상이어야 한다. +{{< /note >}} + +`kubectl debug` 명령어를 사용해서 동작 중인 파드에 임시 컨테이너를 추가할 수 있다. +먼저, 다음과 같이 파드를 추가한다. + +```shell +kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never +``` + +이 섹션의 예시에서는 디버깅 도구가 포함되지 않은 이미지의 사례를 보여드리기 위해 +`pause` 컨테이너 이미지를 사용했는데, 이 대신 어떠한 이미지를 사용해도 +될 것이다. + +만약 `kubectl exec`을 통해 쉘을 생성하려 한다면 다음과 같은 에러를 +확인할 수 있을 텐데, 그 이유는 이 이미지에 쉘이 존재하지 않기 때문이다. + +```shell +kubectl exec -it ephemeral-demo -- sh +``` + +``` +OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown +``` + +이 명령어 대신 `kubectl debug`을 사용해서 디버깅 컨테이너를 생성할 수 있다. +만약 `-i`/`--interactive` 인자를 사용한다면, `kubectl`은 임시 +컨테이너의 콘솔에 자동으로 연결할 것이다. + +```shell +kubectl debug -it ephemeral-demo --image=busybox --target=ephemeral-demo +``` + +``` +Defaulting debug container name to debugger-8xzrl. +If you don't see a command prompt, try pressing enter. +/ # +``` + +이 명령어는 새로운 busybox 컨테이너를 추가하고 해당 컨테이너로 연결한다. `--target` +파라미터를 사용하면 다른 컨테이너의 프로세스 네임스페이스를 대상으로 하게 된다. 여기서는 +이 옵션이 꼭 필요한데, `kubectl run`이 생성하는 파드에 대해 +[프로세스 네임스페이스 공유](/docs/tasks/configure-pod-container/share-process-namespace/)를 +활성화하지 않기 때문이다. + +{{< note >}} +`--target` 파라미터는 사용 중인 {{< glossary_tooltip text="컨테이너 런타임" term_id="container-runtime" >}}에서 +지원해야지만 사용할 수 있다. 만일 지원되지 않는다면, +임시 컨테이너가 시작되지 않을 수 있거나 독립적인 프로세스 +네임스페이스를 가지고 시작될 수 있다. +{{< /note >}} + +`kubectl describe` 명령을 사용하면 새롭게 생성된 임시 컨테이너의 상태를 확인할 수 있다. + +```shell +kubectl describe pod ephemeral-demo +``` + +``` +... +Ephemeral Containers: + debugger-8xzrl: + Container ID: docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb + Image: busybox + Image ID: docker-pullable://busybox@sha256:1828edd60c5efd34b2bf5dd3282ec0cc04d47b2ff9caa0b6d4f07a21d1c08084 + Port: + Host Port: + State: Running + Started: Wed, 12 Feb 2020 14:25:42 +0100 + Ready: False + Restart Count: 0 + Environment: + Mounts: +... +``` + +디버깅이 다 끝나면 `kubectl delete`을 통해 파드를 제거할 수 있다. + +```shell +kubectl delete pod ephemeral-demo +``` + +## 파드의 복제본을 이용해서 디버깅하기 + +때때로 파드의 설정 옵션에 따라 특정 상황에서 트러블슈팅을 하기가 어려울 수 있다. +예를 들어, 만일 여러분의 컨테이너 이미지가 쉘을 포함하고 있지 않거나, 여러분의 +애플리케이션이 컨테이너 시작에서 크래시가 발생한다면 `kubectl exec`을 이용해서 +컨테이너를 트러블슈팅할 수 없을 수 있다. 이러한 상황에서는 `kubectl debug`을 사용해서 +파드의 복제본을 디버깅을 위한 추가적인 설정 옵션과 함께 생성할 수 있다. + +### 새 컨테이너와 함께 파드의 복제본 생성하기 + +만일 여러분의 애플리케이션이 동작은 하고 있지만 예상과는 다르게 동작하는 경우, +파드의 복제본에 새로운 컨테이너를 추가함으로써 추가적인 트러블슈팅 도구들을 +파드에 함께 추가할 수 있다. + +가령, 여러분의 애플리케이션 컨테이너 이미지는 `busybox`를 기반으로 하고 있는데 +여러분은 `busybox`에는 없는 디버깅 도구를 필요로 한다고 가정해 보자. 이러한 +시나리오는 `kubectl run` 명령을 통해 시뮬레이션 해볼 수 있다. + +```shell +kubectl run myapp --image=busybox --restart=Never -- sleep 1d +``` + +다음의 명령을 실행시켜 디버깅을 위한 새로운 우분투 컨테이너와 함께 `myapp-debug`이란 +이름의 `myapp` 컨테이너 복제본을 생성할 수 있다. + +```shell +kubectl debug myapp -it --image=ubuntu --share-processes --copy-to=myapp-debug +``` + +``` +Defaulting debug container name to debugger-w7xmf. +If you don't see a command prompt, try pressing enter. +root@myapp-debug:/# +``` + +{{< note >}} +* 만일 여러분이 새로 생성되는 컨테이너의 이름을 `--container` 플래그와 함께 지정하지 않는다면, + `kubectl debug`는 자동으로 새로운 컨테이너 이름을 생성한다. +* `-i` 플래그를 사용하면 `kubectl debug` 명령이 새로운 컨테이너에 기본적으로 연결되게 된다. + 이러한 동작은 `--attach=false`을 지정하여 방지할 수 있다. 만일 여러분의 세션이 + 연결이 끊어진다면 `kubectl attach`를 사용해서 다시 연결할 수 있다. +* `--share-processes` 옵션은 이 파드에 있는 컨테이너가 해당 파드에 속한 다른 컨테이너의 + 프로세스를 볼 수 있도록 한다. 이 옵션이 어떻게 동작하는지에 대해 더 알아보기 위해서는 + 다음의 [파드의 컨테이너 간 프로세스 네임스페이스 공유]( + /docs/tasks/configure-pod-container/share-process-namespace/)를 참고하라. +{{< /note >}} + +사용이 모두 끝나면, 디버깅에 사용된 파드를 잊지 말고 정리한다. + +```shell +kubectl delete pod myapp myapp-debug +``` + +### 명령어를 변경하며 파드의 복제본 생성하기 + +때때로 컨테이너의 명령어를 변경하는 것이 유용한 경우가 있는데, 예를 들면 디버그 플래그를 추가하기 +위해서나 애플리케이션이 크래시 되는 경우이다. + +다음의 `kubectl run` 명령을 통해 즉각적으로 크래시가 발생하는 애플리케이션의 +사례를 시뮬레이션해 볼 수 있다. + +``` +kubectl run --image=busybox myapp -- false +``` + +`kubectl describe pod myapp` 명령을 통해 이 컨테이너에 크래시가 발생하고 있음을 확인할 수 있다. + +``` +Containers: + myapp: + Image: busybox + ... + Args: + false + State: Waiting + Reason: CrashLoopBackOff + Last State: Terminated + Reason: Error + Exit Code: 1 +``` + +이러한 경우에 `kubectl debug`을 통해 명령어를 지정함으로써 해당 파드의 +복제본을 인터랙티브 쉘로 생성할 수 있다. + +``` +kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh +``` + +``` +If you don't see a command prompt, try pressing enter. +/ # +``` + +이제 인터랙티브 쉘에 접근할 수 있으니 파일 시스템 경로를 확인하거나 +동작 중인 컨테이너의 명령어를 직접 확인하는 등의 작업이 가능하다. + +{{< note >}} +* 특정 컨테이너의 명령어를 변경하기 위해서는 `--container` 옵션을 통해 해당 컨테이너의 + 이름을 지정해야만 한다. 이름을 지정하지 않는다면 `kubectl debug`은 이전에 지정한 명령어를 + 그대로 사용해서 컨테이너를 생성할 것이다. +* 기본적으로 `-i` 플래그는 `kubectl debug` 명령이 컨테이너에 바로 연결되도록 한다. + 이러한 동작을 방지하기 위해서는 `--attach=false` 옵션을 지정할 수 있다. 만약 여러분이 세션이 + 종료된다면 `kubectl attach` 명령을 통해 다시 연결할 수 있다. +{{< /note >}} + +사용이 모두 끝나면, 디버깅에 사용된 파드들을 잊지 말고 정리한다. + +```shell +kubectl delete pod myapp myapp-debug +``` + +### 컨테이너 이미지를 변경하며 파드의 복제본 생성하기 + +특정한 경우에 여러분은 제대로 동작하지 않는 파드의 이미지를 +기존 프로덕션 컨테이너 이미지에서 디버깅 빌드나 추가적인 도구를 포함한 +이미지로 변경하고 싶을 수 있다. + +이 사례를 보여주기 위해 `kubectl run` 명령을 통해 파드를 생성하였다. + +``` +kubectl run myapp --image=busybox --restart=Never -- sleep 1d +``` + +여기서는 `kubectl debug` 명령을 통해 해당 컨테이너 이미지를 `ubuntu`로 변경하며 +복제본을 생성하였다. + +``` +kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu +``` + +`--set-image`의 문법은 `kubectl set image`와 동일하게 `container_name=image` +형식의 문법을 사용한다. `*=ubuntu`라는 의미는 모든 컨테이너의 이미지를 `ubuntu`로 +변경하겠다는 의미이다. + +사용이 모두 끝나면, 디버깅에 사용된 파드를 잊지 말고 정리한다. + +```shell +kubectl delete pod myapp myapp-debug +``` + +## 노드의 쉘을 사용해서 디버깅하기 {#node-shell-session} + +만약 위의 어떠한 방법도 사용할 수 없다면, 파드가 현재 동작 중인 노드를 찾아 +호스트의 네임스페이스로 동작하는 특권 파드를 생성할 수 있다. +다음 `kubectl debug` 명령을 통해 해당 노드에서 인터랙티브한 쉘을 생성할 수 있다. + +```shell +kubectl debug node/mynode -it --image=ubuntu +``` + +``` +Creating debugging pod node-debugger-mynode-pdx84 with container debugger on node mynode. +If you don't see a command prompt, try pressing enter. +root@ek8s:/# +``` + +노드에서 디버깅 세션을 생성할 때 유의해야 할 점은 다음과 같다. + +* `kubectl debug`는 노드의 이름에 기반해 새로운 파드의 이름을 + 자동으로 생성한다. +* 컨테이너는 호스트 네임스페이스(IPC, 네트워크, PID 네임스페이스)에서 동작한다. +* 노드의 루트 파일시스템은 `/host`에 마운트된다. + +사용이 모두 끝나면, 디버깅에 사용된 파드를 잊지 말고 정리한다. + +```shell +kubectl delete pod node-debugger-mynode-pdx84 +``` From 3f51160e89ea5c67a8b1f884e299a69afa27b48a Mon Sep 17 00:00:00 2001 From: Dewan Ishtiaque Ahmed <30604461+dewan-ahmed@users.noreply.github.com> Date: Wed, 25 Aug 2021 17:28:13 -0300 Subject: [PATCH 017/249] Update content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md Co-authored-by: Rey Lejano --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 1dbf20eed7..f09527f977 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -55,7 +55,7 @@ To come up to speed on any open source code base, there are two strategies you c ### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? -SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. +SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms. From f93e3cb2bec9d5f004c3d305fbd9625918dcada7 Mon Sep 17 00:00:00 2001 From: Dewan Ishtiaque Ahmed <30604461+dewan-ahmed@users.noreply.github.com> Date: Wed, 25 Aug 2021 17:28:41 -0300 Subject: [PATCH 018/249] Update content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md Co-authored-by: Rey Lejano --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index f09527f977..4d05fe0790 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -52,7 +52,6 @@ To come up to speed on any open source code base, there are two strategies you c [Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. - ### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn? SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads. From 2b09e539efcb84283df79dccc85c20dbdfbab927 Mon Sep 17 00:00:00 2001 From: Dewan Ishtiaque Ahmed <30604461+dewan-ahmed@users.noreply.github.com> Date: Wed, 25 Aug 2021 17:28:56 -0300 Subject: [PATCH 019/249] Update content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md Co-authored-by: Rey Lejano --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 4d05fe0790..269b6cc140 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -1,7 +1,7 @@ --- layout: blog title: "Spotlight on SIG Node" -date: 2021-08-13 +date: 2021-09-06 slug: sig-node-spotlight-2021 --- From 25a7c8f02e76128bdc270475c73123e45810daad Mon Sep 17 00:00:00 2001 From: Dewan Ishtiaque Ahmed <30604461+dewan-ahmed@users.noreply.github.com> Date: Wed, 25 Aug 2021 17:29:05 -0300 Subject: [PATCH 020/249] Update content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md Co-authored-by: Rey Lejano --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 269b6cc140..8a9b07c514 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -18,8 +18,6 @@ Are you interested in learning about what [SIG Node](https://github.com/kubernet SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability. - - ### "SIG Node is a vertical SIG" could you explain a bit more? EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code. From 47262d1bba539989503d44ff4225d7488bb346b2 Mon Sep 17 00:00:00 2001 From: bang9211 Date: Fri, 27 Aug 2021 23:42:24 +0900 Subject: [PATCH 021/249] Translate tasks/inject-data-application/define-interdependent-environment-variables in Korean --- ...ne-interdependent-environment-variables.md | 77 +++++++++++++++++++ .../pods/inject/dependent-envars.yaml | 26 +++++++ 2 files changed, 103 insertions(+) create mode 100644 content/ko/docs/tasks/inject-data-application/define-interdependent-environment-variables.md create mode 100644 content/ko/examples/pods/inject/dependent-envars.yaml diff --git a/content/ko/docs/tasks/inject-data-application/define-interdependent-environment-variables.md b/content/ko/docs/tasks/inject-data-application/define-interdependent-environment-variables.md new file mode 100644 index 0000000000..34a91a600c --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/define-interdependent-environment-variables.md @@ -0,0 +1,77 @@ +--- +title: 종속 환경 변수 정의하기 +content_type: task +weight: 20 +--- + + + +본 페이지는 쿠버네티스 파드의 컨테이너를 위한 종속 환경 변수를 +정의하는 방법에 대해 설명한다. + + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} + + + + +## 컨테이너를 위한 종속 환경 변수 정의하기 + +파드를 생성할 때, 파드 안에서 동작하는 컨테이너를 위한 종속 환경 변수를 설정할 수 있다. +종속 환경 변수를 설정하려면, 구성 파일에서 `env`의 `value`에 $(VAR_NAME)을 사용한다. + +이 예제에서, 한 개의 컨테이너를 실행하는 파드를 생성한다. 파드를 위한 구성 파일은 일반적인 방식으로 정의된 종속 환경 변수를 정의한다. 다음은 파드를 위한 구성 매니페스트 예시이다. + +{{< codenew file="pods/inject/dependent-envars.yaml" >}} + +1. YAML 구성 파일을 활용해 파드를 생성한다. + + ```shell + kubectl apply -f https://k8s.io/examples/pods/inject/dependent-envars.yaml + ``` + ``` + pod/dependent-envars-demo created + ``` + +2. 실행 중인 파드의 목록을 조회한다. + + ```shell + kubectl get pods dependent-envars-demo + ``` + ``` + NAME READY STATUS RESTARTS AGE + dependent-envars-demo 1/1 Running 0 9s + ``` + +3. 파드 안에서 동작 중인 컨테이너의 로그를 확인한다. + + ```shell + kubectl logs pod/dependent-envars-demo + ``` + ``` + + UNCHANGED_REFERENCE=$(PROTOCOL)://172.17.0.1:80 + SERVICE_ADDRESS=https://172.17.0.1:80 + ESCAPED_REFERENCE=$(PROTOCOL)://172.17.0.1:80 + ``` + +위에서 보듯이, `SERVICE_ADDRESS`는 올바른 종속성 참조, `UNCHANGED_REFERENCE`는 잘못된 종속성 참조를 정의했으며 `ESCAPED_REFERENCE`는 종속성 참조를 건너뛴다. + +환경 변수가 참조될 때 해당 환경 변수가 미리 정의되어 있으면 `SERVICE_ADDRESS`의 경우와 같이 참조를 올바르게 해석할 수 있다. + +환경 변수가 정의되지 않았거나 일부 변수만 포함된 경우, 정의되지 않은 환경 변수는 `UNCHANGED_REFERENCE`의 경우와 같이 일반 문자열로 처리된다. +일반적으로 환경 변수 해석에 실패하더라도 컨테이너의 시작을 막지는 않는다. + +`$(VAR_NAME)` 구문은 이중 $로 이스케이프될 수 있다. (예: `$$(VAR_NAME)`) +이스케이프된 참조는 참조된 변수가 정의되었는지 여부에 관계없이 해석을 수행하지 않는다. +이는 위의 `ESCAPED_REFERENCE`를 통해 확인할 수 있다. + +## {{% heading "whatsnext" %}} + + +* [환경 변수](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/)에 대해 알아본다. +* [EnvVarSource](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvarsource-v1-core)를 확인한다. + diff --git a/content/ko/examples/pods/inject/dependent-envars.yaml b/content/ko/examples/pods/inject/dependent-envars.yaml new file mode 100644 index 0000000000..2509c6f47b --- /dev/null +++ b/content/ko/examples/pods/inject/dependent-envars.yaml @@ -0,0 +1,26 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dependent-envars-demo +spec: + containers: + - name: dependent-envars-demo + args: + - while true; do echo -en '\n'; printf UNCHANGED_REFERENCE=$UNCHANGED_REFERENCE'\n'; printf SERVICE_ADDRESS=$SERVICE_ADDRESS'\n';printf ESCAPED_REFERENCE=$ESCAPED_REFERENCE'\n'; sleep 30; done; + command: + - sh + - -c + image: busybox + env: + - name: SERVICE_PORT + value: "80" + - name: SERVICE_IP + value: "172.17.0.1" + - name: UNCHANGED_REFERENCE + value: "$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT)" + - name: PROTOCOL + value: "https" + - name: SERVICE_ADDRESS + value: "$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT)" + - name: ESCAPED_REFERENCE + value: "$$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT)" From a0d63d1125776e25195ba503de02ee542543661a Mon Sep 17 00:00:00 2001 From: Vaibhav Date: Wed, 1 Sep 2021 01:49:22 +0530 Subject: [PATCH 022/249] Remove the --record flag in https://kubernetes.io/docs/concepts/workloads/controllers/deployment.md --- .../workloads/controllers/deployment.md | 21 +++++++------------ 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index 22b95255c5..abe88d9281 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -75,11 +75,6 @@ Follow the steps given below to create the above Deployment: kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml ``` - {{< note >}} - You can specify the `--record` flag to write the command executed in the resource annotation `kubernetes.io/change-cause`. - The recorded change is useful for future introspection. For example, to see the commands executed in each Deployment revision. - {{< /note >}} - 2. Run `kubectl get deployments` to check if the Deployment was created. @@ -169,13 +164,13 @@ Follow the steps given below to update your Deployment: 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image. ```shell - kubectl --record deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubectl deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 ``` or use the following command: ```shell - kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 --record + kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` The output is similar to: @@ -370,7 +365,7 @@ rolled back. * Suppose that you made a typo while updating the Deployment, by putting the image name as `nginx:1.161` instead of `nginx:1.16.1`: ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true + kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 ``` The output is similar to this: @@ -485,9 +480,9 @@ Follow the steps given below to check the rollout history: ``` deployments "nginx-deployment" REVISION CHANGE-CAUSE - 1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml --record=true - 2 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true - 3 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true + 1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml + 2 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + 3 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 ``` `CHANGE-CAUSE` is copied from the Deployment annotation `kubernetes.io/change-cause` to its revisions upon creation. You can specify the`CHANGE-CAUSE` message by: @@ -506,7 +501,7 @@ Follow the steps given below to check the rollout history: deployments "nginx-deployment" revision 2 Labels: app=nginx pod-template-hash=1159050644 - Annotations: kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true + Annotations: kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 Containers: nginx: Image: nginx:1.16.1 @@ -567,7 +562,7 @@ Follow the steps given below to rollback the Deployment from the current version CreationTimestamp: Sun, 02 Sep 2018 18:17:55 -0500 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=4 - kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true + kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 Selector: app=nginx Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate From f6c813de44bf12934125684d7ddeee48cbd668ad Mon Sep 17 00:00:00 2001 From: Vaibhav Date: Wed, 1 Sep 2021 15:21:31 +0530 Subject: [PATCH 023/249] Remove the remaining last flag --record from deployment.md --- content/en/docs/concepts/workloads/controllers/deployment.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index abe88d9281..541b72b245 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -488,7 +488,6 @@ Follow the steps given below to check the rollout history: `CHANGE-CAUSE` is copied from the Deployment annotation `kubernetes.io/change-cause` to its revisions upon creation. You can specify the`CHANGE-CAUSE` message by: * Annotating the Deployment with `kubectl annotate deployment.v1.apps/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"` - * Append the `--record` flag to save the `kubectl` command that is making changes to the resource. * Manually editing the manifest of the resource. 2. To see the details of each revision, run: From bc8814864cd5827432dcdee24a4142cf5abb6f3a Mon Sep 17 00:00:00 2001 From: bang9211 Date: Thu, 2 Sep 2021 04:45:50 +0900 Subject: [PATCH 024/249] Translate tasks/inject-data-application/environment-variable-expose-pod-information in Korean --- ...ronment-variable-expose-pod-information.md | 157 ++++++++++++++++++ .../pods/inject/dapi-envars-container.yaml | 45 +++++ .../examples/pods/inject/dapi-envars-pod.yaml | 38 +++++ 3 files changed, 240 insertions(+) create mode 100644 content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md create mode 100644 content/ko/examples/pods/inject/dapi-envars-container.yaml create mode 100644 content/ko/examples/pods/inject/dapi-envars-pod.yaml diff --git a/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md b/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md new file mode 100644 index 0000000000..033e0afa79 --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md @@ -0,0 +1,157 @@ +--- +title: 환경 변수로 컨테이너에 파드 정보 노출하기 +content_type: task +weight: 30 +--- + + + +본 페이지는 파드에서 실행 중인 컨테이너에게 파드가 환경 변수를 사용해서 자신의 정보를 노출하는 방법에 +대해 설명한다. 환경 변수는 파드 필드와 컨테이너 필드를 노출할 수 있다. + + + + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + + + +## 다운워드(Downward) API + +파드 및 컨테이너 필드를 실행 중인 컨테이너에 노출하는 두 가지 방법이 있다. + +* 환경 변수 +* [볼륨 파일](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api) + +파드 및 컨테이너 필드를 노출하는 이 두 가지 방법을 *다운워드 API*라고 한다. + + +## 파드 필드를 환경 변수의 값으로 사용하자 + +이 연습에서는 하나의 컨테이너를 가진 파드를 생성한다. 다음은 파드에 대한 구성 파일이다. + +{{< codenew file="pods/inject/dapi-envars-pod.yaml" >}} + +구성 파일에서 5개의 환경 변수를 확인할 수 있다. `env` 필드는 +[EnvVars](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core)의 배열이다. 배열의 첫 번째 요소는 `MY_NODE_NAME` 환경 변수가 파드의 `spec.nodeName` 필드에서 값을 가져오도록 지정한다. 마찬가지로 다른 환경 변수도 파드 필드에서 이름을 가져온다. + +{{< note >}} +이 예제의 필드는 파드에 있는 컨테이너의 필드가 아니라 파드 필드이다. +{{< /note >}} + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-pod.yaml +``` + +파드의 컨테이너가 실행중인지 확인한다. + +```shell +kubectl get pods +``` + +컨테이너의 로그를 본다. + +```shell +kubectl logs dapi-envars-fieldref +``` + +출력은 선택된 환경 변수의 값을 보여준다. + +``` +minikube +dapi-envars-fieldref +default +172.17.0.4 +default +``` + +이러한 값이 로그에 출력된 이유를 보려면 구성 파일의 `command` 및 `args` 필드를 확인하자. +컨테이너가 시작되면 5개의 환경 변수 값을 stdout에 쓰며 10초마다 이를 반복한다. + +다음으로 파드에서 실행 중인 컨테이너의 셸을 가져오자. + +```shell +kubectl exec -it dapi-envars-fieldref -- sh +``` + +셸에서 환경 변수를 보자. + +```shell +/# printenv +``` + +출력은 특정 환경 변수에 파드 필드 값이 할당되었음을 보여준다. + +``` +MY_POD_SERVICE_ACCOUNT=default +... +MY_POD_NAMESPACE=default +MY_POD_IP=172.17.0.4 +... +MY_NODE_NAME=minikube +... +MY_POD_NAME=dapi-envars-fieldref +``` + +## 컨테이너 필드를 환경 변수의 값으로 사용하기 + +이전 연습에서는 파드 필드를 환경 변수의 값으로 사용했다. 이 다음 연습에서는 컨테이너 필드를 +환경 변수의 값으로 사용한다. 다음은 하나의 컨테이너가 있는 파드의 구성 파일이다. + +{{< codenew file="pods/inject/dapi-envars-container.yaml" >}} + +구성 파일에서 4개의 환경 변수를 확인할 수 있다. `env` 필드는 +[EnvVars](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core)의 배열이다. 배열의 첫 번째 요소는 `MY_CPU_REQUEST` 환경 변수가 `test-container`라는 컨테이너의 +`requests.cpu` 필드에서 값을 가져오도록 지정한다. 마찬가지로 다른 환경 변수도 컨테이너 필드에서 +값을 가져온다. + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-container.yaml +``` + +파드의 컨테이너가 실행중인지 확인한다. + +```shell +kubectl get pods +``` + +컨테이너의 로그를 본다. + +```shell +kubectl logs dapi-envars-resourcefieldref +``` + +출력은 선택된 환경 변수의 값을 보여준다. + +``` +1 +1 +33554432 +67108864 +``` + + + +## {{% heading "whatsnext" %}} + + +* [컨테이너를 위한 환경 변수 정의하기](/docs/tasks/inject-data-application/define-environment-variable-container/) +* [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core) +* [컨테이너](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) +* [EnvVar](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core) +* [EnvVarSource](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvarsource-v1-core) +* [ObjectFieldSelector](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#objectfieldselector-v1-core) +* [ResourceFieldSelector](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcefieldselector-v1-core) + + + diff --git a/content/ko/examples/pods/inject/dapi-envars-container.yaml b/content/ko/examples/pods/inject/dapi-envars-container.yaml new file mode 100644 index 0000000000..55bd4dd263 --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-envars-container.yaml @@ -0,0 +1,45 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-resourcefieldref +spec: + containers: + - name: test-container + image: k8s.gcr.io/busybox:1.24 + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_CPU_REQUEST MY_CPU_LIMIT; + printenv MY_MEM_REQUEST MY_MEM_LIMIT; + sleep 10; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + env: + - name: MY_CPU_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.cpu + - name: MY_CPU_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.cpu + - name: MY_MEM_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.memory + - name: MY_MEM_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.memory + restartPolicy: Never diff --git a/content/ko/examples/pods/inject/dapi-envars-pod.yaml b/content/ko/examples/pods/inject/dapi-envars-pod.yaml new file mode 100644 index 0000000000..071fa82bb3 --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-envars-pod.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-fieldref +spec: + containers: + - name: test-container + image: k8s.gcr.io/busybox + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_NODE_NAME MY_POD_NAME MY_POD_NAMESPACE; + printenv MY_POD_IP MY_POD_SERVICE_ACCOUNT; + sleep 10; + done; + env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: MY_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: MY_POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: MY_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: MY_POD_SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName + restartPolicy: Never From 2e55433aec3a83c28b804ffc70bead5da69ee694 Mon Sep 17 00:00:00 2001 From: bhumijgupta Date: Wed, 8 Sep 2021 19:01:08 +0530 Subject: [PATCH 025/249] Fix release cadence in deprecation policy page Signed-off-by: bhumijgupta --- content/en/docs/reference/using-api/deprecation-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/using-api/deprecation-policy.md b/content/en/docs/reference/using-api/deprecation-policy.md index eed4341628..d3677b031e 100644 --- a/content/en/docs/reference/using-api/deprecation-policy.md +++ b/content/en/docs/reference/using-api/deprecation-policy.md @@ -109,7 +109,7 @@ objects. All of this is best illustrated by examples. Imagine a Kubernetes release, version X, which introduces a new API group. A new Kubernetes release is made -every approximately 3 months (4 per year). The following table describes which +every approximately 4 months (3 per year). The following table describes which API versions are supported in a series of subsequent releases. From d3dcda21a5a14da0cec7495c17075af0892bbfc9 Mon Sep 17 00:00:00 2001 From: bhumijgupta Date: Wed, 8 Sep 2021 19:05:01 +0530 Subject: [PATCH 026/249] Revert change to double spacing Signed-off-by: bhumijgupta --- content/en/docs/reference/using-api/deprecation-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/using-api/deprecation-policy.md b/content/en/docs/reference/using-api/deprecation-policy.md index d3677b031e..6bbd9be9e1 100644 --- a/content/en/docs/reference/using-api/deprecation-policy.md +++ b/content/en/docs/reference/using-api/deprecation-policy.md @@ -109,7 +109,7 @@ objects. All of this is best illustrated by examples. Imagine a Kubernetes release, version X, which introduces a new API group. A new Kubernetes release is made -every approximately 4 months (3 per year). The following table describes which +every approximately 4 months (3 per year). The following table describes which API versions are supported in a series of subsequent releases.
From 1e4a57b563e8b169ef4ca736608174b41b1ac7be Mon Sep 17 00:00:00 2001 From: likakuli <1154584512@qq.com> Date: Thu, 9 Sep 2021 19:18:57 +0800 Subject: [PATCH 027/249] Update nodelocaldns.md use ",_PILLAR__DNS__SERVER_" as old pattern instead of "_PILLAR__DNS__SERVER_" when use ipvs mode. In English localization, the pattern is ",_PILLAR__DNS__SERVER_" but when in zh localization, the comma lost. --- content/ja/docs/tasks/administer-cluster/nodelocaldns.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/tasks/administer-cluster/nodelocaldns.md b/content/ja/docs/tasks/administer-cluster/nodelocaldns.md index 5b7b9e58e8..4a5e59a2ca 100644 --- a/content/ja/docs/tasks/administer-cluster/nodelocaldns.md +++ b/content/ja/docs/tasks/administer-cluster/nodelocaldns.md @@ -73,7 +73,7 @@ NodeLocal DNSキャッシュは、クラスターノード上でDNSキャッシ * kube-proxyがIPVSモードで稼働中のとき: ``` bash - sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml + sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml ``` このモードでは、node-local-dns Podは``上のみで待ち受けます。node-local-dnsのインターフェースはkube-dnsのクラスターIPをバインドしません。なぜならばIPVSロードバランシング用に使われているインターフェースは既にこのアドレスを使用しているためです。 `__PILLAR__UPSTREAM__SERVERS__` はnode-local-dns Podにより生成されます。 From 309065ce9b1a022bd0cdb91c9b05022deafb1cdb Mon Sep 17 00:00:00 2001 From: ixodie Date: Mon, 13 Sep 2021 16:42:43 -0400 Subject: [PATCH 028/249] Removed Contiv.io Project did not have direct integration with k8s. URL to project info goes to malicious browser plugin install and ad-based squatter on contiv.io. --- content/en/docs/concepts/cluster-administration/networking.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md index c517b13175..424b889078 100644 --- a/content/en/docs/concepts/cluster-administration/networking.md +++ b/content/en/docs/concepts/cluster-administration/networking.md @@ -164,10 +164,6 @@ network complexity required to deploy Kubernetes at scale within AWS. [Coil](https://github.com/cybozu-go/coil) is a CNI plugin designed for ease of integration, providing flexible egress networking. Coil operates with a low overhead compared to bare metal, and allows you to define arbitrary egress NAT gateways for external networks. -### Contiv - -[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](https://contiv.io) is all open sourced. - ### Contrail / Tungsten Fabric [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is a truly open, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide different isolation modes for virtual machines, containers/pods and bare metal workloads. From 0506656da1859035fd23b7145ab27392397ddb48 Mon Sep 17 00:00:00 2001 From: Arhell Date: Wed, 15 Sep 2021 00:17:28 +0300 Subject: [PATCH 029/249] [ja] Improvement: Runtime Class --- content/ja/docs/concepts/containers/runtime-class.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/concepts/containers/runtime-class.md b/content/ja/docs/concepts/containers/runtime-class.md index bc4e285c66..15cf7446f0 100644 --- a/content/ja/docs/concepts/containers/runtime-class.md +++ b/content/ja/docs/concepts/containers/runtime-class.md @@ -100,7 +100,7 @@ Kubernetesのビルトインのdockershim CRIは、ランタイムハンドラ 正しいハンドラーは、その`runtime`セクションで設定されます。 ``` -[plugins.cri.containerd.runtimes.${HANDLER_NAME}] +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}] ``` containerdの設定に関する詳細なドキュメントは下記を参照してください。 From 4b2661e932d8a55d7c70dd095306fe70944c3c8a Mon Sep 17 00:00:00 2001 From: Pushkar Joglekar <3390906+PushkarJ@users.noreply.github.com> Date: Mon, 13 Sep 2021 13:28:50 -0700 Subject: [PATCH 030/249] Replace k/security with k/committee-security-response Co-authored-by: Rey Lejano --- content/en/releases/release-managers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/releases/release-managers.md b/content/en/releases/release-managers.md index 0b68ca0b87..e8b75492bc 100644 --- a/content/en/releases/release-managers.md +++ b/content/en/releases/release-managers.md @@ -31,7 +31,7 @@ The responsibilities of each role are described below. ### Security Embargo Policy -Some information about releases is subject to embargo and we have defined policy about how those embargos are set. Please refer [Security Embargo Policy](https://github.com/kubernetes/security/blob/master/private-distributors-list.md#embargo-policy) here for more information. +Some information about releases is subject to embargo and we have defined policy about how those embargoes are set. Please refer to the [Security Embargo Policy](https://github.com/kubernetes/committee-security-response/blob/master/private-distributors-list.md#embargo-policy) for more information. ## Handbooks From 49d7b493bbf641d0764a28d26f9466eaaea13124 Mon Sep 17 00:00:00 2001 From: Andrey Regentov Date: Wed, 15 Sep 2021 15:46:26 +0700 Subject: [PATCH 031/249] semantic fix use "latest" => "Always" use nothing => "Always" use any but "latest" => "IfNotPresent" --- content/en/docs/concepts/containers/images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/containers/images.md b/content/en/docs/concepts/containers/images.md index 9300561e46..5dfd349f80 100644 --- a/content/en/docs/concepts/containers/images.md +++ b/content/en/docs/concepts/containers/images.md @@ -108,7 +108,7 @@ When you (or a controller) submit a new Pod to the API server, your cluster sets `:latest`, `imagePullPolicy` is automatically set to `Always`; - if you omit the `imagePullPolicy` field, and you don't specify the tag for the container image, `imagePullPolicy` is automatically set to `Always`; -- if you omit the `imagePullPolicy` field, and you don't specify the tag for the +- if you omit the `imagePullPolicy` field, and you specify the tag for the container image that isn't `:latest`, the `imagePullPolicy` is automatically set to `IfNotPresent`. From 39301f0e1fdb84cfcb69787342d9cdc0d7fa58d8 Mon Sep 17 00:00:00 2001 From: Jihoon Seo Date: Fri, 17 Sep 2021 11:57:57 +0900 Subject: [PATCH 032/249] [ko] Update outdated files in dev-1.22-ko.1 (p1) --- .../tasks/manage-daemon/update-daemon-set.md | 8 ++++-- .../manage-hugepages/scheduling-hugepages.md | 6 ----- .../docs/tasks/network/validate-dual-stack.md | 2 +- .../horizontal-pod-autoscale-walkthrough.md | 2 +- .../horizontal-pod-autoscale.md | 26 +++++++++++-------- .../docs/tasks/tools/install-kubectl-macos.md | 2 +- .../ko/docs/tutorials/clusters/apparmor.md | 5 ++++ .../create-cluster/cluster-interactive.html | 2 +- .../deploy-app/deploy-interactive.html | 4 +-- .../explore/explore-interactive.html | 4 +-- .../expose/expose-interactive.html | 4 +-- .../expose/expose-intro.html | 2 +- .../scale/scale-interactive.html | 4 +-- .../update/update-interactive.html | 2 +- .../stateful-application/cassandra.md | 2 +- .../stateful-application/zookeeper.md | 8 +++--- .../stateless-application/guestbook.md | 2 +- 17 files changed, 46 insertions(+), 39 deletions(-) diff --git a/content/ko/docs/tasks/manage-daemon/update-daemon-set.md b/content/ko/docs/tasks/manage-daemon/update-daemon-set.md index 50a3a6ad2b..a3575704ae 100644 --- a/content/ko/docs/tasks/manage-daemon/update-daemon-set.md +++ b/content/ko/docs/tasks/manage-daemon/update-daemon-set.md @@ -11,6 +11,8 @@ weight: 10 ## {{% heading "prerequisites" %}} +{{< include "task-tutorial-prereqs.md" >}} + ## 데몬셋 업데이트 전략 @@ -33,9 +35,11 @@ weight: 10 `.spec.updateStrategy.type` 에 `RollingUpdate` 를 설정해야 한다. [`.spec.updateStrategy.rollingUpdate.maxUnavailable`](/ko/docs/concepts/workloads/controllers/deployment/#최대-불가max-unavailable) -(기본값은 1)과 +(기본값은 1), [`.spec.minReadySeconds`](/ko/docs/concepts/workloads/controllers/deployment/#최소-대기-시간초) -(기본값은 0)으로 +(기본값은 0), +[`.spec.maxSurge`](/ko/docs/concepts/workloads/controllers/deployment/#최대-서지-max-surge) +(베타 기능, 기본값은 25%)를 설정할 수도 있다. ### `RollingUpdate` 업데이트 전략으로 데몬셋 생성 diff --git a/content/ko/docs/tasks/manage-hugepages/scheduling-hugepages.md b/content/ko/docs/tasks/manage-hugepages/scheduling-hugepages.md index 981efb6719..40ca1a8726 100644 --- a/content/ko/docs/tasks/manage-hugepages/scheduling-hugepages.md +++ b/content/ko/docs/tasks/manage-hugepages/scheduling-hugepages.md @@ -113,9 +113,3 @@ spec: - 네임스페이스에서의 huge page 사용은 `hugepages-` 토큰을 사용하는 `cpu` 또는 `memory` 와 같은 다른 컴퓨트 리소스와 비슷한 리소스쿼터(ResourceQuota)를 통해 제어할 수 있다. -- 다양한 크기의 huge page 지원이 기능 게이트로 제공된다. - {{}} 및 - {{}} - (`--feature-gates=HugePageStorageMediumSize=true`)의 `HugePageStorageMediumSize` - [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 - 사용하여 비활성화할 수 있다. diff --git a/content/ko/docs/tasks/network/validate-dual-stack.md b/content/ko/docs/tasks/network/validate-dual-stack.md index 5a3fdc477e..97753165f1 100644 --- a/content/ko/docs/tasks/network/validate-dual-stack.md +++ b/content/ko/docs/tasks/network/validate-dual-stack.md @@ -16,7 +16,7 @@ content_type: task * 이중 스택 네트워킹을 위한 제공자 지원 (클라우드 제공자 또는 기타 제공자들은 라우팅 가능한 IPv4/IPv6 네트워크 인터페이스를 제공하는 쿠버네티스 노드들을 제공해야 한다.) -* 이중 스택을 지원하는 [네트워크 플러그인](/ko/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) (예. Kubenet 또는 Calico) +* 이중 스택을 지원하는 [네트워크 플러그인](/ko/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) (예: Calico, Cilium 또는 Kubenet) * [이중 스택 활성화](/ko/docs/concepts/services-networking/dual-stack/) 클러스터 {{< version-check >}} diff --git a/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md index 61f1dbc758..a3c6285aac 100644 --- a/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md +++ b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md @@ -183,7 +183,7 @@ CPU 사용량은 0으로 떨어졌고, HPA는 레플리카의 개수를 1로 낮 첫 번째로, `autoscaling/v2beta2` 형식으로 HorizontalPodAutoscaler YAML 파일을 생성한다. ```shell -kubectl get hpa.v2beta2.autoscaling -o yaml > /tmp/hpa-v2.yaml +kubectl get hpa php-apache -o yaml > /tmp/hpa-v2.yaml ``` 에디터로 `/tmp/hpa-v2.yaml` 파일을 열면, 다음과 같은 YAML을 확인할 수 있다. diff --git a/content/ko/docs/tasks/run-application/horizontal-pod-autoscale.md b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale.md index b4cc1b3be5..bc39ab4d91 100644 --- a/content/ko/docs/tasks/run-application/horizontal-pod-autoscale.md +++ b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale.md @@ -1,4 +1,8 @@ --- + + + + title: Horizontal Pod Autoscaler feature: title: Horizontal 스케일링 @@ -9,10 +13,6 @@ content_type: concept weight: 90 --- - - - - Horizontal Pod Autoscaler는 CPU 사용량 @@ -181,6 +181,7 @@ HorizontalPodAutoscaler API 오브젝트 생성시 지정된 이름이 유효한 API 오브젝트에 대한 자세한 내용은 [HorizontalPodAutoscaler 오브젝트](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#horizontalpodautoscaler-v1-autoscaling)에서 찾을 수 있다. + ## kubectl에서 Horizontal Pod Autoscaler 지원 Horizontal Pod Autoscaler는 모든 API 리소스와 마찬가지로 `kubectl`에 의해 표준 방식으로 지원된다. @@ -197,14 +198,17 @@ Horizontal Pod Autoscaler는 모든 API 리소스와 마찬가지로 `kubectl` ## 롤링 업데이트 중 오토스케일링 -현재 쿠버네티스에서는 기본 레플리카셋를 관리하는 디플로이먼트 오브젝트를 사용하여 롤링 업데이트를 수행할 수 있다. -Horizontal Pod Autoscaler는 후자의 방법을 지원한다. Horizontal Pod Autoscaler는 디플로이먼트 오브젝트에 바인딩되고, -디플로이먼트 오브젝트를 위한 크기를 설정하며, 디플로이먼트는 기본 레플리카셋의 크기를 결정한다. +쿠버네티스는 디플로이먼트에 대한 롤링 업데이트를 지원한다. +이 경우, 디플로이먼트가 기저 레플리카셋을 알아서 관리한다. +디플로이먼트에 오토스케일링을 설정하려면, +각 디플로이먼트에 대한 HorizontalPodAutoscaler를 생성한다. +HorizontalPodAutoscaler는 디플로이먼트의 `replicas` 필드를 관리한다. +디플로이먼트 컨트롤러는 기저 레플리카셋에 `replicas` 값을 적용하여 +롤아웃 과정 중/이후에 적절한 숫자까지 늘어나도록 한다. -Horizontal Pod Autoscaler는 레플리케이션 컨트롤러를 직접 조작하는 롤링 업데이트에서 작동하지 않는다. -즉, Horizontal Pod Autoscaler를 레플리케이션 컨트롤러에 바인딩하고 롤링 업데이트를 수행할 수 없다. (예 : `kubectl rolling-update`) -작동하지 않는 이유는 롤링 업데이트에서 새 레플리케이션 컨트롤러를 만들 때, -Horizontal Pod Autoscaler가 새 레플리케이션 컨트롤러에 바인딩되지 않기 때문이다. +오토스케일된 레플리카가 있는 스테이트풀셋의 롤링 업데이트를 수행하면, +스테이트풀셋이 직접 파드의 숫자를 관리한다(즉, +레플리카셋과 같은 중간 리소스가 없다). ## 쿨-다운 / 지연에 대한 지원 diff --git a/content/ko/docs/tasks/tools/install-kubectl-macos.md b/content/ko/docs/tasks/tools/install-kubectl-macos.md index 90fefb0c3a..cd03eb91b7 100644 --- a/content/ko/docs/tasks/tools/install-kubectl-macos.md +++ b/content/ko/docs/tasks/tools/install-kubectl-macos.md @@ -228,7 +228,7 @@ kubectl은 Bash 및 Zsh에 대한 자동 완성 지원을 제공하므로 입력 1. kubectl-convert 바이너리를 시스템 `PATH` 의 파일 위치로 옮긴다. ```bash - sudo mv ./kubectl /usr/local/bin/kubectl-convert + sudo mv ./kubectl-convert /usr/local/bin/kubectl-convert sudo chown root: /usr/local/bin/kubectl-convert ``` diff --git a/content/ko/docs/tutorials/clusters/apparmor.md b/content/ko/docs/tutorials/clusters/apparmor.md index 43b07e293b..7b11ea7722 100644 --- a/content/ko/docs/tutorials/clusters/apparmor.md +++ b/content/ko/docs/tutorials/clusters/apparmor.md @@ -348,6 +348,11 @@ Events: ### PodSecurityPolicy로 프로파일 제한하기 {#restricting-profiles-with-the-podsecuritypolicy} +{{< note >}} +PodSecurityPolicy는 쿠버네티스 v1.21에서 사용 중지되었으며, v1.25에서 제거될 예정이다. +더 자세한 내용은 [PodSecurityPolicy 문서](/ko/docs/concepts/policy/pod-security-policy/)를 참고한다. +{{< /note >}} + 만약 PodSecurityPolicy 확장을 사용하면, 클러스터 단위로 AppArmor 제한을 적용할 수 있다. PodSecurityPolicy를 사용하려면 위해 다음의 플래그를 반드시 `apiserver`에 설정해야 한다. diff --git a/content/ko/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html index fcad9b42b3..7c66225037 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html @@ -26,7 +26,7 @@ weight: 20
홈으로 이동 - 모듈 2로 진행하기 > + 모듈 2로 진행하기 >
diff --git a/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html index ce5be2cfc0..d5aca028e7 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html @@ -37,9 +37,9 @@ weight: 20
- < 모듈 1로 돌아가기 + < 모듈 1로 돌아가기 홈으로 이동 - 모듈 3으로 진행하기 > + 모듈 3으로 진행하기 >
diff --git a/content/ko/docs/tutorials/kubernetes-basics/explore/explore-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/explore/explore-interactive.html index e3b67a1dae..d467fc9b2c 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/explore/explore-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/explore/explore-interactive.html @@ -29,9 +29,9 @@ weight: 20
- < 모듈 2로 돌아가기 + < 모듈 2로 돌아가기 홈으로 이동 - 모듈 4로 진행하기 > + 모듈 4로 진행하기 >
diff --git a/content/ko/docs/tutorials/kubernetes-basics/expose/expose-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/expose/expose-interactive.html index 09dde78cb8..fa68a4c40e 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/expose/expose-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/expose/expose-interactive.html @@ -26,9 +26,9 @@ weight: 20
- < 모듈 3으로 돌아가기 + < 모듈 3으로 돌아가기 홈으로 이동 - 모듈 5로 진행하기 > + 모듈 5로 진행하기 >
diff --git a/content/ko/docs/tutorials/kubernetes-basics/expose/expose-intro.html b/content/ko/docs/tutorials/kubernetes-basics/expose/expose-intro.html index aac6298a7a..8dbfde0c63 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/expose/expose-intro.html +++ b/content/ko/docs/tutorials/kubernetes-basics/expose/expose-intro.html @@ -37,7 +37,7 @@ weight: 10
  • ClusterIP (기본값) - 클러스터 내에서 내부 IP 에 대해 서비스를 노출해준다. 이 방식은 오직 클러스터 내에서만 서비스가 접근될 수 있도록 해준다.
  • NodePort - NAT가 이용되는 클러스터 내에서 각각 선택된 노드들의 동일한 포트에 서비스를 노출시켜준다. <NodeIP>:<NodePort>를 이용하여 클러스터 외부로부터 서비스가 접근할 수 있도록 해준다. ClusterIP의 상위 집합이다.
  • LoadBalancer - (지원 가능한 경우) 기존 클라우드에서 외부용 로드밸런서를 생성하고 서비스에 고정된 공인 IP를 할당해준다. NodePort의 상위 집합이다.
    • -
  • ExternalName - CNAME 레코드 및 값을 반환함으로써 서비스를 externalName 필드의 내용(예를 들면, `foo.bar.example.com`)에 매핑한다. 어떠한 종류의 프록시도 설정되지 않는다. 이 방식은 kube-dns v1.7 이상 또는 CoreDNS 버전 0.0.8 이상을 필요로 한다.
    • +
  • ExternalName - CNAME 레코드 및 값을 반환함으로써 서비스를 externalName 필드의 내용(예를 들면, foo.bar.example.com)에 매핑한다. 어떠한 종류의 프록시도 설정되지 않는다. 이 방식은 kube-dns v1.7 이상 또는 CoreDNS 버전 0.0.8 이상을 필요로 한다.

    • 다른 서비스 타입들에 대한 추가 정보는 소스 IP 이용하기 튜토리얼에서 확인 가능하다. 또한 서비스들로 애플리케이션에 접속하기도 참고해 보자.

    부가적으로, spec에 selector를 정의하지 않고 말아넣은 서비스들의 몇 가지 유즈케이스들이 있음을 주의하자. selector 없이 생성된 서비스는 상응하는 엔드포인트 오브젝트들 또한 생성하지 않는다. 이로써 사용자들로 하여금 하나의 서비스를 특정한 엔드포인트에 매핑 시킬수 있도록 해준다. selector를 생략하게 되는 또 다른 가능성은 여러분이 type: ExternalName을 이용하겠다고 확고하게 의도하는 경우이다.

    diff --git a/content/ko/docs/tutorials/kubernetes-basics/scale/scale-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/scale/scale-interactive.html index 22b5d41342..18a371fff2 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/scale/scale-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/scale/scale-interactive.html @@ -26,9 +26,9 @@ weight: 20
    - < 모듈 4로 돌아가기 + < 모듈 4로 돌아가기 홈으로 이동 - 모듈 6으로 진행하기 > + 모듈 6으로 진행하기 >
    diff --git a/content/ko/docs/tutorials/kubernetes-basics/update/update-interactive.html b/content/ko/docs/tutorials/kubernetes-basics/update/update-interactive.html index 4038e3b358..0f25073ca3 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/update/update-interactive.html +++ b/content/ko/docs/tutorials/kubernetes-basics/update/update-interactive.html @@ -26,7 +26,7 @@ weight: 20
    - < 모듈 5로 돌아가기 + < 모듈 5로 돌아가기 쿠버네티스 기초로 돌아가기
    diff --git a/content/ko/docs/tutorials/stateful-application/cassandra.md b/content/ko/docs/tutorials/stateful-application/cassandra.md index 3ebb7ec387..4e82a6d07f 100644 --- a/content/ko/docs/tutorials/stateful-application/cassandra.md +++ b/content/ko/docs/tutorials/stateful-application/cassandra.md @@ -50,7 +50,7 @@ weight: 30 ### 추가적인 Minikube 설정 요령 {{< caution >}} -[Minikube](https://minikube.sigs.k8s.io/docs/)는 1024MiB 메모리와 1개 CPU가 기본 설정이다. +[Minikube](https://minikube.sigs.k8s.io/docs/)는 2048MB 메모리와 2개 CPU가 기본 설정이다. 이 튜토리얼에서 Minikube를 기본 리소스 설정으로 실행하면 리소스 부족 오류가 발생한다. 이런 오류를 피하려면 Minikube를 다음 설정으로 실행하자. diff --git a/content/ko/docs/tutorials/stateful-application/zookeeper.md b/content/ko/docs/tutorials/stateful-application/zookeeper.md index 3fca0a6749..5ddfa51f42 100644 --- a/content/ko/docs/tutorials/stateful-application/zookeeper.md +++ b/content/ko/docs/tutorials/stateful-application/zookeeper.md @@ -929,7 +929,7 @@ kubernetes-node-i4c4 [`kubectl drain`](/docs/reference/generated/kubectl/kubectl-commands/#drain)를 이용하자. ```shell -kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data +kubectl drain $(kubectl get pod zk-0 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` ``` @@ -964,7 +964,7 @@ zk-0 1/1 Running 0 1m `zk-1` 이 스케줄된 노드를 비워보자. ```shell -kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data "kubernetes-node-ixsl" cordoned +kubectl drain $(kubectl get pod zk-1 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data "kubernetes-node-ixsl" cordoned ``` ``` @@ -1007,7 +1007,7 @@ zk-1 0/1 Pending 0 0s `zk-2`가 스케줄된 노드를 비워보자. ```shell -kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data +kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` ``` @@ -1094,7 +1094,7 @@ zk-1 1/1 Running 0 13m `zk-2`가 스케줄된 노드를 비워보자. ```shell -kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-local-data +kubectl drain $(kubectl get pod zk-2 --template {{.spec.nodeName}}) --ignore-daemonsets --force --delete-emptydir-data ``` 출력은 diff --git a/content/ko/docs/tutorials/stateless-application/guestbook.md b/content/ko/docs/tutorials/stateless-application/guestbook.md index 1a5e4a6079..9f6481396e 100644 --- a/content/ko/docs/tutorials/stateless-application/guestbook.md +++ b/content/ko/docs/tutorials/stateless-application/guestbook.md @@ -19,7 +19,7 @@ _(운영 수준이 아닌)_ 멀티 티어 웹 애플리케이션을 빌드하고 이 예제는 다음과 같은 구성으로 이루어져 있다. -* 방명록 항목을 저장하기 위한 단일 인스턴스 [Redis](https://www.redis.com/) +* 방명록 항목을 저장하기 위한 단일 인스턴스 [Redis](https://www.redis.io/) * 여러 개의 웹 프론트엔드 인스턴스 ## {{% heading "objectives" %}} From 178bfdb879c59984bbffb924b7105610ad175420 Mon Sep 17 00:00:00 2001 From: Hanchul Kwon Date: Mon, 30 Aug 2021 10:47:58 +0900 Subject: [PATCH 033/249] [ko] Translate /concepts/storage/storage-capacity.md in Korean Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com> Co-authored-by: Seokho Son --- .../docs/concepts/storage/storage-capacity.md | 117 ++++++++++++++++++ content/ko/docs/reference/glossary/csi.md | 21 ++++ 2 files changed, 138 insertions(+) create mode 100644 content/ko/docs/concepts/storage/storage-capacity.md create mode 100644 content/ko/docs/reference/glossary/csi.md diff --git a/content/ko/docs/concepts/storage/storage-capacity.md b/content/ko/docs/concepts/storage/storage-capacity.md new file mode 100644 index 0000000000..4aeb1ba8c1 --- /dev/null +++ b/content/ko/docs/concepts/storage/storage-capacity.md @@ -0,0 +1,117 @@ +--- + + + + + + +title: 스토리지 용량 +content_type: concept +weight: 45 +--- + + + +스토리지 용량은 제한이 있으며, 파드가 실행되는 노드의 상황에 따라 달라질 수 있다. +예를 들어, 일부 노드에서 NAS(Network Attached Storage)에 접근할 수 없는 경우가 있을 수 있으며, +또는 각 노드에 종속적인 로컬 스토리지를 사용하는 경우일 수도 있다. + +{{< feature-state for_k8s_version="v1.19" state="alpha" >}} +{{< feature-state for_k8s_version="v1.21" state="beta" >}} + +이 페이지에서는 쿠버네티스가 어떻게 스토리지 용량을 추적하고 +스케줄러가 남아 있는 볼륨을 제공하기 위해 스토리지 용량이 충분한 노드에 +파드를 스케줄링하기 위해 이 정보를 어떻게 사용하는지 설명한다. +스토리지 용량을 추적하지 않으면, 스케줄러는 +볼륨을 제공할 충분한 용량이 없는 노드를 선정할 수 있으며, +스케줄링을 여러 번 다시 시도해야 한다. + +스토리지 용량 추적은 {{< glossary_tooltip +text="컨테이너 스토리지 인터페이스(CSI)" term_id="csi" >}} 드라이버에서 지원하며, +CSI 드라이버를 설치할 때 [사용하도록 설정](#스토리지-용량-추적-활성화)해야 한다. + + + +## API + + 이 기능에는 다음 두 가지 API 확장이 있다. +- CSIStorageCapacity 오브젝트: + CSI 드라이버가 설치된 네임스페이스에 + CSI 드라이버가 이 오브젝트를 생성한다. 각 오브젝트는 + 하나의 스토리지 클래스에 대한 용량 정보를 담고 있으며, + 어떤 노드가 해당 스토리지에 접근할 수 있는지를 정의한다. +- [ `CSIDriverSpec.StorageCapacity` 필드](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#csidriverspec-v1-storage-k8s-io): + `true`로 설정하면, 쿠버네티스 스케줄러가 + CSI 드라이버를 사용하는 볼륨의 스토리지 용량을 고려하게 된다. + +## 스케줄링 + +다음과 같은 경우 쿠버네티스 스케줄러에서 스토리지 용량 정보를 사용한다. +- `CSIStorageCapacity` 기능 게이트(feature gate)가 true이고, +- 파드가 아직 생성되지 않은 볼륨을 사용하고, +- 해당 볼륨은 CSI 드라이버를 참조하고 + `WaitForFirstConsumer` + [볼륨 바인딩 모드](/ko/docs/concepts/storage/storage-classes/#볼륨-바인딩-모드)를 사용하는 + {{< glossary_tooltip text="스토리지클래스(StorageClass)" term_id="storage-class" >}}를 사용하고, +- 드라이버의 `CSIDriver` 오브젝트에 `StorageCapacity` 속성이 + true로 설정되어 있다. + +이 경우 스케줄러는 파드에 제공할 +충분한 스토리지가 있는 노드만 고려한다. +이 검사는 아주 간단한데, +볼륨의 크기를 노드를 포함하는 토폴로지를 가진 `CSIStorageCapacity` 오브젝트에 +나열된 용량과 비교한다. + +볼륨 바인딩 모드가 `Immediate` 인 볼륨의 경우에는 스토리지 드라이버는 +볼륨을 사용하는 파드와 관계없이 볼륨을 생성할 위치를 정한다. +볼륨을 생성한 후에, 스케줄러는 +볼륨을 사용할 수 있는 노드에 파드를 스케줄링한다. + +[CSI 임시 볼륨](/ko/docs/concepts/storage/volumes/#csi)의 경우에는 +볼륨 유형이 로컬 볼륨이고 +큰 자원이 필요하지 않은 특정 CSI 드라이버에서만 사용된다는 가정하에, +항상 스토리지 용량을 고려하지 않고 +스케줄링한다. + +## 리스케줄링 + +`WaitForFirstConsumer` 볼륨을 가진 파드에 대해 +노드가 선정되었더라도 아직은 잠정적인 결정이다. 다음 단계에서 +선정한 노드에서 볼륨을 사용할 수 있어야 한다는 힌트를 주고 +CSI 스토리지 드라이버에 볼륨 생성을 요청한다 + +쿠버네티스는 시간이 지난 스토리지 용량 정보를 기반으로 +노드를 선정할 수도 있으므로, 볼륨을 실제로 생성하지 않을 수도 있다. +그런 다음 노드 선정이 재설정되고 쿠버네티스 스케줄러가 +파드를 위한 노드를 찾는 것을 재시도한다. + +## 제한사항 + +스토리지 용량 추적은 첫 시도에 스케줄링이 성공할 가능성을 높이지만, +스케줄러가 시간이 지난 정보를 기반으로 +결정해야 할 수도 있기 때문에 이를 보장하지는 않는다. +일반적으로 스토리지 용량 정보가 없는 스케줄링과 +동일한 재시도 메커니즘으로 스케줄링 실패를 처리한다. + +스케줄링이 영구적으로 실패할 수 있는 한 가지 상황은 +파드가 여러 볼륨을 사용하는 경우이다. +토폴로지 세그먼트에 하나의 볼륨이 이미 생성되어 +다른 볼륨에 충분한 용량이 남아 있지 않을 수 있다. +이러한 상황을 복구하려면 +용량을 늘리거나 이미 생성된 볼륨을 삭제하는 등의 수작업이 필요하며, +자동으로 처리하려면 +[추가 작업](https://github.com/kubernetes/enhancements/pull/1703)이 필요하다. + +## 스토리지 용량 추적 활성화 + +스토리지 용량 추적은 베타 기능이며, +쿠버네티스 1.21 이후 버전부터 쿠버네티스 클러스터에 기본적으로 활성화되어 있다. +클러스터에서 스토리지 용량 추적 기능을 활성화하는 것뿐만 아니라, CSI 드라이버에서도 이 기능을 지원해야 한다. +자세한 내용은 드라이버 문서를 참조한다. + +## {{% heading "whatsnext" %}} + +- 설계에 대한 자세한 내용은 + [파드 스케줄링 스토리지 용량 제약 조건](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1472-storage-capacity-tracking/README.md)을 참조한다. +- 이 기능의 추가 개발에 대한 자세한 내용은 [개선 추적 이슈 #1472](https://github.com/kubernetes/enhancements/issues/1472)를 참조한다. +- [쿠버네티스 스케줄러](/ko/docs/concepts/scheduling-eviction/kube-scheduler/)에 대해 살펴본다. diff --git a/content/ko/docs/reference/glossary/csi.md b/content/ko/docs/reference/glossary/csi.md new file mode 100644 index 0000000000..64053db20b --- /dev/null +++ b/content/ko/docs/reference/glossary/csi.md @@ -0,0 +1,21 @@ +--- +title: 컨테이너 스토리지 인터페이스(CSI) +id: csi +date: 2018-06-25 +full_link: /ko/docs/concepts/storage/volumes/#csi +short_description: > + 컨테이너 스토리지 인터페이스(CSI)는 컨테이너에 스토리지 시스템을 노출하는 표준 인터페이스를 정의한다. + + +aka: +tags: +- storage +--- + 컨테이너 스토리지 인터페이스(CSI)는 컨테이너에 스토리지 시스템을 노출하는 표준 인터페이스를 정의한다. + + + +CSI를 통해 공급업체는 쿠버네티스 저장소(트리 외 플러그인)를 추가하지 않고도 쿠버네티스용 사용자 스토리지 플러그인을 생성할 수 있다. 스토리지 제공자가 CSI 드라이버를 사용하려면, 먼저 [클러스터에 배포](https://kubernetes-csi.github.io/docs/deploying.html)해야 한다. 그런 다음 해당 CSI 드라이버를 사용하는 {{< glossary_tooltip text="스토리지클래스(StorageClass)" term_id="storage-class" >}}를 생성할 수 있다. + +* [쿠버네티스 문서에서 CSI](/ko/docs/concepts/storage/volumes/#csi) +* [사용 가능한 CSI 드라이버 목록](https://kubernetes-csi.github.io/docs/drivers.html) From cb9c4853ac90e48e6f8a8bbcfb88342d9fb32302 Mon Sep 17 00:00:00 2001 From: Mauren Berti Date: Tue, 24 Aug 2021 17:21:26 -0400 Subject: [PATCH 034/249] Translate ConfigMap page to Brazilian Portuguese. --- .../docs/concepts/configuration/configmap.md | 280 ++++++++++++++++++ .../pt-br/docs/reference/glossary/addons.md | 17 ++ .../docs/reference/glossary/configmap.md | 18 ++ .../pt-br/docs/reference/glossary/image.md | 17 ++ .../pt-br/docs/reference/glossary/index.md | 11 + .../pt-br/docs/reference/glossary/kubelet.md | 2 +- .../docs/reference/glossary/namespace.md | 18 ++ .../reference/glossary/operator-pattern.md | 24 ++ content/pt-br/docs/reference/glossary/pod.md | 2 +- .../pt-br/docs/reference/glossary/secret.md | 18 ++ .../pt-br/docs/reference/glossary/selector.md | 17 ++ .../pt-br/docs/reference/glossary/service.md | 20 ++ .../docs/reference/glossary/static-pod.md | 19 ++ .../pt-br/docs/reference/glossary/volume.md | 20 ++ 14 files changed, 481 insertions(+), 2 deletions(-) create mode 100644 content/pt-br/docs/concepts/configuration/configmap.md create mode 100644 content/pt-br/docs/reference/glossary/addons.md create mode 100644 content/pt-br/docs/reference/glossary/configmap.md create mode 100644 content/pt-br/docs/reference/glossary/image.md create mode 100644 content/pt-br/docs/reference/glossary/index.md create mode 100644 content/pt-br/docs/reference/glossary/namespace.md create mode 100644 content/pt-br/docs/reference/glossary/operator-pattern.md create mode 100644 content/pt-br/docs/reference/glossary/secret.md create mode 100644 content/pt-br/docs/reference/glossary/selector.md create mode 100644 content/pt-br/docs/reference/glossary/service.md create mode 100644 content/pt-br/docs/reference/glossary/static-pod.md create mode 100644 content/pt-br/docs/reference/glossary/volume.md diff --git a/content/pt-br/docs/concepts/configuration/configmap.md b/content/pt-br/docs/concepts/configuration/configmap.md new file mode 100644 index 0000000000..f4038706ea --- /dev/null +++ b/content/pt-br/docs/concepts/configuration/configmap.md @@ -0,0 +1,280 @@ +--- +title: ConfigMaps +content_type: concept +weight: 20 +--- + + + +{{< glossary_definition term_id="configmap" prepend="Um ConfigMap é" length="all" >}} + +{{< caution >}} +O ConfigMap não oferece confidencialidade ou encriptação. +Se os dados que você deseja armazenar são confidenciais, utilize +{{< glossary_tooltip term_id="secret" >}} ao invés de um ConfigMap, +ou utilize ferramentas adicionais (de terceiros) para manter seus dados privados. +{{< /caution >}} + + + +## Motivação + +Utilize um ConfigMap para manter a configuração separada do código da aplicação. + +Por exemplo, imagine que você esteja desenvolvendo uma aplicação que pode ser executada +no seu computador local (para desenvolvimento) e na nuvem (para manipular tráfego real). +Você escreve código para ler a variável de ambiente chamada `DATABASE_HOST`. +No seu ambiente local, você configura essa variável com o valor `localhost`. Na nuvem, você +configura essa variável para referenciar um {{< glossary_tooltip text="serviço" term_id="service" >}} +do Kubernetes que expõe o componente do banco de dados ao seu cluster. +Isto permite que você baixe uma imagem de contêiner que roda na nuvem e depure o exato +mesmo código localmente se necessário. + +Um ConfigMap não foi planejado para conter grandes quantidades de dados. Os dados armazenados +em um ConfigMap não podem exceder 1 MiB. Se você precisa armazenar configurações que são maiores +que este limite, considere montar um volume ou utilizar um serviço separado de banco de dados +ou de arquivamento de dados. + +## Objeto ConfigMap + +Um ConfigMap é um [objeto](/docs/concepts/overview/working-with-objects/kubernetes-objects/) +da API que permite o armazenamento de configurações para consumo por outros objetos. Diferentemente +de outros objetos do Kubernetes que contém um campo `spec`, o ConfigMap contém os campos `data` e +`binaryData`. Estes campos aceitam pares chave-valor como valores. Ambos os campos `data` e `binaryData` +são opcionais. O campo `data` foi pensado para conter sequências de bytes UTF-8, enquanto o campo `binaryData` +foi planejado para conter dados binários em forma de strings codificadas em base64. + +É obrigatório que o nome de um ConfigMap seja um +[subdomínio DNS válido](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). + +Cada chave sob as seções `data` ou `binaryData` pode conter quaisquer caracteres alfanuméricos, +`-`, `_` e `.`. As chaves armazenadas na seção `data` não podem colidir com as chaves armazenadas +na seção `binaryData`. + +A partir da versão v1.19 do Kubernetes, é possível adicionar o campo `immutable` a uma definição de ConfigMap +para criar um [ConfigMap imutável](#configmap-immutable). + +## ConfigMaps e Pods + +Você pode escrever uma `spec` para um Pod que se refere a um ConfigMap e configurar o(s) contêiner(es) +neste Pod baseados em dados do ConfigMap. O Pod e o ConfigMap devem estar no mesmo +{{< glossary_tooltip text="namespace" term_id="namespace" >}}. + +{{< note >}} +A `spec` de um {{< glossary_tooltip text="Pod estático" term_id="static-pod" >}} não pode se referir a um +ConfigMap ou a quaisquer outros objetos da API. +{{< /note >}} + +Exemplo de um ConfigMap que contém algumas chaves com valores avulsos e outras chaves com valores semelhantes +a fragmentos de arquivos de configuração: +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: game-demo +data: + # chaves com valores de propriedades; cada chave mapeia para um valor avulso + player_initial_lives: "3" + ui_properties_file_name: "user-interface.properties" + + # chaves semelhantes a fragmentos de arquivos + game.properties: | + enemy.types=aliens,monsters + player.maximum-lives=5 + user-interface.properties: | + color.good=purple + color.bad=yellow + allow.textmode=true +``` + +Existem quatro formas diferentes para consumo de um ConfigMap na configuração de um +contêiner dentro de um Pod: + +1. Dentro de um comando de contêiner e seus argumentos. +1. Variáveis de ambiente para um contêiner. +1. Criando um arquivo em um volume somente leitura, para consumo pela aplicação. +1. Escrevendo código para execução dentro do Pod que utilize a API do Kubernetes para ler um ConfigMap. + +Os diferentes métodos de consumo oferecem diferentes formas de modelar os dados sendo consumidos. +Para os três primeiros métodos, o {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} utiliza +os dados de um ConfigMap quando o(s) contêiner(es) do Pod são inicializados. + +O quarto método envolve escrita de código para leitura do ConfigMap e dos seus dados. No entanto, +como a API do Kubernetes está sendo utilizada diretamente, a aplicação pode solicitar atualizações +sempre que o ConfigMap for alterado e reagir quando isso ocorre. Acessar a API do Kubernetes +diretamente também permite ler ConfigMaps em outros namespaces. + +Exemplo de um Pod que utiliza valores do ConfigMap `game-demo` para configurar um Pod: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: configmap-demo-pod +spec: + containers: + - name: demo + image: alpine + command: ["sleep", "3600"] + env: + # Define as variáveis de ambiente + - name: PLAYER_INITIAL_LIVES # Note que aqui a variável está definida em caixa alta, + # diferente da chave no ConfigMap. + valueFrom: + configMapKeyRef: + name: game-demo # O ConfigMap de onde esse valor vem. + key: player_initial_lives # A chave que deve ser buscada. + - name: UI_PROPERTIES_FILE_NAME + valueFrom: + configMapKeyRef: + name: game-demo + key: ui_properties_file_name + volumeMounts: + - name: config + mountPath: "/config" + readOnly: true + volumes: + # Volumes são definidos no escopo do Pod, e os pontos de montagem são definidos + # nos contêineres dentro dos pods. + - name: config + configMap: + # Informe o nome do ConfigMap que deseja montar. + name: game-demo + # Uma lista de chaves do ConfigMap para serem criadas como arquivos. + items: + - key: "game.properties" + path: "game.properties" + - key: "user-interface.properties" + path: "user-interface.properties" +``` + +ConfigMaps não diferenciam entre propriedades com valores simples ou valores complexos, +que ocupam várias linhas. O importante é a forma que Pods e outros objetos consumem tais valores. + +Para este exemplo, definir um volume e montar ele dentro do container `demo` no caminho `/config` +cria dois arquivos: `/config/game.properties` e `/config/user-interface.properties`, embora existam +quatro chaves distintas no ConfigMap. Isso se deve ao fato de que a definição do Pod contém uma lista +`items` na seção `volumes`. +Se a lista `items` for omitida, cada chave do ConfigMap torna-se um arquivo cujo nome é a sua chave +correspondente, e quatro arquivos serão criados. + +## Usando ConfigMaps + +ConfigMaps podem ser montados como volumes de dados. ConfigMaps também podem ser utilizados +por outras partes do sistema sem serem diretamente expostos ao Pod. Por exemplo, ConfigMaps +podem conter dados que outras partes do sistema devem usar para configuração. + +A forma mais comum de utilização de ConfigMaps é a configuração de contêineres executando em +Pods no mesmo namespace. Você também pode utilizar um ConfigMap separadamente. + +Por exemplo, existem {{< glossary_tooltip text="complementos" term_id="addons" >}} ou +{{< glossary_tooltip text="operadores" term_id="operator-pattern" >}} que adaptam seus comportamentos +de acordo com dados de um ConfigMap. + +### Utilizando ConfigMaps como arquivos em um Pod + +Para consumir um ConfigMap em um volume em um Pod: +1. Crie um ConfigMap ou utilize um ConfigMap existente. Múltiplos Pods + podem referenciar o mesmo ConfigMap. +1. Modifique sua definição de Pod para adicionar um volume em + `.spec.volumes[]`. Escolha um nome qualquer para o seu volume, e + referencie o seu objeto ConfigMap no campo + `.spec.volumes[].configMap.name`. +1. Adicione um campo `.spec.containers[].volumeMounts[]` a cada um dos + contêineres que precisam do ConfigMap. Especifique + `.spec.containers[].volumeMounts[].readOnly = true` e informe no campo + `.spec.containers[].volumeMounts[].mountPath` um caminho de um diretório + não utilizado onde você deseja que este ConfigMap apareça. +1. Modifique sua imagem ou linha de comando de modo que o programa procure + por arquivos no diretório especificado no passo anterior. Cada chave no + campo `data` do ConfigMap será transformado em um nome de arquivo no + diretório especificado por `mountPath`. + +Exemplo de um Pod que monta um ConfigMap em um volume: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: mypod +spec: + containers: + - name: mypod + image: redis + volumeMounts: + - name: foo + mountPath: "/etc/foo" + readOnly: true + volumes: + - name: foo + configMap: + name: myconfigmap +``` + +Cada ConfigMap que você deseja utilizar precisa ser referenciado em +`.spec.volumes`. + +Se houver múltiplos contêineres no Pod, cada contêiner deve ter seu +próprio bloco `volumeMounts`, mas somente uma instância de `.spec.volumes` +é necessária por ConfigMap. + +### ConfigMaps montados são atualizados automaticamente + +Quando um ConfigMap que está sendo consumido em um volume é atualizado, as chaves projetadas são +eventualmente atualizadas também. O Kubelet checa se o ConfigMap montado está atualizado em cada +sincronização periódica. +No entanto, o kubelet utiliza o cache local para buscar o valor atual do ConfigMap. +O tipo de cache é configurável utilizando o campo `ConfigMapAndSecretChangeDetectionStrategy` na +[configuração do Kubelet (KubeletConfiguration)](/docs/reference/config-api/kubelet-config.v1beta1/). +Um ConfigMap pode ter sua propagação baseada em um _watch_ (comportamento padrão), que é o sistema +de propagação de mudanças incrementais em objetos do Kubernetes; baseado em TTL (_time to live_, +ou tempo de expiração); ou redirecionando todas as requisições diretamente para o servidor da API. +Como resultado, o tempo decorrido total entre o momento em que o ConfigMap foi atualizado até o momento +quando as novas chaves são projetadas nos Pods pode ser tão longo quanto o tempo de sincronização +do kubelet somado ao tempo de propagação do cache, onde o tempo de propagação do cache depende do +tipo de cache escolhido: o tempo de propagação pode ser igual ao tempo de propagação do _watch_, +TTL do cache, ou zero, de acordo com cada um dos tipos de cache. + +ConfigMaps que são consumidos como variáveis de ambiente não atualizam automaticamente e requerem uma +reinicialização do pod. + +## ConfigMaps imutáveis {#configmap-immutable} + +{{< feature-state for_k8s_version="v1.21" state="stable">}} + +A funcionalidade _Secrets e ConfigMaps imutáveis_ do Kubernetes fornece uma opção +para marcar Secrets e ConfigMaps individuais como imutáveis. Para clusters que utilizam +ConfigMaps extensivamente (ao menos centenas de milhares de mapeamentos únicos de +ConfigMaps para Pods), prevenir alterações dos seus dados traz as seguintes vantagens: + +- protege de atualizações acidentais ou indesejadas que podem causar disrupção na execução + de aplicações +- melhora o desempenho do cluster através do fechamento de _watches_ de ConfigMaps marcados + como imutáveis, diminuindo significativamente a carga no kube-apiserver + +Essa funcionalidade é controlada pelo [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +`ImmutableEphemeralVolumes`. É possível criar um ConfigMap imutável adicionando o campo +`immutable` e marcando seu valor com `true`. +Por exemplo: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + ... +data: + ... +immutable: true +``` + +Após um ConfigMap ser marcado como imutável, _não_ é possível reverter a alteração, nem +alterar o conteúdo dos campos `data` ou `binaryData`. É possível apenas apagar e recriar +o ConfigMap. Como Pods existentes que consomem o ConfigMap em questão mantém um ponto de +montagem que continuará referenciando este objeto após a remoção, é recomendado recriar +estes pods. + +## {{% heading "whatsnext" %}} + +* Leia sobre [Secrets](/docs/concepts/configuration/secret/) (em inglês). +* Leia [Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/) (em inglês). +* Leia [The Twelve-Factor App](https://12factor.net/) (em inglês) para entender a motivação da separação de código +e configuração. diff --git a/content/pt-br/docs/reference/glossary/addons.md b/content/pt-br/docs/reference/glossary/addons.md new file mode 100644 index 0000000000..c3659ed738 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/addons.md @@ -0,0 +1,17 @@ +--- +title: Complementos +id: addons +date: 2019-12-15 +full_link: /pt-br/docs/concepts/cluster-administration/addons/ +short_description: > + Recursos que estendem a funcionalidade do Kubernetes. + +aka: +tags: +- tool +--- + Recursos que estendem a funcionalidade do Kubernetes. + + +[Instalando Complementos](/pt-br/docs/concepts/cluster-administration/addons/) explica mais sobre a utilização de complementos em seu cluster e lista +alguns complementos populares. diff --git a/content/pt-br/docs/reference/glossary/configmap.md b/content/pt-br/docs/reference/glossary/configmap.md new file mode 100644 index 0000000000..a1ccdea668 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/configmap.md @@ -0,0 +1,18 @@ +--- +title: ConfigMap +id: configmap +date: 2021-08-24 +full_link: /docs/concepts/configuration/configmap +short_description: > + Um objeto da API usado para armazenar dados não-confidenciais em pares chave-valor. Pode ser consumido como variáveis de ambiente, argumentos de linha de comando, ou arquivos de configuração em um volume. + +aka: +tags: +- core-object +--- +Um objeto da API usado para armazenar dados não-confidenciais em pares chave-valor. +{{< glossary_tooltip text="Pods" term_id="pod" >}} podem consumir ConfigMaps como variáveis de ambiente, argumentos de linha de comando ou como arquivos de configuração em um {{< glossary_tooltip text="volume" term_id="volume" >}}. + + + +Um ConfigMap ajuda a desacoplar configurações vinculadas ao ambiente das {{< glossary_tooltip text="imagens de contêiner" term_id="image" >}}, de modo a tornar aplicações mais facilmente portáveis. \ No newline at end of file diff --git a/content/pt-br/docs/reference/glossary/image.md b/content/pt-br/docs/reference/glossary/image.md new file mode 100644 index 0000000000..883c5114e9 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/image.md @@ -0,0 +1,17 @@ +--- +title: Image +id: image +date: 2021-08-24 +full_link: +short_description: > + Instância armazenada de um contêiner que contém o conjunto de softwares necessários para rodar uma aplicação. + +aka: +tags: +- fundamental +--- + Instância armazenada de um {{< glossary_tooltip text="contêiner" term_id="container" >}} que contém o conjunto de softwares necessários para rodar uma aplicação. + + + +É uma forma de empacotamento de software que permite que este seja armazenado em um _container registry_, copiado para um sistema local e executado como uma aplicação. Metadados são incluídos na imagem para indicar qual programa deve ser executado, quem produziu a imagem, além de outras informações. diff --git a/content/pt-br/docs/reference/glossary/index.md b/content/pt-br/docs/reference/glossary/index.md new file mode 100644 index 0000000000..89cea39e44 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/index.md @@ -0,0 +1,11 @@ +--- +title: Glossário +layout: glossary +noedit: true +default_active_tag: fundamental +weight: 5 +card: + name: reference + weight: 10 + title: Glossário +--- diff --git a/content/pt-br/docs/reference/glossary/kubelet.md b/content/pt-br/docs/reference/glossary/kubelet.md index 16b55527fb..6a0d0ac415 100644 --- a/content/pt-br/docs/reference/glossary/kubelet.md +++ b/content/pt-br/docs/reference/glossary/kubelet.md @@ -2,7 +2,7 @@ title: Kubelet id: kubelet date: 2020-04-19 -full_link: /docs/reference/generated/kubelet +full_link: /docs/reference/command-line-tools-reference/kubelet short_description: > Um agente que é executado em cada node no cluster. Ele garante que os contêineres estejam sendo executados em um pod. diff --git a/content/pt-br/docs/reference/glossary/namespace.md b/content/pt-br/docs/reference/glossary/namespace.md new file mode 100644 index 0000000000..e20031277d --- /dev/null +++ b/content/pt-br/docs/reference/glossary/namespace.md @@ -0,0 +1,18 @@ +--- +title: Namespace +id: namespace +date: 2021-09-17 +full_link: /docs/concepts/overview/working-with-objects/namespaces +short_description: > + Uma abstração utilizada pelo Kubernetes para suportar múltiplos clusters virtuais no mesmo cluster físico. + +aka: +tags: +- fundamental +--- + Uma abstração utilizada pelo Kubernetes para suportar múltiplos clusters virtuais no mesmo {{< glossary_tooltip text="cluster" term_id="cluster" >}} físico. + + + +Namespaces são utilizados para organizar objetos em um cluster e oferecer uma forma de separar os recursos do cluster. Os nomes dos recursos precisam ser únicos dentro de um namespace, +mas não entre todos os namespaces existentes. diff --git a/content/pt-br/docs/reference/glossary/operator-pattern.md b/content/pt-br/docs/reference/glossary/operator-pattern.md new file mode 100644 index 0000000000..001207f5cf --- /dev/null +++ b/content/pt-br/docs/reference/glossary/operator-pattern.md @@ -0,0 +1,24 @@ +--- +title: Padrão Operador +id: operator-pattern +date: 2021-09-17 +full_link: /pt-br/docs/concepts/extend-kubernetes/operator/ +short_description: > + Um controlador especializado que gerencia um recurso personalizado. + +aka: +tags: +- architecture +--- + O [padrão Operador](/docs/concepts/extend-kubernetes/operator/) é um design de +sistema que vincula um {{< glossary_tooltip text="controlador" term_id="controller" >}} +a um ou mais recursos personalizados. + + + +Você pode estender a funcionalidade do Kubernetes adicionando controladores ao seu cluster, +além dos controladores que são distribuídos como parte do Kubernetes. + +Se uma aplicação em execução age como um controlador e tem acesso à API para desempenhar +tarefas em um recurso personalizado que está definido na camada de gerenciamento, este é +um exemplo do padrão Operador. diff --git a/content/pt-br/docs/reference/glossary/pod.md b/content/pt-br/docs/reference/glossary/pod.md index 9f6e21d6b3..2414223e43 100644 --- a/content/pt-br/docs/reference/glossary/pod.md +++ b/content/pt-br/docs/reference/glossary/pod.md @@ -2,7 +2,7 @@ title: Pod id: pod date: 2020-04-19 -full_link: /docs/concepts/workloads/pods/pod-overview/ +full_link: /docs/concepts/workloads/pods/ short_description: > O menor e mais simples objeto Kubernetes. Um Pod representa um conjunto de contêineres em execução no seu cluster. aka: diff --git a/content/pt-br/docs/reference/glossary/secret.md b/content/pt-br/docs/reference/glossary/secret.md new file mode 100644 index 0000000000..9eead5cbf0 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/secret.md @@ -0,0 +1,18 @@ +--- +title: Secret +id: secret +date: 2021-08-24 +full_link: /docs/concepts/configuration/secret/ +short_description: > + Armazena dados sensíveis, como senhas, tokens OAuth e chaves SSH. + +aka: +tags: +- core-object +- security +--- + Armazena dados sensíveis, como senhas, tokens OAuth e chaves SSH. + + + +Permite mais controle com relação a como as informações sensíveis são armazenadas e reduz o risco de exposição acidental, incluindo [encriptação](/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted) em disco. Um {{< glossary_tooltip text="Pod" term_id="pod" >}} pode referenciar o Secret como um arquivo em um volume montado, ou o kubelet pode usar este dado quando baixa imagens para um pod. Secrets são úteis para dados confidenciais, enquanto [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) são úteis para dados não-confidenciais. diff --git a/content/pt-br/docs/reference/glossary/selector.md b/content/pt-br/docs/reference/glossary/selector.md new file mode 100644 index 0000000000..6d0d8baf71 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/selector.md @@ -0,0 +1,17 @@ +--- +title: Selector +id: selector +date: 2021-09-17 +full_link: /docs/concepts/overview/working-with-objects/labels/ +short_description: > + Permite ao usuário filtrar uma lista de recursos com base em rótulos (labels). + +aka: +tags: +- fundamental +--- + Permite ao usuário filtrar uma lista de recursos com base em {{< glossary_tooltip text="labels" term_id="label" >}} (rótulos). + + + +Selectors são aplicados quando um usuário faz uma pesquisa por listas de recursos para filtrá-los por labels (rótulos). diff --git a/content/pt-br/docs/reference/glossary/service.md b/content/pt-br/docs/reference/glossary/service.md new file mode 100644 index 0000000000..b5a415be0b --- /dev/null +++ b/content/pt-br/docs/reference/glossary/service.md @@ -0,0 +1,20 @@ +--- +title: Service +id: service +date: 2021-09-17 +full_link: /docs/concepts/services-networking/service/ +short_description: > + Uma forma abstrata de expor uma aplicação que está executando em um conjunto de Pods como um serviço de rede. + +aka: +tags: +- fundamental +- core-object +--- +Uma forma abstrata de expor uma aplicação que está executando em um conjunto de {{< glossary_tooltip text="Pods" term_id="pod" >}} como um serviço de rede. + + + + O conjunto de Pods referenciado por um Service é (geralmente) determinado por um {{< glossary_tooltip text="selector" term_id="selector" >}}. Se mais Pods são + adicionados ou removidos, o conjunto de Pods que atende ao critério do selector será alterado. O Service garante que o tráfego de rede pode ser direcionado ao + conjunto atual de Pods para a carga de trabalho. diff --git a/content/pt-br/docs/reference/glossary/static-pod.md b/content/pt-br/docs/reference/glossary/static-pod.md new file mode 100644 index 0000000000..55ba56bb53 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/static-pod.md @@ -0,0 +1,19 @@ +--- +title: Pod estático +id: static-pod +date: 2021-09-17 +full_link: /docs/tasks/configure-pod-container/static-pod/ +short_description: > + Um pod gerenciado diretamente pelo daemon do kubelet em um nó específico. + +aka: +tags: +- fundamental +--- + +Um {{< glossary_tooltip text="pod" term_id="pod" >}} gerenciado diretamente pelo + daemon do kubelet em um nó específico, + + + +que não é observado pelo servidor de API. \ No newline at end of file diff --git a/content/pt-br/docs/reference/glossary/volume.md b/content/pt-br/docs/reference/glossary/volume.md new file mode 100644 index 0000000000..502d2db6f1 --- /dev/null +++ b/content/pt-br/docs/reference/glossary/volume.md @@ -0,0 +1,20 @@ +--- +title: Volume +id: volume +date: 2021-08-24 +full_link: /docs/concepts/storage/volumes/ +short_description: > + Um diretório contendo dados, accessível aos contêineres em um pod. + +aka: +tags: +- core-object +- fundamental +--- + Um diretório contendo dados, accessível aos {{< glossary_tooltip text="contêineres" term_id="container" >}} em um {{< glossary_tooltip term_id="pod" >}}. + + + +Um volume do Kubernetes existe enquanto o Pod que utiliza ele existir também. Consequentemente, um volume dura mais tempo que qualquer contêiner que rodar em um Pod, e os dados no volume são preservados entre reinicializações do contêiner. + +Veja [armazenamento](/docs/concepts/storage/) para mais detalhes. From 6b610b8e33e8f02aaacced0cfee01009dd687424 Mon Sep 17 00:00:00 2001 From: Jacob Valdemar Date: Sun, 19 Sep 2021 17:55:11 +0200 Subject: [PATCH 035/249] Remove prefixed spaces in code environment --- .../kubeadm/kubeadm-upgrade.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index da07ed672e..d3b2fd9c07 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -166,11 +166,10 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no ### Upgrade kubelet and kubectl -- Upgrade the kubelet and kubectl +- Upgrade the kubelet and kubectl: -{{< tabs name="k8s_install_kubelet" >}} -{{< tab name="Ubuntu, Debian or HypriotOS" >}} - 
    +{{< tabs name="k8s_kubelet_and_kubectl" >}}
+{{% tab name="Ubuntu, Debian or HypriotOS" %}}
     # replace x in {{< skew latestVersion >}}.x-00 with the latest patch version
     apt-mark unhold kubelet kubectl && \
     apt-get update && apt-get install -y kubelet={{< skew latestVersion >}}.x-00 kubectl={{< skew latestVersion >}}.x-00 && \
@@ -179,14 +178,11 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no
     # since apt-get version 1.1 you can also use the following method
     apt-get update && \
     apt-get install -y --allow-change-held-packages kubelet={{< skew latestVersion >}}.x-00 kubectl={{< skew latestVersion >}}.x-00
-
    -{{< /tab >}} -{{< tab name="CentOS, RHEL or Fedora" >}} - 
    +{{% /tab %}}
+{{% tab name="CentOS, RHEL or Fedora" %}}
     # replace x in {{< skew latestVersion >}}.x-0 with the latest patch version
     yum install -y kubelet-{{< skew latestVersion >}}.x-0 kubectl-{{< skew latestVersion >}}.x-0 --disableexcludes=kubernetes
-
    -{{< /tab >}} +{{% /tab %}} {{< /tabs >}} - Restart the kubelet: From 483779104b88b4297b2967d5a66c5f7ec5c8887a Mon Sep 17 00:00:00 2001 From: Jacob Valdemar Date: Sun, 19 Sep 2021 18:01:51 +0200 Subject: [PATCH 036/249] Undo accidental change to tabs name --- .../en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index d3b2fd9c07..f855c2be5c 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -168,7 +168,7 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no - Upgrade the kubelet and kubectl: -{{< tabs name="k8s_kubelet_and_kubectl" >}} +{{< tabs name="k8s_install_kubelet" >}} {{% tab name="Ubuntu, Debian or HypriotOS" %}} # replace x in {{< skew latestVersion >}}.x-00 with the latest patch version apt-mark unhold kubelet kubectl && \ From cc8bda3f410cfe51f2d04060a6e1da2389b24f98 Mon Sep 17 00:00:00 2001 From: Jacob Valdemar Date: Sun, 19 Sep 2021 22:07:09 +0200 Subject: [PATCH 037/249] Add another newline after the tab widgets --- .../docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index f855c2be5c..288852cd1f 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -89,6 +89,7 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc {{% /tab %}} {{< /tabs >}} + - Verify that the download works and has the expected version: ```shell @@ -185,6 +186,7 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no {{% /tab %}} {{< /tabs >}} + - Restart the kubelet: ```shell @@ -265,6 +267,7 @@ without compromising the minimum required capacity for running your workloads. {{% /tab %}} {{< /tabs >}} + - Restart the kubelet: ```shell From 2cac383a7c373611f23804febc78edba49977d54 Mon Sep 17 00:00:00 2001 From: Jacob Valdemar Date: Sun, 19 Sep 2021 22:17:40 +0200 Subject: [PATCH 038/249] Undo "Add another newline after the tab widgets" --- .../docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 288852cd1f..f855c2be5c 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -89,7 +89,6 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc {{% /tab %}} {{< /tabs >}} - - Verify that the download works and has the expected version: ```shell @@ -186,7 +185,6 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no {{% /tab %}} {{< /tabs >}} - - Restart the kubelet: ```shell @@ -267,7 +265,6 @@ without compromising the minimum required capacity for running your workloads. {{% /tab %}} {{< /tabs >}} - - Restart the kubelet: ```shell From 54185d178df59c1981fd3ac9ce18f69b88ef3d5a Mon Sep 17 00:00:00 2001 From: august Date: Tue, 7 Sep 2021 16:41:52 +0300 Subject: [PATCH 039/249] add blog post on handling data duplication in data-heavy environments --- ...-data-duplication-in-data-heavy-k8s-env.md | 243 ++++++++++++++++++ .../create-volume-snapshot-gcp.png | Bin 0 -> 69860 bytes .../identify-volume-aws.png | Bin 0 -> 303818 bytes 3 files changed, 243 insertions(+) create mode 100644 content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md create mode 100644 static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png create mode 100644 static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png diff --git a/content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md b/content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md new file mode 100644 index 0000000000..bec8c0b892 --- /dev/null +++ b/content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md @@ -0,0 +1,243 @@ +--- +layout: blog +title: "How to Handle Data Duplication in Data-Heavy Kubernetes Environments" +date: 2021-09-07 +slug: how-to-handle-data-duplication-in-data-heavy-kubernetes-environments +--- + +**Authors:** +Augustinas Stirbis (CAST AI) + +## Why Duplicate Data? + +It’s convenient to create a copy of your application with a copy of its state for each team. +For example, you might want a separate database copy to test some significant schema changes +or develop other disruptive operations like bulk insert/delete/update... + +**Duplicating data takes a lot of time.** That’s because you need first to download +all the data from a source block storage provider to compute and then send +it back to a storage provider again. There’s a lot of network traffic and CPU/RAM used in this process. +Hardware acceleration by offloading certain expensive operations to dedicated hardware is +**always a huge performance boost**. It reduces the time required to complete an operation by orders +of magnitude. + +## Volume Snapshots to the rescue + +Kubernetes introduced [VolumeSnapshots](/docs/concepts/storage/volume-snapshots/) as alpha in 1.12, +beta in 1.17, and the Generally Available version in 1.20. +VolumeSnapshots use specialized APIs from storage providers to duplicate volume of data. + +Since data is already in the same storage device (array of devices), duplicating data is usually +a metadata operation for storage providers with local snapshots (majority of on-premise storage providers). +All you need to do is point a new disk to an immutable snapshot and only +save deltas (or let it do a full-disk copy). As an operation that is inside the storage back-end, +it’s much quicker and usually doesn’t involve sending traffic over the network. +Public Clouds storage providers under the hood work a bit differently. They save snapshots +to Object Storage and then copy back from Object storage to Block storage when "duplicating" disk. +Technically there is a lot of Compute and network resources spent on Cloud providers side, +but from Kubernetes user perspective VolumeSnapshots work the same way whether is it local or +remote snapshot storage provider and no Compute and Network resources are involved in this operation. + +## Sounds like we have our solution, right? + +Actually, VolumeSnapshots are namespaced, and Kubernetes protects namespaced data from +being shared between tenants (Namespaces). This Kubernetes limitation is a conscious design +decision so that a Pod running in a different namespace can’t mount another application’s +[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVC). + +One way around it would be to create multiple volumes with duplicate data in one namespace. +However, you could easily reference the wrong copy. + +So the idea is to separate teams/initiatives by namespaces to avoid that and generally +limit access to the production namespace. + +## Solution? Creating a Golden Snapshot externally + +Another way around this design limitation is to create Snapshot externally (not through Kubernetes). +This is also called pre-provisioning a snapshot manually. Next, I will import it +as a multi-tenant golden snapshot that can be used for many namespaces. Below illustration will be +for AWS EBS (Elastic Block Storage) and GCE PD (Persistent Disk) services. + +### High-level plan for preparing the Golden Snapshot + +1. Identify Disk (EBS/Persistent Disk) that you want to clone with data in the cloud provider +2. Make a Disk Snapshot (in cloud provider console) +3. Get Disk Snapshot ID + +### High-level plan for cloning data for each team + +1. Create Namespace “sandbox01” +2. Import Disk Snapshot (ID) as VolumeSnapshotContent to Kubernetes +3. Create VolumeSnapshot in the Namespace "sandbox01" mapped to VolumeSnapshotContent +4. Create the PersistentVolumeClaim from VolumeSnapshot +5. Install Deployment or StatefulSet with PVC + +## Step 1: Identify Disk + +First, you need to identify your golden source. In my case, it’s a PostgreSQL database +on PersistentVolumeClaim “postgres-pv-claim” in the “production” namespace. + +```terminal +kubectl -n get pvc -o jsonpath='{.spec.volumeName}' +``` + +The output will look similar to: +``` +pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d90 +``` + +## Step 2: Prepare your golden source + +You need to do this once or every time you want to refresh your golden data. + +### Make a Disk Snapshot + +Go to AWS EC2 or GCP Compute Engine console and search for an EBS volume +(on AWS) or Persistent Disk (on GCP), that has a label matching the last output. +In this case I saw: `pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d9`. + +Click on Create snapshot and give it a name. You can do it in Console manually, +in AWS CloudShell / Google Cloud Shell, or in the terminal. To create a snapshot in the +terminal you must have the AWS CLI tool (`aws`) or Google's CLI (`gcloud`) +installed and configured. + +Here’s the command to create snapshot on GCP: + +```terminal +gcloud compute disks snapshot --project= --snapshot-names= --zone= --storage-location= +``` +{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png" alt="Screenshot of a terminal showing volume snapshot creation on GCP" title="GCP snapshot creation" >}} + + +GCP identifies the disk by its PVC name, so it’s direct mapping. In AWS, you need to +find volume by the CSIVolumeName AWS tag with PVC name value first that will be used for snapshot creation. + +{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png" alt="Screenshot of AWS web console, showing EBS volume identification" title="Identify disk ID on AWS" >}} + +Mark done Volume (volume-id) ```vol-00c7ecd873c6fb3ec``` and ether create EBS snapshot in AWS Console, or use ```aws cli```. + +```terminal +aws ec2 create-snapshot --volume-id '' --description '' --tag-specifications 'ResourceType=snapshot' +``` + +## Step 3: Get your Disk Snapshot ID + +In AWS, the command above will output something similar to: +```terminal +"SnapshotId": "snap-09ed24a70bc19bbe4" +``` + +If you’re using the GCP cloud, you can get the snapshot ID from the gcloud command by querying for the snapshot’s given name: + +```terminal +gcloud compute snapshots --project= describe | grep id: +``` +You should get similar output to: +``` +id: 6645363163809389170 +``` + +## Step 4: Create a development environment for each team + +Now I have my Golden Snapshot, which is immutable data. Each team will get a copy +of this data, and team members can modify it as they see fit, given that a new EBS/persistent +disk will be created for each team. + +Below I will define a manifest for each namespace. To save time, you can replace +the namespace name (such as changing “sandbox01” → “sandbox42”) using tools +such as `sed` or `yq`, with Kubernetes-aware templating tools like +[Kustomize](/docs/tasks/manage-kubernetes-objects/kustomization/), +or using variable substitution in a CI/CD pipeline. + +Here's an example manifest: + +```yaml +--- +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshotContent +metadata: + name: postgresql-orders-db-sandbox01 + namespace: sandbox01 +spec: + deletionPolicy: Retain + driver: pd.csi.storage.gke.io + source: + snapshotHandle: 'gcp/projects/staging-eu-castai-vt5hy2/global/snapshots/6645363163809389170' + volumeSnapshotRef: + kind: VolumeSnapshot + name: postgresql-orders-db-snap + namespace: sandbox01 +--- +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: postgresql-orders-db-snap + namespace: sandbox01 +spec: + source: + volumeSnapshotContentName: postgresql-orders-db-sandbox01 +``` + +In Kubernetes, VolumeSnapshotContent (VSC) objects are not namespaced. +However, I need a separate VSC for each different namespace to use, so the +`metadata.name` of each VSC must also be different. To make that straightfoward, +I used the target namespace as part of the name. + +Now it’s time to replace the driver field with the CSI (Container Storage Interface) driver +installed in your K8s cluster. Major cloud providers have CSI driver for block storage that +support VolumeSnapshots but quite often CSI drivers are not installed by default, consult +with your Kubernetes provider. + +That manifest above defines a VSC that works on GCP. +On AWS, driver and SnashotHandle values might look like: + +```YAML + driver: ebs.csi.aws.com + source: + snapshotHandle: "snap-07ff83d328c981c98" +``` + +At this point, I need to use the *Retain* policy, so that the CSI driver doesn’t try to +delete my manually created EBS disk snapshot. + +For GCP, you will have to build this string by hand - add a full project ID and snapshot ID. +For AWS, it’s just a plain snapshot ID. + +VSC also requires specifying which VolumeSnapshot (VS) will use it, so VSC and VS are +referencing each other. + +Now I can create PersistentVolumeClaim from VS above. It’s important to set this first: + + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: postgres-pv-claim + namespace: sandbox01 +spec: + dataSource: + kind: VolumeSnapshot + name: postgresql-orders-db-snap + apiGroup: snapshot.storage.k8s.io + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 21Gi +``` + +If default StorageClass has [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) policy, +then the actual Cloud Disk will be created from the Golden Snapshot only when some Pod bounds that PVC. + +Now I assign that PVC to my Pod (in my case, it’s Postgresql) as I would with any other PVC. + +```terminal +kubectl -n get volumesnapshotContent,volumesnapshot,pvc,pod +``` + +Both VS and VSC should be *READYTOUSE* true, PVC bound, and the Pod (from Deployment or StatefulSet) running. + +**To keep on using data from my Golden Snapshot, I just need to repeat this for the +next namespace and voilà! No need to waste time and compute resources on the duplication process.** diff --git a/static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png b/static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png new file mode 100644 index 0000000000000000000000000000000000000000..795151cb538073b05ad8d6a54a85954620d71614 GIT binary patch literal 69860 zcmeFZbyU>R+BPhRfruy_A}A%@ARwVgcXvp`5Yk;LBHbN>G|bT5s4#SQcX#*jZlC9S z*K>~NIp_axecxKMaAug_Z}#5zeP8!=U3+7IysQ}JBjQJQ?%ctY5dWZf=gtH2J9kjX zAEJVH?3ItA@7y83Bk|##vJ1*~!vl5YiNw7g9{bhq1opMwo7L0=9($VUgfet$qk?k$wH7;g-tHr_klVAm*_Fs3!KZgn{=*~ZVKvLO@12J}|Nikt1Kj@qqeIbDL--7s4bvfmMGbY#$tF?$eYn!d@A#_tE?a#-OFeh({!071p{+|m zV9vw+a5$lA;MFktzrR$T6NE^GJ%0yx0P97kgUZC17;8zz%>8Jre}Bz9+Mlor!4?>h zZJ}Mb;2Og=@4^MTEg{>Yz+=ZWJTugXK0b`7=P8rvUkv}31d*4$gSs_E_Hc{aib}*W zVQ#aU=HR-l+lI5%X1xApr>ay`>l3f&5!R8SSFM(IMpin@(>uP2l*QMYKj-=s{2E@v zI^HYb|1VWgZ;wV^+&G}5sSR&*cFH?oF<@IGrZ)4J?cfEw29f!Kg@z!!;?AH)$ zA_6w#8M-!n!@qnwqvTH3#p9x9rPuddRz=$+xNDedZM{79ai?i&m-B(jjiZRB7rN6_ zS##{3bnckeFVmL~(*Lz4XJt@r^z=CiA6LEK-JbC_yDH@8JfEn$?Jo%DL)2!bEw8M| zBnTvN452G!lVQG1NR6-+SRZUJnUh~;=C%&Z*uD%&!WFdJ=3e{4lx1r49P=&D$4-Yp z?egKGw0Sjh`9S{kPq&^LU|y+pY+HjJ3S2|ak9sdz{ z%Zw=%$%p~J|K%gN;y757fv?o>aM8Jk$0kJ6c!}21}{qvQ4g2vaB~nubv~* zyIa*xs>+o6>{CiEHjkntl64$tZ$liNZuu*MiZ-j*Rn(xo*GnFTAs$6^m;SEQM{9c6 zDT1UKp^3^}-nz-JhHxNH4ZliC{mN=(Bhcp1_@;SXUFbqcnh3xVoB>_o9%|QnI98vXQ%T)CA5cSCf!E6kR9!V19Z-*-bto{sZePGy^(k!8pgxf`sPl;AlV^A z)8GgfQM3(>?PbYhBJ2yiXY0^44_~`hB7xmG?Jp!1E zSY&j;m3No)js~e^95MaqA>!9l^*KTdWwgJ8efqQ-$0@GM>=Xjm*2${xohWHE_I2Dt z3J+hJm#vPZdHd|tok}NhhDpxW-G=k8rKt`Ph9LZ;VinvysAVw2s;g|fP8F8AQ&Co# zHDZ&myEX5=${_2GFY`GJJxD0GT9f;@@GQfVTA}+FRo7-{Kw$;b{`opbl*Zg#w=$J6 zU!Gbco^MPQnQyI7+9m21$-1AFZ^iG}#{_yR^TIMQjCq*8BZ87QBzhy>Ofo$q%7<`b zRcjrP$hA<#g_nt}PJ9zaEtho~`N8i^;k~T_)NAp^N>jx2YHDCXk6@7ot%q};2*Hi- zhi?8I^JXrT3emFM>8CLt*xfl)6KZRX@}QbZCquQRL?GfbjS#F8S5WSq2o;*_Q%R?E zm7m-;O5FbLVOX^eev+?WzkW;Ru(6!Su3{aYKC%%Ced~GA>1e&^cD-z_tG{3HlqT_5 zwN#ofYHJa;*~mU*u;Z*^jKREub=3F5`K{?l$S;D=SST40Iswcz*m{E|84?M^t56xe zJr&i@k`CJ~6g}naHF{*dFL2_^c1?cJ_2pA4t6NNa<(MsYOg}@JkSfG&F^T7x`;5I5 za~`SHkyY=v$)?mDbB;X&*^AENwKsg-Gm=B4yfi#iv$h0BK0LXvG?fkQA#tO0M@-aj zp;Ik?K6U$?_eq9IBuGtHBu%;PbWX|Jnc8_hVq0VFNREGWU5`6*%_9}ogsu1Lu85fZ zaCRY2!)Rmia5Z1O%ci;g4$6Q<5D67g$hNB8Mt$1#0&?Oj?|b8#s(Vz3hG(rfvdfQY z#T=+B6`gY{)E1?XEN$hfo{pbhrVu2klif;IExP@T7gBoJxOUjvYW_Gz`GeA!(vghx zFWoouJ}0_?r;;8L+Sw}vcSW*R-=zwQNPk5XqrP2deDtT7LF2@muGWU1pq8272duqR!?C^&Y+0 z_)+;FXj7qhR7V_rAwl(d{s;N`T>1kl(4M^G<9-=#V_BjzUhBo6{DEGGdA@X^XJj6Q zjJ9Pafb^pq4Zp0I%w`N*s@wYNK_c4&71#3ykMlt|x+27Q<@*y?49Vc%U2(HrgD!=# zap)OSySO#y`H6)RRZUG8%n9|`T<0Z`UzAflm#m_a=?cWzi6`pGyO1h832gjVJJLfM zI8sNJ;QGn%D|WQp=>Tj9-u%j#~pVX23 z3~KmvP#O~LxGbNF5bq?I2n;Mpm+?okh;eSLY2Us+LAq}iVVT}$Yx8Q`wNi!=2XANW zB#>G4QDW{E(GRX~m9ww*^c1zk^?R*jsx@d|NWGgE?x{`kv+C;60oVq!R#r@L@*_}^ zA})~Bvu&84_dLN;!^p%l2cZgGfmH3_s`J_OoX;>s(e!{)+_J%Qnz6Q1*#I2Ct#&YW zYrtGna|Br=rD&ScH|H(afK+|Sh;38Hx@6N;W$Uc0&wnX&!@crKCyH~Us}S#bbrBL7 zScBmmpMfny0TI?uvHuBwSveWrNL{_(?0NDz=;F3!IP(0FHa>>Q3Wl#uU>1UMlFMov9tZ%B)COOUmHl1dipX@vHdGV`6Jmd`?HmTl6Z zjZO;u$UbyNmWR>*$mvb4T4z&%4UwWhe9mfuD#tVJe*K*lCln3M!ZFs0q)7Y9k7l;S zeJ^QU;$)K?E6kfuib@Nin{ApI@**JLh`WljJQt9|2fJ#4<*Q#f)eN_*Xu7V{L#ric z%#6yqvYG{Zq!y%dpeyU*yU)E}iCIbngkri`i9m9iFQ*dd66Ee2`tG0ED^Gt{vC)_5 z9I2`#VHbbBj5~&wq!v+kuG(&Nof{(L_5%sUPdw%XOmo^6#ZWrKxFu&G-+}4tNOiLt zs!ViS8T2I(kME6;W}d0F~C2B^8PaRF%@PEl$+xj8@Wvd` z)!d%gf~u&MN?&cN<(U%dVm3suDD|W4Xc1HW@=&Da=L8;!!k*FWpg~G1CMNbl#ER1+ zxIo*A<`nEz*%LhEWYbzJ4fRlyV6TdXa&~nKV&|!=oD~{=471vqb_!E|M>J9;B2_zj zR-D6q9B%6yQ~&DmTltUPok{tfLs$aIJ&fwKx*Z9ur>-@X?K+=i1&r-_r>y*^d5Y}r zupr>*@L<*;$AQb6P%SzK;Wf0;vXX3+>!#HlT(whs-d&eaAXxSnEPH)fO@#Glo|)Lm z3z1 z*S-vyWNjZrS1O6zksi^Bxi%H&jC6zSytauE_BH*C=+toEPt$ zpsL#%fGJA&FA<_g#pfKIlvmqIZr1#gQ?vUJzAWuDTy0ZLxyk4J(aFh=>x|=N1exM} z(clE~%SY|Ky$Sp_&*Zoay1%loF%6SfP10|QM^7daNR=KqXdN6%!$oO{17%8I7Dpn8 zk?EGE>I3;Cetf?It!d)SPvA$YLk3p6s-?4%Lr(nQ-4+8^mjmOdHbuUBs@6UoW$cLd z%H#LR8hs}vnT-d>+}KhQW=p7Bv)>n+k_r^Phzs-gkrOTzuJ7CeLTLpleR>=9xzrrQ z^$SW*;Rbh!pq+0U4qs?Ie%muhT3w`zqg~CD$@01|Gp$9O_#j^BsAztfBFYJ1{N<^` zuKh=Q_MohZr|Lw8K=#wTNO=AM-lRf!E*}+HbW1gmXqDnzU-=%VKJy4bl6{*B?VIZVKKPl5ah>T;+*c@|){bwslO!fbVdt<>n@Lq&|$w z!7vvg4y%EtI|9YoBOTNJ$;e0ImULB5;&w&jbumwirF293beHe;YtNh8&J^rVa;5k~ zZ=TUwz(MF;-&J;7ro3)aEgPly0&?`6efoyb;nK#}s`Rx1^t)^z14E|?N@6JhwIu<4MPd!SLPt}f7a$34LcQtM9t_ zOR?;Pi9r1>nZ;&-N8a`ExW>mB39kI(9#4R>PY$jdzyU7O?D<>oF?t%9S}x zb28ej&GoW%d6{o;NDOJteGl)}%%&han!M!nYZ+P~A)|#%?n_1RJ12;Cg^#U;V!$)P zRO`cjL7l`givYOM9J8K)x7&+Mg@MXKuS(H?Dwyrza?Ey=0p`q5-7 zIJZS&Xr>|pe5CbPM0;d$RY)|f0V;T&)>rh+-u2(G*yeL@tl+?fUgSq=l-$USY&yWk zRDP7a*QtQgDE`H+>iyA-fIu6ukk=`&$A^zhj!0I%Q%#vz4h+*7P~>Dk!VA+|txAOI z1o>vss~xo@Bow(MKEU$k+@v!MhI1C|Xh1W#dNDh1p)1rgAp%WF#*>WsnWYajA*fq5 zZ!mY>EiateO;mjT9g)o+3dqM=Bt1HuidWQ6umi5W3uzn{2elXkDKtifBprrSR?d}J zvOri8xue%mx3+-7hdTe*NIV7x3*5811;omtVsz}m=SAt!wA)ipKlO{t?|!!`u5Odj z+fe-M^VF^>9EEMNEM2dHv>TJ7`|{fQS=|JSl9$)fB6T_g=5}Z9dDF_B>CO4vEOe*N zWzw}fqy6cISi>YX1YV7TL6m9=-<2Q1@ilE|=|Gc0e!e(2uj&|%3#p>tNjI4i@yYt! zV~bo-on4fAc=$8lLG+`H{!PvFvoZoN_UmpAwOG!O> z123GXx1EPrVyf^IJrT#4${mm`BeWro8-Xg$MZch2s{g#9#|H4H>kMGzJ^G$JRdU~m+F3K_4*7V`Q#O?~AN1Mlgt zFl?z$F+gY0jIv%cu)*rSe8` zeg%HrI^08C0)}o&qMPFD>5Nd$YdyIdMN|C}5h9TI#~WryJrG{GZ1mS_?XXh@AB-5W zIC9)D3cb;jQew4}re7Q@v6P1DsIuam1{Qk{6#cHiX@9TZ8sBRv}Rn1syA zS68mnr`qIB;~&4zN$tg}lRut}%Cj4zjU)i-OIx2RC*k&u;EG15N6C?n3Xy~?=j&o> zldbljD*CM2B z&Q1Z8wv&$QyI*cs@N)Rimw@zgP0wY%}UITt$ATg9--p(K5n)VqfKB?DH2yjCJh&1bvdW75o)yAZFqd1#~4BP zttcKA@7m$lv-w7y>ip>TT3t952M~kriPdaWtsd1ql;4k1hTv3X>;gPuZDNhx)mpf^ zJ05=erdbsc@JjBQ$O?^N05hHQ^Ynl32dGn$E2+gXoJ3jKsVF(Y9X0h=@ZnR=yCnxT zPWrG0`%{aGrVf9kojhOEXQmme^5&e;xS^m?% zj3@asw&5I9TCpMnA0Aw?{G3!GxxvZU4jEh`wxN0G(xuYdkt0od8%CxmlJoRTvHZ4% zBPM?pA8yotIfiFGT}5liAZC?meVN*Y)5gx|e(PX^&!|ST5ML<|kZxq<96R+*nfP3d z$50M&`JZB2VYW&&>B(i!=-+;7xWY{BsR84bXc#Vr$;o(Gf)pO=U5><-dCx*S_D9pd zeDe~BjqI4D(ZO6CZ-HSS;Nfsky6tp&M8M=a;=GMCBJCY0zXyE@5q9Fx7uLy2Z$Wd^ zHuU_Bx&_fgA^B0r%exu=;;%k(LkX&IggB&|3?eg8GHUJ{Q)h(N`oY-cs+YghSJ4@5 z6naX!Y}M#-Qd^pLJZ`CA$iM9^4T~btR78&}(p_uaoo*O2C~;&|w9()7qBh?5hsD5& z!Bzkf9k8FY)#{&<#DlGpq4VGHyX>|V;4y)Krm7QM#3|EnxGk*M@plgZnsa9*u|-Cw*t$T$FhGi+w~-A z-~6tRE_z}t#}hMUCAJC-Jpe9#+N_^^PbMHyh`Aoclc^}q6NlaT-xEgPE^Ov<>`LMh zcTyR7kH0XyDD~R3(3uwZ`I^_RTy_e zMV@@hL&3kjIn<}xnzUWH!&rMzG3pFCa0<*ZD7_$%*|pwUol7E6-!M7YI()3e-AAf< z@1OnB{a&2n+(N}7h_Inwv*s4X=C9T+ZS;_80`rW<9P|J&RMfbO0^QZBOL3w3p|2lE z0`^3w*IHAtYK9A(S%8bx?cfs7 ztuq-xdI*dwmpBh~h!MZo19B*&XtT^yW1{t2SY?s}ll4zYj?zHPCAsD6o99vOBG6tf z_VmH>swkPHi|Z!iyyu$Q+2#}!z($ze5;>#2Fj;DGXq}Y_E@@(5O)w3#AU2e?(f1m| zXjujR=3C9ilcy8db_$VWTE^rBLW283_#0;R1T#)X0v;M4ha~;XPAq=~JjYI>T>(vC zfXm1jm?VI2=Se?NZm%p; z^N>9A?rGAt_pPm0V8u>8q2ZscK>oN?<@ixH#C67SDau|AuTITbqCG0Cwp*OI*3z5B zqFxvkBH5VQ-5i7D&VQcXiDrJ!&8<~2%Q7+A%88NJ0en-q@N z1STrzEV^wP$Xk)ZJU-BIL;2;K+AB248Ed^G^OT%;sPw$IecizQvuG;6g|BQvURG9j zpTE}WdJwPvcygb-c=Tt~XX?L?mI%dc(!I>r2&s_8fR7O2IJSx0F=RMj zwaTh6fA?rM8NSRL+d?{~vTu~GS6fKV%&*OtkT+6h6#v}+o3BKg zWgTPEhSLz2vG+zBuCOBH-&l>-gZ;+gdDBU_tMcLOjsNVWJR5PyQ5-|YX>igT&bN!R z*^GRmAy?lF@EK3s58fLWM09lYxy&joHBOWpi@CHk*y7|U&bneWaTC(*Jh3TA=EZAT z4(xca;%sU+=K=er7ky(L`)BjHld8`)f!EH@74C816;dvvV{%=!CnI||RtL816w8_o z?6q0#x#aG=h+6rF4_OeX=`#7bQ-N^^N!d5aAKCYPI≷ITu_eIA-@dzs5ovbzV10 zr~ixf@1dgjgIf3`V-ElCJ>=-Gzb3EFRNGT(SwvG?1S@9RIJdaXD`H~N2?$_n1rz#w zV;S+|wf%fzQ~E5WMRmC4Y}@P&*d2K94w-4lYfF?>WLs4u`+h}7U(-p)w+=qNk1~a(8O_%) z<5G8DJB~O+dR1oJX=HCo;URo)ib%}+-9H~o+I-{NU;K$FgHdBMXLP)xR4KdU+c!VH zfu1UcV&cd5n^d-3)*x?-?M`ZI4d-jjx1LOqMo*L=5d0-6UOSgPvdMNv9SS5`RxANY z$IrM`ElLDt((Af6h1dxv;)RApPdhh}#XdCW7k5VIynq1el-V%jEvPl(;&=&+@S;HdI!!7b%1=JcsoxFu40-B)mE_K$n90)k zHE(k8{x7=)6i4RUiA@IudcAhL8_(&@ zik|073){+*=BZ?ME5}W;TtibTNBUm#)Zvf!_V3vNl%~l zZ(+JM{8ab!t~yozSG9Tv)iLG{0C;bcQJYE0_nU_?kj2!NqdvB#5!@ya>#8})@_){g zf7)FadIt`

    ~jj?F%|3e^rLu?#{Z;hj2f_`^yXZ$8&PZGIvlRBf%DYB1Xq|A%k?S zb5ClNrmTc6{;H{ed?^b~r){E@C-&L3->!pAlW4Xr@bk&@L;o*Li8P_uf93N3j4_S+ zp^$z7dLXBQs73-84Y%4UE&b5whl$sB|Cz)7;~dje|6GFye~tU^KQr$Mj(VMVzUf;3 z3-10=q6!s&VQbQ>E4=<^Nc_joG+zmUGgw!jVh60z*dFKd#(g&nw=8#V^no`EmO6%_qH^3nbk8_T@;cnjhAmA*xgE?+IoCyK1Z1 zvw~j@FB9djy)E$4ms<`DILS1Wcp^yKGb|+mzZLhxpK$w#pz=GJG`5c{q5cb30HbgW z0mG|x5}ge?EuTyNs37!?^3PX}llOy~a<>imK=ZFb^il!G$dLK6V|)YS{Lfms&~Trl z{uvPo+UW<}CLDtI4ygYUqWKxHRwpw&aZRzQ$J6)L?(Fkmpn?;SzCew8Gmi0=R90gqmqlpQ@+)`X>F?-T&z&xx`a-)<^%7{r~V4|IeGv z5`S)k=5P5=*ZqC$(-+V1UoZWKcakaZ_>M??YQO)_)c>Dv1BP<{{~yZVEA;=aRk3-m z?`UsNBItF!F_c5b?}m$mV}E(Pp^zcUGcnsaVLe-)pnOpz5`-TURlj26<>e(IA>r%m zJ7O;O;ll?-#le;U9GKGgkoqKUn?h+;^W*h_(ZMR~8O_*pL$?=9N7sH79Ewcz{T(ejghTwPrUJWRlWaRbEWSGm-; z3DMEzmJ_9$xS#8vJR5ItcLt;EP2nT$^lRB;^wlI~iK3B{m6Uvhf#JS((8$hjHC8wS zZfrAM{R{^Ohm4F2kH~Qe5kls5{xg-G5EmC{9u9A$hdXxDlzIM_gBj{0>tpNv`JZ4R zpNxop!6|W?7|VH(f7JKhzw_iTpzUn#nj{V zDR}lHxCsxW>GpeV2b+f`SCLxJVE9b|4&kMU6g0xB*5MeQ6+4`QVkjr4qmv~|N<491 zZmj|RvN=^od&}5;QArV2bRhKm+peDHtik3vR<_K!{<#{X7Dw4h-BdB0q(btjyYBY` z%pLP!fB&`-m>MUsOonL4wuBzRU_A}@R;|w^X4S{-@e(rc>jRR);oeOUuL`q)&#Q0g zRpG^D-E>iYetxV7uwJ^xl9ai$sB66`hQTSkj``V-%GG7Z3Uxd^JT^zl9Qmu_4^qOm z`+xkdwp}vmj%T%9Y|(9StErt#RH&<|IoX@@K3wjY^Sated-o;Z1KV=4eB=8wifj-e zuj}(dojO(Yp_v)a{WIkKdDPL^l8GmP2JZq4IELl;MD>#_YW}nLw`fO4Bq~Q|7HdaG~G*rM7OeKZSxiUVRmw#sjHU^TPMkeuiBL}L=bDv?< zTF$5|mXXhK^Z4h3N=viLlPxuMb#ZRggSoA-qA8`;46!gmL&zw+B%)fQ%mmERVz#~} zgu~`A+}~d;=TJ~i4i<)evNe`7H&AXq^z8}hWVyN06eFBkbjojRbHMFn6O))@eafaO z<*lpTE%L@nuaQ#1Vy^K9tO=Ou?#_-?jtUe8qerRY8s5ka@h-r8;m+jDDPjW_ESR-H z*QP{BjCs5C{ab$ijj^*<0=@Pp5|PWBo0E0UdlFI9GD%!k?cp!Mm&j-5L0Bv3i!05C zM6(plIT91tO>1jv6xOyzU{$V1CW!Har8dYTvAIUijTS(qs1<8_#-FOU9(tcI`0M30 z^P0Fom@0zL^RveDEWVN5!!jWfP(u}jnecGN%sAU!2GhF2br9lV@(*$uWtB0A0`2(eaD=qE)ByLjciI-zr`(mM!j^l zw9$BLHQvv^H1JUJ_6RBG0*Uc;+L_#$sXJ>8B3NW^hN%_FCcpW(!>Xv!t`+85Wix6+ zx<_IyROhrqX!zjCbC=&IlNvfPsIZ)zwTQ!}kH5c7Xqd!wkQb{4xeDPL&0_u5n%8G~{ zgBVM?8_e#QR!&yxnR~tjS>pJsw&G*we3}9JZhKKVFKlr z^8H16O`kubBGli#`w8k!g)T<7sLAVwSEq!NIHa|z(Q(-_u*U^tje*>p|30WPuMt#; zRWijxc*VOBrv%w`<4TNj7Y=Ib`=~JSY@4~p*4EbGlEp>fa>-dwl&*}I81nG&1hZxg zym8!AW-XzVC^P9v_g`LLUk6dQ887yQ;Ys4y$;ru~77lI5LIoO3dJ^wGcmn)5+$D)l?TiVs*Yh}NEi>)=?nL@bT`cT{%uY!qhZ8{9XmV<5lfBsn057Y|3NHk`EC;?e zT@vm*(EUQYkwRq1z`@}F(rK#FisEITK8tib%R5ztzEpnm#+0NacQ6vrQlLEc-|!Db zMMO2&(a(;Lrz;?F17?(&;QWOCmbb&eREeQKoc~mB*N7+uX(6NUxcjlESLW>=ly^$Pt9BHVr`x9DRIi)Gz#mc&TX@jv{My@|Ed=Mqk%|<;DJ!eU4~{B#`?BqNQe9 z2Z%}ijC49ib*p8^%cp3nIRiW?0}2h@0fEFEWl6k)0b!xo%|mY_-1qe(=zS4zP~oHms^FfIT;A~G~KHom!1o8|`Lkg|)h z+w%wT6{W~!h{Bj0290`Y6cx)_!5aZ#)`5`kinOiD^hOpFNg zb__oJ=IV@wT&v0&ukXZp&Qt6Ff2Ri)rhr~qX;|G~Z`6UUtJdIl!lYB%LfF;YYdu}f z8Wu+xK)uk>(XqI)+r`|(Kuh~E_~POsAm9P$e1?eN=4K(cdyadv{OP7L`gliy3vvP% zA999PR#z)6MkIrs*X?Lb;IiZC&713fGn#k7b~>Y0c^`fBpV#+Ly|Re3Ga3@#9CpVVCDKz$8GC zTU1;w{PqWBp;i590kl~_470|zjJ^0lFzYQw@^jmBr>CbQBO`~0hlg_Hun1Y2xoG*I z6Psg2O)dwE`}_OJ$;pwC*mf9vcFW?%3xIF{RJb^3BZ~pXSv?UpHrprt)=nkKp$7Le z095(P`D#VF{2u3@dJ;Lo3Z&6~79oQJ^q6V%yaJTPuJhKwz~K7k=IV3@29H2f1$8ec z*9QQMhML-9I1vtq1CBe}z$h330K?;GwTG3J_1?XE01H7>ug>=N#!D=!zEqh0oZ2vSXIDy76gf2T;zgy0h~wVmd2{US!As%czWNGU!=ZG4e;q;&cMJb- zm@B(qilFCjm#5RZ+g7t0d2BP@_hK$WTiF@kuvlzwlvvn~xvMpH2a%~<8b~{B=W3bW zmo05P>s>C{`Stm}aa*Wl0I`>yO%oF-KVT6{D&Iq&@H(2cEVT+nA@H@)wkO7e2I0Dv zcGe?xT1!X%f}VJMrP7?M8!0clR=skMvsH>xr}A5M46Qv=2n6X~TDHf=fN0p=)UWND=U*w1mA=L}pc@vQsy)~kDe zRF*@id$0krBs?V``rXslLuXG-(*c5$7D^whYk?17+y$P;=`*6ih) zv$F3$;zGT+EX3)Fj=}_vGhf6AkO>QOU+l`ywuRe&X}p~Nwd$2J#XVW~V=OBCrHI6A zR$p7D2$rDdepKJ3rh3O>zt6LU@h4~Nlkwbkv(ETT0!khN^XQkyEF0fKPaVeN7;8K( zK0fF={ZSv0-pp$+`GA|-Ji=zJjr#!smFrqd$fHN=b2@~Ef_qnAC!H?S{EV+W-MAJQ zp>KP)XSZf!nDbaSO=owe5=c?2>*^5Gmx+r~sqROJc2p$am3(6Txx%7;wvor`Pn)OO z`Z1JwfNmZy+Zfuv*KovdI$a<5V*RnErY4L`P`A=D0qBYOpI<=50RHx)TM@>+rCOkA zHB-yU#Z{S>CRCsbaQo`&s_VtQJPDh5lY;HKop|P^>q7wh)VJamGVk8KQ&x_KLZO`g zHBP4vAa`HBds=wiXr!a^#?0am;Hg zD;p=5fZ6O$Rms)#_bCJe;S25pny*1{R!)wR&b3}p>P&KK?0B|}7-^ypVDy39F{pq5 zmg;n!A1oDV*MKrrl$YOMYz^w~@2|06(|ntQe@_*_AHVzQ*RNln3wmjrZyI{zrfdla*v}l|Me9;A|k?~LK8NEXlrBE zsa5vKq#T<9zYii{u2ioE!L5Ldy?_56pGo^Q^(TA#BmvLM5E5=M4uFnjWo2K!d;#<% zz2)_4KHD@PB5o}zOp3>+r`f{kUBf_3MzEePwqhtaW@(BK*$#KmD(+p{`~Sl zX@RXR4U9Lq1YldICh}xV4@mogsk*_Qxcl9Gg16hdyNw2|fgr}Qv9b9ho5jT}D-=I6 z_^BpX`T2DsDM8Nxd<7p5&%9!iMlO{>w_bh1QzCNPyEOKsu(QY^KsAw<(|S67``3pC zIM8t|sx=?nhI17h#Ff84-`N16Hh=8mqU=U!4vdA}nL5+%cm<_GJfJpm$F?Y5y;|)_ zGF8w|5%A=_3Cwtvk*oLY*|WtxUP{UjJjG%m#DBbpulKs9mivs z$3McfioGA;xQ~y|S|1cLoLFC94+tpG2O7zS;*mzTE4yIK!6obKW)m^}U0p%ZPi?ol zK<^*oQIRc^=#-O^!l{c&OqATPx3i07)|)NT6TGBDdbAwl|Nhf1&V(haLZ0aC=6>?0 z*4Ne5<#XD00sOh($spg9J7tG$HN(o&}I8k0$U4qMVb=H2kkd3ab2l1|=006&+h3-w==U zZt}QqETa}1;&f+51Nwt9K@|baq7?fcAlw&9xUT*`{n08s9!MhBQl@une3ghq~q80M+8R~ zYfuOWbxZ>s&cMsd%fuvGG$tk{c6&S|htFNJIh2E?;Rzer1O=R%`??l``0+O|*ACux zMs-6&L(KaC%|1(JrE6t~W++&|f!4`l$jZtBJQ?UIDl7`XqbeJPRU;@RDrkxi37*(^ z9{C^pn-CrUija#zbLk#FAX`_IK{CZaO`i$Z_`KiX?4xtli8fZy%yQi97sum~jty?S z+*|)W)qHnK^(36CSB{Y1&CHKRqwp|XoK8wwkI~xBaVyxhtcXMqi?F{aq1%(PYk#7o zmc~Z(f;~)Y-0xlKp`nd;gT5Qh3k_1=5qI^i>#TMLRIJLo`g{ zc%8RWk;4rT!xv5ZZd6GRODImi5?nZO)jw~OT0%Kt-z)jOe6_*jCW~dc(Ss3vuaNXs zTY2IGpNxj&C9lGduCVJ`JT=2s2B74#y_L{2GVUT?SVtpzWZW*AueIFWOuw@kE_0T> zO1N-Av*K7YlcNbSZ*y163i=3Y=k4wn&>9z9X~Ur`i8kyNA8TCqXn_MJVN>ILn*z}b zYdE&`ShzjGfeKb5RSsu7xoq$d1pY;04*0Rx&*vOdu>pP1)Ty;KvTmTTX-rGwEZk*8lK`N5+%3V^<45&C} zdU^`Mdhl1+K;r=F2_#PjCLIL@h3gS8c|?>dA}GWFvjWHr@@Th>?&ZsOr`A&n!NBrU zZfy0AVd}_dXIw}r#sB1hidgfiKj_5kG|&OCkQGfALkFxZ0fE%##MG3OuXl(TAHg~B zx;r{HMhlYEODQFu;4yV+t&9<|n}loVsj8|1fR~nj%tOV%zz|Ku)bGPe_?Ux_k(M?f zBqWR9oXC(aMv^wEP_rViczJpGgP0iTyGlBM*h$`ldJTY5l9iPm-Eau&x?Ly8SA|`` zC}uN&pb5W;(=Cz{7ylj{?18-IQE@)q)|jAR4VnGzkA>3>nCsWCXpI*IhgprnVRx0X zr5Bby5RKP4GuxEmpt$)+i;KteIWy6|16|}g{pD2U)^bNwG-5H!Yq2F@`0apd39wT* z$cYRS5)uv$4vcCLgrMvb6}PIIFc>w<hXC!1Rq=JMm{KC}`*;1|06My& zpcez6&$&nguO<&ItDQ<>5OjJX!}R{`jSUI8AgGrGM76Ig2L*?aP@Go;5MLlW62KoA zA;32tL`PEcOe#f^s?g!3Oa}))j>OItLqS0?Sja4#yf9+yQeRQy=B~0`LI5T$<+>F? z86f&O_Z1-b8ove1`pJ1I={|&>MsJ^6>RH0Jm1EEoIn2`)^Bz5V#OtumV%qyW2$ep- zV3-1O+o@bYB^4{*381>YUAz`!Z=%fKNK7H){IH7|;6(yyk4(pxn19MQeDgfLpJUkq9=poBB@F~d*R(~dE zBk2NBRZ;b2*b@lk*JI-&S;^K1NDvKFB$5G!<1Ug4VB|7_FFzbMKC-P3hKpnwQlF74+VpM}Za{@6AD};fqv4k(pp?W9YZ=r%x=O==k9N0FfTu@&+j)3f(X?YJM)u`9(ZO39l3PJa}gUeLgByL%D(?^X~`*VL%;(Paw*GvPA&s%1ZiO|I~ALQLj zmB{5pu7mkkC$RB|AP>I2lu1@bw>K5qH{Dv5MGeAtMV&e=y=O{G7h8pjtf@&)-9-DO)?IOx;Z6WhI6@1YwPU^q$p;K%FYNQ}=pKZJLa$ujY0 zL!VDu%wjd@Y~3%15_auhmV zYwRg&IH8R(yCUAl>$FWsi^1lrqMDbXH#0M1KC=Sa!#U>SQkbCE#UYhUnSr5UD;wm* zWTxKLzIJ7%+V+c0vSyQ)XRTjMOiTx5{$!&@5NQ!x{ilWZh~euq7kF(91B z9;J?3swI3+K4vC-i;GhO`DoZZH#Y}6gpT)Y%1O02(fyW{ZU!LpYf37Lo22g=fGY4oEVF=@iiHUl@=&m1aC|Fon%*@OKpCYkdfPV%238oCa<7f5pfNzG6ANlS%(MQ{r z_vL}}uj0JJieQjjh*cxH;OVK z+H)cxV-uKAqJ2BpY^_w91NURAc7XfoFtNZY{O$3N2hZ~(xFq%_D`1>K;>0XqQeb~h zJIs2roE9?}xW_B>uXN)Gi^oA<0OD%c;v|@eJy*io+8WTu9B8C8XXR~&CcHLr0T}qv z)(NOwz0 zNGlx@B4v=$9g-6X2>}7=l<#tb6%Yp@YZ$6pF6DOHZ&J=_}KC&z;i-|wtAtANT{Ndh@q)&09NfZFnH?p2->{q#j$E)|Ra2-gVAjbhk$({}M>)bd}Zt`nBHvLe3H| z1KBO$qn&u(#&^oRy4fW(vRMDn(}RbG|7zQyJ(`sJDVFbGWPIhHq0A|PcCCxoy{99A zpD+xjBhD@2-6P%~lCY?JuJ3{}NWp7BLryL)jqp5L$2j3uVL^F)x|HNv5QkG(poHxZr%H)CkTKvr;Z>Z02X z_zn~$+r9brjEgob6@C3QyO(VCgs%_RwaU_PNLboBI&!kIoSsESZ$tOU&!WeuF4=1e zN`ezkn(*Vjy)OZ00}AwEOGAtYfSa?*52kFmZH#oKxelzFN51~sD`MiM$NGbkq{J>- znm{dpNpbjuMThCx9;Tnk!dlBZ1oW9O;g8}`@PxjLV{%*l6>#7%pr0gjoBhWOpjecq zFxrw&>v!VTTonM`6NrgdsaG^$m6a>mt$ZXH$zPu8FkN?;yX1FZcLB#SKsm?T4{fk4 z|HBaNsdt|DR$0kuXnvY@3i67Jr$joh-hV6p3dVT?+C>%uA7p4$7G@IDYN&g913AA@ zQEnX5o(qS|ldbi2rQE00g~OMuwq>ZpIM=Y(fzZAx1G$jvs#=kD!O$8xx85UO!u8>T z<7tms>)%w#FB3DRmLdwNg={LV2JSXRBCP~x>OG#G+VNC(U+(_S(*VmNnBffW7_J*X zKh2<^1N?7ZfId-{2bmL0nb}86usM5AH6g5W?Vw$?z*+YI_ak46T`=VY&@H ziM&hV{k?4q8r1SOXq{PAUB*g{FJd#3B)n=YpWV`J_LX`e_3Dq{!r;bXIjSuAs_Os( zw&bV0c)Qi(bQUH&&X}C3+v;cvydDIqNGc(lNAZsbgo>CTM`D;U)$o~-+rjs~q$9&R z*T!aHuGny=OrlToHr6#pR5>6zIgzC7Rc`8`N{jl4xX8NKVK{^0V~CcIrL1)?#=Z z_vw>iqHmk+Q==loX6Z&c(v^dTT!-yLd52n2-&v`>dk)9x0e{`ysTPMCTpUZHrDSL% zJx|W?K5`e@JZuo84lsAAcY4{DF{kMHnzbg^?5wgZW0+~YM3+;B;U`bPaBFK7NKw4q zYz00W-fLnS&+2cg7H)KDiu2^EN%0}peaXwu%{^uA&e9{`WAn)9-$-uYmgQ^CYQlMS z%^e4_f*o{^)NJF7y^9$&f)TNRD%10~T+QjMIbedoP$6m@oBwGWN^)Sg4n zGTGVYS$1OOlNq7MaW>jULfkhS?8tjbzH^K_cD(r@75^4fi)t%pnlE_BzCQQ4UtwwN zQtQ*3p#(gS0)+T|zJ6^qxXHIem!o>S$2!#1XmONGe#u)nW$AK?HFJHK#GrXMZ7HDR zBb|sZ9g>;^+WwRL9A~{Dk?q2TKT$2+nQ=^>-rhrzu@(%K0jdk_{DDuRoayGbx20e0&H+Lg zG07tOTjLf2F#9Lm4DH&P{PwQ$euBqK)z9>KHIzbpOgN<3#e(hm$l3|VvWGZYNdKGC)01?HHPbi@$+HdvL|EDQ1@#;qE z7AxrT)oFmB464vezWV$#JCdRO3-AE$q042XC5B1u@SjgAt!9UV0=(HMOg&Ewtq-93 zbw&%@PbB!pWh_W4&|`i5IYEv{w4HoB^jKAuOhRrWp-QjlXK>&x3Uc!B7<6bkwNhf# ztbxs=5wubRF~la#unBd-AKs>uyP~2em;cVqdJ|$}?^Kj66kP0I1H}R`qc3oOdH&GK%F5NX(x8M8AOG<9xcTM64g4aQg7;3NdEI$9 zIO39pmE#_v^YZb8h}m@FW|;8J&rcmUM=2R>>?S* zlhe`BnW&BA?xcurmD($|BsPobNZ4-8jB5#uvA#=WYPY;i|q9Ah+AOtZ%)CEW8 zM6LK-#{bl@?LU9+n+O^fwkN^Dn0|ex}+y!u{pfMxW(AJh$*1X=F$5Z5N zw>Tc}l@<7uerc%Oez?r`oRrlj?Z6Gr;G_Ki_wqB?t)TEr;=H<1a14^0`^je z{SAlrl7m0Ow5`jv57$S6IT=C6B5l_B#h%_zD~tDBaW9??Z_MQh^JBf$$z(#l7wco8%7%8&hfRU0)F(_N<9r-f8$MON+4I|dz_Imv zO6Yk0!Sv@VL6g7TbiR0X`cekQ3B7|qUXEe_^2&%5(s=)e(SUsZ;m;D4+WTb%1qpkd zn+W&OI@~ue%s{q~OAz(S;Qb$$o$jP5Y`yn!eoJrH%a7S4QMqYpp<{qL)TJJ`tshzT zwUf=^!=0R(dp?_E{2d?UWE4UO>EHjSqjQIuM`8XN8>4<&971V2&s|dOIMQu_h+s%c zNKSrF@vh4mpuJm&({5WmO-`HM#aq$+LZ^TI2QjXxa6#v`hXlrpugf%yik2&bbu9Vk z+AvhOa+Lzy0Kq-LPESaH!F@FMDgDa_oaA!GJf$ggNmj{7E&8D3Pd6d-Y(6?4F>$OnL2{jiwbKmpQO!5oQcZH4d+Ab0cFD5XO#(q zR)YH>K9A!7_ER(i@L5p8;s> z+`JqxfAX^^_s3|DYw(l;%eiRz%ZM*@B>}f)r7K0GH4LRj$TTuBApuA!h2=NNNn+63 zdY$U%>Df%=m-+a;10n`$S~98Jen`3$GitjLpH-FKvuvBetjAVXY}`LI&m5ME@&)F$ zw|zmGQKZe8QZSl;RC2xom@~2WP7Kl!L^*!QNGhRjE5YucGg}mh5CU4IXhRC-CPG!j1g zy-IJ7XKrapj(00CC`fLh!Lj+DcERvqJ-w}CiD#E1=iCS6`uchuh*Rp`{Z){lBf=a> zNFYtw$jZ%)7!m+>cz)i*E4BdIBvU(>u2~e^R8!Q%$fY=S9*4xdutnq4jnnhK{iMQj zn=r6>CPC~&t_54J?c%9o^=iGq)z%*b1jBdXIHzYrf>9x%H!@aLne1_KBX-JOs*A2L>%BS+J-Xb50NOj*n zB{X*NB!0(bcO?d*0^^6r=Q_R5CHE{!Z_-00Gkii2ds|gQ&R$RJ!=5&K=@Y7<=t+%; zB@F}%Z#Zq)P~h;BNS5P>k97Jpv!zpoza0?{zWvWWi0UOA(r&GJkVub#oMHyQ(Y<(! zUHrw1=1LR6{Q*wgLcG@!5sl}izbymdW-NJ>+TJ&aEtbDYv1Tt`PWcx+S}v> zN2ez1_mV`o%z)o#wF;5;iBB)GhLVN;n*EzDHAbJl17u~IyWAt+vts>k8HEw=mBK{! zpy6_fxEFB`)xnFM$MU7(ZP$5=A~O^2w+1^hp8IWjX#6t+ALEeSpXL>f$ciQ*&TIaR z!A2IO*H#X9aQ&cC(UvI~6+;o7mix`3k9_UY(! z=7tG+8bSOc^Ow2N(C}&eFwZ9$w}&ZtSE-8v!&GItsNemmr@OAvaK4sjc=L;T_pZ1w zLlArk*^QL~uTq343)HWJsck?$4+LbtDe7prb;}=Rj#&Ms5(tHXg_f2qy9~fQ^>b^^ zBhWsq!Kup0$$_q9@vhqMp)QVO{tl_#{pCHW?m7>Vr!pu0IJmg;U9bGZ!y_UV_)R`{ ze{>EQo)^ouuf1(LFIIKcl~S_^O_q9EM#ERdc&+#<8A8GsP-R2`1P9}263+_(FmY$T zgaT3?K`F-F^|O;k#DPX|b|1~}^1`#shdNVdT!XFxn3FQ1$8nc_N5!qY`3qrJ zym(lO?<`O;h&74s6W{0J=O9-sRftEBJU5XOysxU3$#fLR%u)6z2scB%G++q>^%;R2 z@_~$lAN@UxsZHw5_-J3t%%PWDVI#&6PTQc+XCLyGLhpC4nfX1P_l z0SWmJ^zfkM=(YF@@=er!-6|Wh)K996ip*%GM{7e^pC7tQnc2qcZJUwtVT+x>2UW`{ z`t5hI(jYnNOEY^+wL?O>rKF^cLv{Vhe5wOkRrb36TMIyCB#`V5z{>mgH-5-K^M%Hv zE5W9IWOx|WVx(4_Jv=;2`7Df+nTLJ}MVsa~z{%I@Y0%@q#H?v&w+SkCwrnn>o|YD7 zF0-4PTj7hFqQ<9ixY!I(>%Sxf-t0hEIZ8{;R9YZjDHifSgn>Ol-S&^P@u^XiWwtOi zAioLf!_>^y*4Fv=i~Zk2a|x*#7h} zMByx}@c#R0zJFkNn$_;nC*pW-LgiiTQXBVNHp|2IkH2e1*FTX(R-~-$(C09pOp8jO z?U4$XNeEr+=@8H$9V1*qIcmSUx_o;x<`n!wl)v|>X#{3V8d#6A zQ+|pf67b%Cjz#f1VRygfq|mfgz@1R6>2EsgY##=K0fR>Ecy9l>pUo@MGM6El3vG1E zHg@w6f7N<%>fHl08F_M}rKvCFE@7C6Y5kj&-yAwv9cipSOU!YynvRq6OW7bqT%+yS zV-$|5KD6kW^wiaS@O`-V$kuaLih2>3(HC$dGDzJ3Vc30kElfm9WGE&=LK9)D6Kxr8 zw;10?JaYE}<7NN8Qf2~IwlI2Onr%<>$We{KoG$6}@F>M3Y~+M{6-KIt>R6%wefzs8 zoFDqQ^=tX=-HVsgbDU|&o8m=9;u444_ta>VXdj&4+TuotJy+t}T3i(L++T$4w8CNs zd?j*@x71>|kW}>agp<^n1GON{ghB>YEryaGO;Q+vj?fjNJ|~P#%SU|#)geAU9_RsC zWvqX~Qc~&lV?09M<~8WQ`Ux-*TD>#GhI0+5MqfFnd|T(8)&`l)*qJKXmjo6Vr(`g- z7n<{DCQGrC5OuEX*_GMZn2ym4o`XfZ>+1Yhq0I_q$xXl305_5(xAfm zoKU{}0%{`Wr?5~Q6VBq|Vp&Y;z#A-)`S!fQ8Mh=^HPpjV3Qj(R?-agc6E$ib2*99F z!njw!0`>ET^6ia>EL>bSrERi@W&kSqE?fF+6vc=vi}XcZ0oq+B&CCN)c13HLZpZEV ze*Ro!k@ZF`l$G$d@YjD+Qn-aua3?#MC=bU4c>jv=!i#OaDx<2J8U^LY_UqUT>et_1 z{dT!JkU6p&`TMt~!=1uEk{_Z}$HvEt3>vAN2v0(R#{e~y5a{MZU-m=xolk8Ue)5LR z2DD~r>FErbEGtWIxg+i*sv8?s+^3=mLr5%mx(ujUe2dpY&Pi$wM|I_X^6f?~EXR{r z-&L}Gpcy9SpGb6zp1!|Mp@2_|ORw@P!2NuHO`@a_|E`I^5d} zV3z+DA>q18gjEVUF|-*yo##K^Vqsci_{ti6os%3g;zO1fd&y??(mufMdtplx@H8;- z2O>8$mAW+XFw{o%1u$7?Z0VSRf9%2~#)Hm^zF{Lr7zO~|1-86p4=+36uPY-#>8%G; zR8(x)`FdQVU)LAB(GgnSL#VvO^wwPVwC(;!-mWew0s|s$_N%=z^5txlmi~{i@5^L` zu6NpHFAy=WQq`P~i#p?)7Q0 zyh8F2f77nRUR~NK#7&eBu4CK^CK47T*}#9_+nS@R_H%NG*pG^K5uX~?&ycY{w3Hv?U* zpO>19os~E6uZFa=PNPQD&>E4MF{*V(D=L;gf3+%cMg8#mY_FeLglIP6QSz%72P3&{ z^dT94bf{6ZA9Nn=%uX~|Zr*-?{q4MF2u17O5yKz4(BdYZl+UhGUp85r#+#Sy5#Pdn z5-aHa{QTTgd(jGE4S2>>J!VF}lk_GHZ*S2K#l(ktKz;K@Y-xTz$61|8RHwKug3&So z$N#H0S>gu?!YLw-_p9#iUoHpJ`Vh11u#zjVORAIK}l%zW-9BY?Ck% z#>&Y_sXxBa`XkszP)5MP!G4aTWj$5G6b#~6P)>hGXefbO3agsVwu0ipn!1v1)C+eB z6nr-}FTG>S?} zbhMvf2Po%kR%5f=q^QB~z65DWg8ImnzfJmy`eSxvG z3hr0S$;dQ=cuHPgKCr;h($W$Upi41m=S!Y-3l)K?qTlExq=O^f&Euh7OZ2GC3zSL1 zAvoC)C=Wt^e~&hz^%xH%%2>aPl6u3{8whsv8#hsuZFbl6pFi(O&nuO~80xP2-QWLI zi?Zd7x4Ey@DI02OAa+&)$7=>ufzG~mY7qzj8C^}y;F__`%}qMVSF-GH;{QSuo0ycu zXe;|ey)k5dV}qKUJm=tLL_;fo#f{QH+7n=VO0ijxNqipPmO+G;pq@RAy8gMa?nZ82 zJ~7JNdikh+GF4(m<}doMr`|`OdTF(*S7YTdY*r*(d*Ll5cRsKhONhzAxpA#V@nq{W%Rd`<8%swA2 zJLdhzNJYtjk#++pdYlL+6wXbTG3Cq;8cgrf(k?8XXD>^VPBE6Fa57{)d?Un$lb?dp zW(_6=voJ5-?%GDkYa{e-p^XcZ7EPMlHH`WI#8! z1m|aGc;p&-bK825t6fyc?hdAa?k%{z57P$m-g8(x5mVNX#VX($B~BB_lF;Iosw<5@ zWV9lFFEKn8M6{k?S2tnA$E912@tL`8GuCm}`m2Lao%6(J?EQkG0Z){*Q0bc?qF;kf)8YSxDMdCskK5^=4p91K61k8)!p<=Z5w(~A6uNF(JemkLsN=StiJWg7@;9_6ex)$X?{UFpO$iq{Po_5 zhuJH|(YJA7p?BS)BT#}`wsl)Xtt!_BvRaHCRn>H73!%Y2I>JyD{WJ>}SU1@mOw3vX zQ7RWU``wX4bf@2AU#2eqJ!eJ^wSKvKWlxyL-eTcD+%r`6<%HYY@gsG>-s}tgo1#7< zebGp|dF6Nf*Q?*!t<%x_?Nk@laL)?8@zB_q3C)9*c@yNH5rUH7@n4JMebH%BqNKBq zSAF??ZQqBc8)n8a8MCXIi8On~g*UaApDXV&yPSO;VNuM3s^+^=R4~*XSnF@#C}XLh za86P`bNUeZv{e}T!8kWN%F^v$M>W+eGLCA-cTzs9Ykar-)U?NX=g{NOjH}Z5w8mJ! zMaXtUP07~A#_6~d4PKsv%166x7G3aYfjA>dL+KmI3HZPRjh0YG2*^f03+LIOd4BU$ z!g~XMqf*hhT-@3!q9RPMGW;NDeTzIHI&U!ZdQc}s08b*d-bi~%?rcs5SzWz^7>8dZz#eXMa zKqnhP&aKYTIQDpCfnj^&5sG$Xd`HXi%*@Q3XLa!tsZOnD&xonCWLE}M*d;HfUw+Qb zwKOrwSbw8G_EOUOC^GxSf0dhuK{xON+e?RO1t!b1hp?ZOQH*^dz4JQvNxHgr;FsdJ zg!BZ@cD@lAt6`{IYKcgLpGP=V*Yb1sTfrIG-}*|{R%|;BJ>%ow^7F&haqmZdiGxBx zV}$K6yS=`S>m*YngaDEqnu0{>A*1+`HhI)XM$d!Y|HRd{o9yj*=KgFAE<2(J`(M57;ZN=}tl!BsR1q_pGbj934l+M&+glHD5$#UPGy1Kbx zCZQE(K4w?ib2tZamF@ug47V{i*_1bjb=gr)qdo=_HES>$ci`(2bi}0XcdJ@#l&PHW znqLHSD>_!9Q4j-cT5kAh;H}r!r@?dWdy0A8{d;02qb8K&8TmYxAa5_6_0@6#^WExpTP&=^b`4?IYdQr zNf*;<5IsT#-GY~>`;={&$pO2Ne|p1ANch{Knv;{0*RTnAEc_>d;A;UxFFZU_6ciMTaz;jXxMuV6@;2L21k7=1 zOul~fy1#clx&^DrapN$+6U6lraG7yDRUjJPwz@nCZzUB;?X{EaE-fuxU$_56MgioCcRu<>ruNKSS-{vo`1t;+bNfBE-Sd<&* z@w)Z3j{3M=@T%0;q4D2~x53_}ql3f-ZS!2;%UOlcGKTw=HV>!wFQ*%3jkQ){+KlLZ z^-08iU1LO0tGXA64eY!G*(I|$??&43aoQ+#N(*l;&U`(a$64)+t1_a!wO!_^0k_Ttw#0&Cn^uJ0r zem`!qqWAhcgSA={UdLX39renpRsT^)tOts# z=~|nXAyy3au#~Y~|Ft~yg~Y4&n6<8FxG|ewUzTlRGXPXT zWHJvd6BLzeb-L%i*$eyhq@>Xg{H8|zBV%K1R<@rv{eyzgKfDJ4<{cUm3JM(!4fG^_ zG5@NEZ`?qMibya0@?xMYecW!D;0UHMPS5OrH(c9b3p-LzO-+67;J}kz2@JKCOcomq zV=$0dE}$K*4xndA*3ieAFbbf$n}C8SZGb1_OXV}}mSs(na$HKCFkxi)adAS@Lxwvi z0PpF_;gu;L5qN`|#eZdL{2P9vOZup9uxJl(-V1!azW>gKM@C?@QqjP@e(~RD%kpFX z#)bwS!={_sqzURC$9}aaXb({%+P^>0f%Wk~#^qHqcn0C1(E`!S<|>0HouS_@pG379 z?e8aazioGg!vKbdDU4rqq}w_t+g#*5fCuTa^1%#(L2}QQT5(sV8eoO@@5w91f1VK}M143&W}4OBiGL8^0z$HtVBlr$1Wez-9U1LybQQAjbt zHzo$31$G(2IN#_vD42-P(i1eswrBrfe{b>Slm1Akg1Z|&Zub#HXTntEV`DGqvTTbW*4`X~!)r zx-LG97C*4vf0cd^6cNdM>;FJk%v+0{Ir!w{1le8j`rO^DJ~x-OQz5>qty^JJ>-nuM z%u*H@U|0I--nnm#6xG&JR-aBtY`9k%*_Ys$;u~Rhj+vAJ78dH#6ojIEGxA0>bD0d( z(l@p{T$HT!Q|)bS$PeYjePD)@)nrdM4*c5P-~ZzcbhZ8p@9HMRwg_de9O8OWN17v!Rocz7NmTGTC+F>(oy^S4`e5GxTeWG~9oIIDo;Kit>14ZyWgz9-C!pknwa=F(v&d@Y{?+hT%xMEkXp+6kSM zib{$7&Anj|UW4b;5q$EjT^qSifaa@vXA*Hz>#;ra?j?wO81zlZdhzXlMrzd@Xe+0w)=`<_z|@VrNH3$I>M` zD=Se@N7&e`gVQ@TB?WZHSfgj6@oooVum6010|0t%R@PLrUvqvwu7mQ+%{#Ulqea!y;euX9UGhXu+CG5v4Dj3L7E5U28yO`K1giw@HzG(DHEJ!ma zo{9dF@IEOC_?m0|4p@&~pM9lH`LcE&+VE%3-VInaxU59)>!YQ}zU*_5tR0c~Gtkv_ ziQi_b^f@P(esmXWXDq`0oX_Z#2KAkfHU{Vj*gKhY6fnPlamv?T>coxR4cajO@%k-Jq-=C6`7ug$B|M{7X?$Q6NT#tN}$;Rm`hY z;BQur(97+DFBgi6fnElyEnq2^;8hV45~7J4ldsAeenzB&JUiOL6S3YkREz5F>DlWA z_+gBX3!FOu&^$#)`n={C1z$@x_y7AQdRi|$0gB5^cF_A>Jg17_JSsaduWOi>M zCE*tx9xkphK<3U%-8)U^+W_YpF(cn_$Dtu`7(gHJMvLv0!YeaevzPH+ymSlc*gknKP(o?%6Js7P@<;c=mB_(OG~@XI{-PBtYk8*?Y^K? zQlt|0y9t6jI7Uin47p&N^FG~Q%E-72#WtDG)b|~R|4ZHrK^6kYow?nji+MICQVCZ(i!?Jqtay8{c%a4iX`CzZeRBZ?a9Q_Q#>;CNnu zB0YIRupO`aNmxx%T)eKS>7)AD#pz!VQPE4d-?Wf3HC+fLZqRHumBTi0h4_oH01l~L zlUbuyIriS39;hJnZ6|Nzxp|lb+uS6b7drtAgA0%v7grAF1#v+w3D-*F5gZ(LC76+I zmDV7U0rU#~rga`W2C=2!>^9;|n$WGl#2mVWx0RWj%R=^v zw!HqsGc$D2(LXKzf=^ec5K#tFm+tOv_*l_Du+4QiJd%?myZlRNv;;j|d5Qh`*@W$P zAU6JR(Kn88q}Sxn@T}i~DGKd4`;Oi#V6v5zl-%8gS3{H()XRHlF^Dn}X23KKI0~GK ze;4dw=G*oD2%kIRhD|=6;N^qOGwA#U+`aGHeE5SyL*r$>2dl(BLE!E^fzA6+)RGUC zuXyk_zXdY*#F@3gUo}M#N7PP`3I^|H$jX6L^qh;_iNmA}QA!vat2g%V;$kkoBtIYY z57)^>@R6tn=*un}D6iZ?%puICwfWx>^;j|3%@d?H|I{5YNWzrw80mWP5A}9%nEHzs zpyYw9HcLLrIResbQd-O@FcL!6Q|JMnIw%t%T7w2L`y~-OxC3>(#mV-OoV;CGeHikf0-LX_v?d?wF#9_FzAXWrh zo*l4RmE@g{f>f=otu1jnYc|N?;p^pXw<=zX3a6pnck9teAemxxSlFL2gtm4yBwT4v zb9;i{ZL$VVH6MO&ULMXX%z#c&i|-5Il!O(jYiKw-HfAN9`dm-1HpqnQFaV`6ICg3_ z1bugJ#X9j-&Y1CWF@ZI^v6jc|%zfX8#DpI!m(o*IN)NDaldrkK3_X zMaChpKlxAph|eFZNXHnF+wXa?ZFsyRuIMgL)8jQBD1NW~*i299#cKOId!aRs{P@tx zBb0W(uYg6bqif#$8fj~5X>k~zk``lqOQuibb~N{^Klh-#hPBXTve;qA;h0gTokWA> ztm&KFWlhx`A!g0Rm~&q_KW2xGwLiNZi+^|0jw@usZ*QJ<*lzu+F|ZxwS>v237RYTe zI__<_({OoZK>)}9PugVu16}vea#N~Xo9@nzq~dq|aYhj0dk2?n*)Ii!uP2QR&dk}q zj^;ibDkT;J@yi(HB?1EX$q_ zSJ>8q!ypXI$NQ?_TC}%snUJ`)SI)0yVdWA{wcNeNGGcu5$qRpN?KEG}p`}-ldjhrB zUGx$;aSriCz&4cUv)}wRYUtO?;8BiZ3z`U^aboJcG&D54yu=YjEUc_S>z(4bC~RAh zw-Xr=;eE2pP`(GrX@xE-?_+@*2{4zkaImrI?(9?(=a0oiFqphOaYykYBO*dE!ad5Z zu08_6{^qm`k5^jle`^7%ukKo-AU;WY&V!T+v?j>8JL-RdeoeGsvr9L!%!-WYS!M`6 z*q%OzWsia{+h0&n09ww%DO)E@IXbMHluqlv**ArM`)!PHo40X)E~H=MXdi zZC(Yr4dOW%vG1zJHUo(Zr)#J7sFO>XJs~_Q>do_E#3|gZZ(76>LmiKRX$_7AX#(Rf zR2FkU9uFbY<3w-sC7Sq?M-Bxn2u^Eufl2f0LM>)pJmFHw+~(jx13PCj>=gR=B0@4$fLe2jgO$-gWK>jFr ze25zIt@$Y7retJhhTv0=0Mep;-^@;t@6Xbb5VXGFst4fWNiuoiADQH>^qQgl6N4u0 z=~A>UcXvmgD|v-i1Tp*J*lsI#sBJG^x(Ofbd8|<^j`ZY4a2Wg@&8Dv3jV&GslJs`> zFxgleY?o$3Cc3dBFS~POXs?cz1%#L*Dc5^1f1g_4te)9tqI2A<$)=uepur0HMo7E( zXEHOw{f$(eSo_VRP3qPB^djzboYg}Mf|~>`qwdN!e2Ou7K|wEOrHfndHkyRIdV^6f zCOQ^&Fa!{)BmHPoKr#HRZstU9I}zC*VzFOAt@TnO^|@Nn5i5N`_m!%D z1ZP3JIM6dXPW9AodwrX}6_umNTl5Pf;&^4PCRcj&_Kzm2M)TLD&ua{d~DBHm#xn@32?5SH+^Eqao4-ZbrUN z*m$|RU2x#h(GfIxME7#FOeCIgztch@^5q6Ue*6fiB9*kyD>E~ujikp}b_Jf|mClxy zE1)t1S`wl(>a^&cirf=>K{QE5OG`^k+|mNA*Aw77Er*|=Aog-Bjf{*SsX$Q?J2j-} z_E1w(6JX##4?uRv=HL=N*6@cP%fY|hQS`uYNF(ZGq^PKP-2xU@q9enf#e9~5e_dY9 zUgV7I>S7lZgv7oazpKk`Ro#O2s(YehVxSOj@8}?a48p7xh%kBQ_b<#0fV=J4WC;naX_iHz2_U15HyIPzpv44vxEbGlnn96K*{KVJ$qL85tRs6!TI{meVEDuV;TWum&`_ebc}(^kE{XE|?lc=5nxd05K@{b;ca#5-Gq_-HY!ZiWW#oWZ|pFhNObac0G=ecT4MIVIg zFjq~5aT>urf%q+uJcUO`lix>z)&lSP*0F+G2GYP<$dEsNSVG1p$m5GRlIIIIex+5T z6wcF5fEF?P0KnSFs3=Uiww|7i>tr-;Zop+~mh}`He5G>xP^m?KO;6(R?mHq*VR@MufnY`@6+ z>6KY|p8313ZdR5D-EIV~Xjo?_^>g86rb9?~$CpM|9y7sWLwdIM{~#hVkUyzt7}@9n zR}K%)^mX*Zx5`O8&y*4a;%ev57D75uF%XH%zVv!JhnE*JPADka=3D_+SQA2482-a% z0@jNIeuHA4U1V4M`%u%-#j!5sXjT!>tCoZ6wl;ZiYARA>9!92qPqj0OGL@ZmKYw%y zVSEgDxsd2gqiyPTg$J6*@)JU2L`S_ZHWuTLcLOt&J*yYQIF^g{o4bEL&E8PY%?XA+ zCUBxDc@)dLHU+2lcSrTXyg1ejBi0Rkrx?0LG4uI)N?X+!7R&>i_T}lf1oWp%TOwNT zWcs?oV(cs)z7ZxxRn`Jrg^q;~==tJBM188`!-rn)`u_M6l)4jqI@HDx`;1qf9j^Xorng=|hjrzI#2?A=WhbEyVN6zrm~L)5g>UodRZkR)_%o|O4_ zK!U*s5n|4hj^EuMeL55u5`|-@`Z?gY?nAJOQozyg=C75=eWrYUJ6!*BCc8UJvX&@? zSKu9yx*~KziQ=JFW8Jyqa7-f^N~;SCtBi>lE1&SRxe_mVr(sE z>LS}dBDxp+{+XBRSbB1lMn!qCr=!o)7@>~jD8!HW4LxlrGwYa4m@}k*y<1aRNh9Wrfi{Y*o@IG1dwrnvowBRbDAxzNuJY^u?{bSH&vs}f8dvP+ z{TroDeQZhqu!SeY|JPsp{9+B<^TSO6p_!e@)B9b@=Y5}U6VVM;X{2a5GD&7_Y<4w{ zaN!y>vb|c)^(}gfr%@v3SKu=Da7p&j*Yn7y7fm0QKk-%Px;x!VWnTrH_x)%1Y*-p& z`cEauZ{px2L!t?Irdd=z{DKkDy->F#|DG6V;iZpDWLi#`(#J&_i>@9LK+LZxh*JNZ zRbBB9H>V_nnDdi_M{!l?*p1MxzDuMcVP?ps1f&n z9+3b0asU0q{I4%u%3QCFt4|BX8b$xJ&i|jEFS{Zu;WXEbUJ6-I6qko05Ju>36loxb zl+~RMPcT z*lG=crb&%rZx+zIny14%dGE%CUv>_;l+QS;4b~LY`tQ#ABCT!q@*m1?_N8_PWH+bt-l+j)g6>H z_e-B|HkQT_&9AK`$HvZWY@9cVBde)tXwu+stE-(&KFw_sypS1?_6#C?M>8Kt}~GZVvV?}@%X9xg6BFo!`2_^sVOx!Hcq}n z3Ts!vEDidc*Ps%%FI3>YWN`XiTYGKvqn(H+H2aRNo|2M+TkvhY8TbTK&tV*t4On3y z>_NJ*NJvOP-?|m{(bte>mBwGo>8Od9Wp&G4@cN=;WrkIKxdhNeUCGUV_{KiSWYF3e8{X@D_ z+dPPF1K88AbJuwXllt|GS34+$Ti>IFwf7+u`n4#EEy%J%Zj@HPYxLS93_*vkfIK2Q z6Lo|zz~H44SP_-kUjxjQGclX5Re*37r!rDiROCzTO&G-0XooRE#D2mRu$bj#xMQHh z0`g|^w$wiZ(O)>A4&8jsOY@6*3xmK%&E_%dFV4weg3@m1&40_IP$Dlk$P4rB1`O9} ztPBaTw+o?9H)!@%Wq}n7i@#?S=g`$U~sf`kIDH|N7t=Jp4eu z&gDWX^NTvyyquhz;_7Xv6Zw}X;4x|fHv@1YPwzkfn3EFF56NF)+$y?o{#s9j-bq)4Lo z9vmA}WWPodCwZwG*k2yDERuvJ^0~H5qqv7-C&$YA;0Uvw*_ks@s z`eiOpTFnHcUY`lVvSMHeAS#DwK={>AJB@p*ncG-Z)YU;VLvERk9R;AaG$*IBX$sC! zZ*Q+XIN$-H(1MRUJOqSPA(|#EI$E2XM$X%?Fe$DXQQ2RQxk*|((sh((DX$Vbz7!j25rV0d_B z`kRX-;MG7h{slOI=mQNUWhn3XBR~mw^=o4j6NSL~{OYQU^%HW1o|Lq0nJM@JjP?nG z5LM~CBU7o+vQ%u?>B-6U2g5V(bvbGX#p4=FxO+Dt8P(_?EJRpT8tUp$ z<5cily53?S%hvA#TFVzMLU?$1U7g5XXSjU%(ao3? zs3%Vr78c-^nr051K*9^C4dC@+5UUC7S%c4Md1WOjxo5E>jE*pAvh(m17Zeo~7XyL{ zy4Qm7Axz=s6XzfAZ#@pej);sr*x#Q3l|`4CKx!}8J8s{;oo^HQ`@;k1u&FA+djZsf zrj8Eer9xUO70M{V=VqVh0+tah9ZV_C@ z*thwNknnN`U{Un|4|w+Me3EGiyTUL z2qd?PyMC>#geM{wE~8$(yFH*@IV#D(V^nw_JJ_lcJrGATBD;EGZ)6H@YPgT;4eJjy zDL{49xvrrR*l4~k4fmrGw%VNUN$(&+oIg%e-)-?){lyT}QVM1)*uh00mV9Ic`j1Fz zkv_eD>O1CIy--UaIJEA&<})~6$lOU!mx5Q5z^J(;FIU}kBj^w8czr#+a9wpDKkfqj z4+89@ef-^EuuQA#lmb#ky(xl__N14h(#8g_)4&L z`4L@vlEnB{5dLi59cn-aL-}4R$(`J&qv6yqOays}5+nx#(U5xIT?I2c@^@byJeZ#Y zICBqzaY*KJs|^`RFxU{wK6k8NV*b1#HPY2(J&+*>g~1mjx0{%dP+0gl02VhaSZ;#` z%Gg-2akGGQ7l1w}Xs{J3As?y5$VvAo;g63;;6G-5ly#Hgky5Z5k*skL9l(diPs-sP z9dfBMfeGCXK8q;Yd;{>9{D5M{mJO{ORD`Oks!{{CAUAkTjhYgm;GmE^11--(;frG! za~G{(PzJO#Ct+lC)UeTu>=#1G+Qp?}Frx|9DuOJvlzg!p(&2eU2sa@b_b)8aT^Re> zJ|rZx9R?E#n0J$+sDKS}n(%Ith?r2@Kv6&w2rh=UlKA5ehC$}L-_ zO5hl>c5v8+vOHH^GGMi6rBJz<0po8vLLxo;iT25ABd9~Oe|DXjh z$I;Qz*3VN=Ho-RxkQ~%zNL%*yKVt-MHDNtH5+5|7Vb^L~K#x&Qyi-kXP0 zz4!g2=Bdb3rY3e|PBKqPQ7sB9!y>Uvm3gQ{rj;~_3@t+$Qdya2%aAfgAq_H9X`lhA zL?z|C*51$aJNJ2>?*Gnpo$FlJ{m;Jly%o#%`x)Nv*Yp;Hz^JRM%XvP9N(JE!{*ifY za^(P7Wmg=MY!6_s?U7SY??wbIG7iD{i~hu73lyEiaFtqS_jSo`AVKk?UB)$iM{(Sq z%u=94FBO@cN{)<2Zz9;Qz%C$Xk0NS_ZGf?Fs>e7_G1IZRikC8%)9Q;ciY%JGHN-l1%xh6sF zf!IpRiEZeBgG15XnY*d=aD;H*|FF7$2m1*v+_`{~CMG6EB7a?*$i6#7Yp3$Fbg|RE zq7DY_1*V@foe>Wlc-DvL$t`&o6ke|{;l0GDhGyxlTVSCWqCs#@1P>P%Da|6pI%ZS% zrm0BE%NIaQyONDc{P@$`tC0r}9Xho>udw*Fu>4`;FoE^a2>Co^k0K%% z1RO4y>Z&RSR0j_);YGL4WrTYlr&tG>(FV1SzI%*wzleh{Gg{}Wkq$<#0c=&@dRyYv zGezyn^wDxXk=>@2*bFGBoiBaCYQ241FYVH*3vI(U4Ra(Cb`tt9NP;1WcytYAH`X>@ z{KsnZGD@<_0jS@X9$E6 z{Bx5q_rr(*298m~8a?RPX}5>MFZJ64PveI5HhC-_aruWPM(2vSE?uX( zCz=co4T-gfBS~9)2;Raya{B&bOcMg@_Q^^qyXOS;Lqmznl~HwUWaDN4XDZXhVrJ`= zhPeYMM)g?_m#iI_|BAl(ImXY^RJT9%e2ZvkeQ=Ysk>_TPqxY;X-RFrQIOxdjjzUSQyOS`tE@| z1$2ANqbIjt;+8~(I>;J@We-Kvtpx~kQAmUfyh{?{KtQ+l2!|VvmFLnlbykKIC5C zWyl*@gcH77DYq4rm2D1u)Jh}T7<4{#yvRSjv+3(P3}x||=Mz^uSG+AufRg_kx$0xn zFzB!cqcUuOYY}RyY)kL7b9lw9QFZVzt7S3lakIUnqYfX>?Fy@74d#)R53~PImTYmaO#j3)P z@Y@}}1Hnh_#PRn%>{g71DqDwnkEs$1%Ixbu4ghC?9?xPs_yc72QF3u`9@K87jfjww z*X%PorzHX7#mkTP!LlawFuc<|HY`OISI-MFj}_s3_n zgzJWUsQ6hP-O1g0&*O`5Ko6PA$jkG5C5@|Zs(yxJ3p$Up^mjneqoZ(wK|~q#I%Wk! zMzs7u@9&6TbG8HzTs_CVHC*m;4Q8U$TyHV{`t=cX&G+L4#k?&~Dn(tP^yVg}ers3T z`Li@9hwYwaP~D|x2D`#@iQXHccb!+UBP1FZ%QO{``-|+vwO{tDdhv{CBZvy=#s%=ulGkE4=wj%s zO`u!pHU$ld-OJ1S3z*R$flKn$2}Q{rHlFtxYbO7m4K0?+HEBUHvWb_DJ~}>;u|Ll%1q{2~qfb^BqA&k06w70YC zD6orWtU;qg!z^W;dqwSd&r9d@sjaTCwzXC8S!X!=S|fq+=$SXTpc!&77Zemk7hvu8 z>r~46SEtmsY(aLUTU*RZ?{N@e>pxZP`L;h(#XGb3CUS;Zg#N;b)wR6L6~(;Z%9S9E zKQM+Vp+a56$^5sB)WOQ?D_{`$R0AfG94#Z0jwL@>j!0Eu-S;@wk`?Y92C7Y7f1Xp2 zM7aF!MdQ}2V7bU&hK#cV0Pxw(caFT}(+tAGzex?uRV7~Ku*tf5^^r%bf3+p#r7sx! zMyGXcBeN~EF(W9yff%K`SIh@~`!a5-QR(x=GHkxWhknKP*!-(3&H_OvHYPHS%Ch3u zuU{u=dG?DKEJN)Y<5M6G?&DXcwa~%L-Ixaofs^-j=cB%k})!l}zPi$({UM;28ZEzFntIkDg-JP8kE$NR2h=EF&K*3b>D>cb!UMaI6PItXNEzPXsdY*W(usrF#Ecx?!U z^HUnH&01fAkP!}ln&Ku* z5X$HIOV9{-OjGRX?ou-^QkfBN#(B4Q0Q)liT+y5F}NGh@5mN?M-oyNxJDPGb1eVdJSh56n; zeYZk#hsBtO6^m^y{0i3-bHkZ!&?8f}LaX>d13(r}+Qb!gufcB#t8#i#K3CV0k61#i zVr9ivbG(@r9vrN-U%RAt^}92NBr=TUcg&AHJo$-(S4+};()K1QVn-5*OS4{Ek$Cxs zenX=DNB{jay(?$3FQM__zh*r5w5c=smDsnXsZZL_bFb#_&2rY@Q9fSXv9ozdp+!m~ zern3StHNVkiiv3w&Exv?%E~Q8i| z*12Br%5^qNK6?(pmOoebUaO(a>=)9}nZO7=?OWCulP3-S-9NDC=-24SWrAe(#;XXc zNWL`P3Vn$jZPZ8Nei<}C?zea38#6t0vRkI=E#8}i#D@Rc71_|^3v=IP%Q#ngn96$< zu15#g)B_*O3Zs{CvJr#JUb$;6?-DLN_Sc;;sCwgq`&We$jr6eXY`#}`>ytfyIBl8R`X1}oT z@NJrubSaJ4d@A*%iO-At&NDs2lzt>6Nl{i=$lv)+pm!euBRdj=yC);>O zyfv|yx2%&WU;nwBos8qHAZdB+2ihXH_jh`LrqQ{~%r)Lyt@K41vUttx@a~EtvH0IlBJ69p;j!sSN1$X_?bR;ql)*yMXgj6o2r@xn*YRIQg6IDny^#*UMr^dKx^JeMjB7`g$oP|6BB6N%~Sz zUax_?7UarAQBQ<~Fs}eU0@#(?DeummUx1+zuVO40zW9B!qr^_9Jhb7ReqxGzwL=n6 za|xZN_h9HG2m*BkAoyQ*JKfm0HB1;0JsAR*RcC*O|3x2tSxxXT)-9A@>(i@VL5URIZPjBRh3oIFV%wzi{SvP$<47RvMIa1x z_^@EtjYxPONn2%UieVh=VK!207@gf!s?_rNCZG$&%5sm-LJ8-nZGx+sJxul8^OO8i zbgriN=RQaZ)wWpmkrK8}A8z(+O}ox~l1oBe_Vg*vRb=|hr|YA+Jg1uU^T+wbl#>#! z3$IieXWk&`hUyZg@6+|WG%;PTfIILl8vTN~Q-#O()>dlMYZ^rL&K?utXhRqEQukL4 zQxT5UF-@`XHDzI<8s4G_PtpI(>*rwZk5(w3aKF?QiqF+E6%zf=A@9;)DvElMuP8y= zA^Ln+*lDmh!0g9b2WPU?3jh7!WoCzm?GsMGnZZ5~V$pwQb8?cZgTmo>+LZRUgD^&@ zTxuqGW0b{IzFw-`nD(d}i7dJ|sP<+lV zz$WXRcE_PKXD$8jGLenKvPY)ppK33=YXt=# zP#-}e*{}HI(-9x*6||RnR_G0BtOr(f%?UvQxwlY1&r+egtaD znhMNf`d$H1M_aeDM!MAr#BFsBrKhGJDo(;tt}uJ#=3SCaSY4}+wS@)$s{j;-N5B`@F0Bb zO_ag2fJgQbM)=Bj2x24cRta(b#+K>CMSm_yjWF?7;=2C0>R9(|mt&{u#4FPBD7&Zk z+=YGHT4+idiogq(682C$3v_HH)93p|%2$rt+<*F1$e^T}wTRea|LH0DvLV;!*;!VR zrGBI1o--|D3k*qZnf-rl4327AW|?yxJRLsA4k>k+u#t>+r14T?HnBiS7Httx*3tdo zK~ZVx(nPwSERD>}BXS8aA6u*QNLq}k81S(}ahGz?P(r5|fFY^y7dV$%+V0P-G8D0G zs=6mU;^TV#^(tlwVU=zAt4ab$^%-pSDj&5t)y?`}oKx>VKM8n8F5h`%YNd$tYU5#| z2{3IIL*`78C$8G6+>*mgg~BwhvKYS+A7&B)XbU#5nzd5x*)uBTPM!2uC+Rv;yw8=& zJ^NZVcNp&URd>qD{7dJy_L>^}`nvD2Rn>I-;7v&-RaRs^n(!*5un%U6dA0uCpL-(ZO{dB156mJ=$`n!@1b-d?08FM2ppRxW47 zu$7zMTq=Cll$A0YYP$C=L?v8N$$QboQskgHCb0bm(}CfuXOe>EJFneObKwQMuxIby zhh1H)rn}#noj7qza5FV(P4mmWM~=)g7UpeorTNaMOw@ax_7-u7XdHBPCA z?Sy3Zi8~Z#QI==8s$cqGIb!Xq1Ym>{- zD{q}+(^U1Q6Cv`C<6_sW=-k|9P5+4c3Icy<5~n#?3(myhMG zJRSnK>z17sorV#gkrneuIz{;akEPJ#Nr8j>4vE+oUS6)(Z-_7purJ@u-|LEfTqy8N z{HBm*dB6{G-5^Dc+$#w^u8d}0m*-*Y)= z23tIl8YI{hGsxCs$OYM%lPnGFgsao55G}_GuOr%aTD?l%DTr9 ziC7Zl$7;&}q`zFwYLb+tM z%V$qTZiju*_nD5DuGj;`WdFcG;@$(}<^26H57@d#Bnl}`|DR1j=*@>MOQn2s6AQ6Z z2GtE>kM#!4$Svw(#NsEG*&)p{FK+Wu8%Uh!nFW2L8=#VDd-%u3zGtV!~c}bg`uhF9aVsXvmlqOs#`AFqan`*Y2 zad+J)$;{_icZWB1=WEMQ0ZQM>@Y(XYsv8%W#=nfeeVZ%Xzy2fbD=9#Og$8fcA&=;) z9%Ga-985*Uo>QHC{6O${ABp~jFI-!nTa1ek3E1WdRRX~GYf8(njGnueV~Kfor-%rkPZbS!$C zpvpBR_naN1TxMaZPvU~-xP2(rF+F=(NY>lWZdc=$ZrUKr@!0R?2bv{?TV6pS(W`(P zc=`Gxw3Ef2`n4fh5Im@E2h@S-gI3oZ6UF#ftqQUj!&C+PA+>KN>W_Ke2ne$>hQKK5 z5Ngtyc>I~X0s~cdOual|#Kp{cB$K<$@z&Av!~(YNfbDraW%8jV3nrc|{6-*?jpVEy zx9p^w*XdgO*v?5$J+R)&3*!+8W<()4ZyVa&7S+i%+Dd#T&&8uh*|gb>R|5)s_WKbd z2iVA&QQXUq(J`#$R5R1P*n)4@vW>P8)~sHA;@d}vq6&t22W_r>zQU`ZZl#N?;3}cB zt(soCn~k_gA!-^L6%C43_1{W^^KXs5{3cydob*erEO76}zoN;9cevLR4nlwp7uP<5 zcDMOzGTqLA=b@wfRr-rv%tgct6vd{PZF*LF&eQY9BCM5oi=>yhV1_0r~>;yD%vWg-LVpG<=e!5d>r|E;o9&F^GA` z#of|YBFu)5QMA#Dcvd;~{pI{%^r8XD=jYzGp<35_e1v?@F$1koSB&odsdiUA$ZdUU zS@meZP}Mq`tTDxUE?bzdYlyOIc#lG4GEv=m-SeKF>&q{@sE9Tl(pjA-FJa_1bcKY2 zQ}^%`{D*I2!1%6>@YHSU>>F)}r^H<`lX2%)sg^wYtLWCUQ@>;i{ce=m5Zq{$>v6Cn zLl_5p|G>%r)n5~X&h6jR8rOlLnyP>ObpzRg4`(75Kkp=X?AXDUb~t-Y-c*`rRf)X& z`o^(`C)#7hKu=oguak;yQqt*vZggWy=T&srRO@gHKPr!CkU19hE)^syc}|T!p=veB z)9wix<>Z*?K6G@uVqzpgPG0Gp$!c-#T{$6%&TL~xplKLH^Y3?HBPp)xcRR)Z>dhNA zLV>DGZMNw8Xm3Usk3F)$+OIN4G_!B#Z8T2U!%A-3vQ9Zg)8UR&)nS3RVN72GeYeJb zbd{y;ulBaQC@pI)-k#ejRonn=78;ua4PkXjO6UbG*OJ*Qaq5ycBN|5ez*XzlJPC9w^dFVhs2& zOMA!(Uo0q@>r`;Ajd7jD9&89%mHi9>7oqUM^P9 zbDgWJqw{!l0)WSD@6t3(6x}DMqN}C?H`4!E_q5zfXs|5_FgV!>RYMzw1~Oh;ozNls zc+C*6NEiBrmd`e@lG~* zBrvyVXjp=YbRi~2>r_CM1zDYcXft4S2ebnP&v?EES;Yy+=cO2WJAK^6u#mP|D;TF~ zsdA;TfTo-xF|D|9{rbU&86CsVJC#`&%p(2bkt^%a{?F$OGfhI2g{`Bxt^W+^=bfoE zmjnfY?c{S`fe}3@Sssl+Pfix>b`V3aTur zKeVX3nH*0nILN@O)6?Z>=;bB~q9(lnk8-oPJVZXQC7~ErDSitks;{{rJNq*xub+U$>XUK{#8ecrF*aze?_oWH((!113Cst1kuNM^sU#rm zTtW3z+4DolTL|bUgpP4te;wWlhla0*2U2T_Mik`s7}|6ieq%!@OtkO`M;Oh422^fD&vFt~M{v{x$F{3_P`|Tr;i@7cgKQyq9{n0~ zH?E@kZ(5kgg8Ybmj`Uy!MC4~;RgA>v%VS)Jdu~5fZXboU zXeeB3@;>_?T5-$?gW zOMCnw`iX4e=Y6PM+gv<6J#|3w)2z6!mbJ43p|bf^x`Q( z+yQ&Iv)QVzQ^BT*y{J47esJ()5k{egFTKn~q*%UB@)Q6nNOUThUqF4z%#104gUuC3 zhpRlOJ&=F&=Opq+DZi~{-6gloX5`I{u_^a_{@teCj%5b?4s6~`q>W~2Of7tV%UHM> z@fgcui*~2_bY*gxuU`=h#jLY*DdIzdCo@iGPNjVurZ4-r+y9+UW5h2{FE2To;)x64 ztE@~X05WKcVO12JDMvA;~HSsZKMjK_c;;EL2qiiQ*zzpDDbZ7D1m+?{gfl%aPd{?1IRWtNYGiDHa?V>d zFvv(#E9pdd>?9OsgWMeZs;*tz)+&9taLpxab=V>~1zLxA6ERAJw2+|}LcGxil-r1; zVe`~=wzWGMPX75}JJ_n)1@Mik$3o4~=ni{pz{stvnnS<%QgZW!a8NlAi|-@skrA47 z27S4{$+I=rx%g`5H)7YC=Z7E$eDz<<30E0U4A{`*+daWspvoGws(2y}+FXd4!ZJ(@4lq%ZJ%(KD5BED@g0RR{J z7b?Lg%$tpb77%e70l2QgSp$ecP{FFNfz~y9Ub#^CRUz4{(%Q8!iP6YI8KS8AH(qq0YfBa zW`+ZjLe{5syEfA+S5HD9bh63^!DauNh2I}%?2Al3wX{TkdLF-fd&DD+bb*PF7d%1$iN7L z;)4-n-_=z~QpB@h`t%hfexXfq%=z;ti4{U;d`#sLx5BP+B5K4$9A|=b^n*-Yk(5#h z#U4C(0I4&qXU%yK83zoW#lfJzW0sd7_tV*h!-%Wo07Q(^Hop_zdN;3-RHBdW_9+CR zSQN&vMEpxahp~3cfxx5w{!Ck%AiKv09VoK9M5R_8dArxpAE$KPC5@r+ha5j-4MJ&W z96jf;cY#U)z>VhDGRmbwqb5D+HE`p@)uK<~2L#9mbWH|`B#hR) zmQWzUx?Nk4)2@cY^TQ@t*+J+kz|1!O{8*2z*z!4Dy-;%-qm*qb*dE8jtV*^3A;Y%QN!n1&f#VP(1j2)WAu6xeOP%7x zevcXFv&eS+nVYFb|Zww86PRDMwkBQFB(}Zb6Z+Np#&ZH7M`4qmV-=xaNZOuv1wUcAb6dAG;GQy+d?CJK_z=5L=ujj^% zoaWmwk7@%rxPw+-5E#V9%;PRSGakpwlUBP{7g0gvFF5ryG3x+{D%;QUF}npIDkvy- zdDR8L9w!v7l%>^mP8hq~yJao+mqLq-VDqBNCG`Ght|D+PafU zd}wL7Yu9}U3?a@8fKVH^Ve?Iak1cQCzC}X8YpyAZXH?v^4HvZdjS#(oc~&WSz3J)o z=&KA_n;mwAF}uI`Z!LgW-G%A&UAI79zY>zs&{!oI%&@imlewYd-|z3A=xdk-p#ANe zHAC*G@QkR)LLwwgI>z= z@=i1uTym}Rxs zH`lUjI`D6PB^noM68H12wD{)Nv8GX`QC9*?&8~~a>7I90yC@N3+03ghg|W^tQu6D$ zGJakOn{hhOXT4|NX^Fdka(bZZg;pLkPZ~cz@Hrmvk@JLg@SnGxir{7CYW%O_0 zbPLMBTp>u^_h+@QSZ9D}skrc64dS<-^^GmQTO9Iwky(&Z8$PcSDU@f|B=71XIl21PnIXX4`}NN#^)!_kulnANt=ud8qK^y6!-kbp|(t2qv9 zYipxUuRiLfK0LALdj8()%b*}3ErVc^5vm1qZe5^L!_~p9XS8RU<>R0eDIFIMnw{u| zX6|zRyhL}&20#dO1X9Jg*y-1`%JOj8EK0$0;QK7m!Kdj+aT(!0C z%MWR{T)qbDdG!}vED5&HN9w75JN`%7#GnF6@N63A7b6Nxn+?3tdK7&&)L3) zC>QnbOcRSoysfNyOB;X7zYTDH)YUz!-FdxGk_8)a>pk%^B@e}Y&HLMJM)zuIch%~j zzSB!FGLb6QaNOd{;&FR4Y%ghV;=^yf(;`=|#5Q}zu6hBpMOs=KNUN^(X}&9}?}Xuc zcK!Xml2d5DrCsnn@zC$`f#Cak0)m1)0Q3h32Qgy!>eZ?4(=-uBGcz+J=NA0(-k^XE z0f9pu#b+UrJd{5jauBTm%IeP_>t~Sv(WS~|Yu$I|^l3~fRf3oq^1=BH4@rf{ECS&W z7ZX#VRtqX4rRzYPwW5$}aMY5YYKpZZ0zYU0?|W#Q+=NQ9RKT6WijOLxdRIe2iY$$x z0tEmpA>q|v_~6jEy4FpkDC3nv`ivVRy;X$aA3*Q`M9L2a+V&3%iHeG@Tq#|u)wLbO zXyw-f6~cM?m3!x{>_>3tqRhh-#cHpChaI}HBjMDO$I;?U1-0F>Zs71c&UgIrOVifx z4`rSW;TPO!?%JgiIUjdIFAKFFvuJ-&4nWDB{!*bw3uL z;VN+Q=5YRBt@-Q5XZ7IvE&auwMl3e2Wl{I<-^U|`$El#l{Ki__dsT4Kz;Sl|!i5WQ zaq3QIyn7E`&HH>W2_~K&37+p&559hI;bT}r- zTG3b%FO6ljU0+TnRL6TzCA7!NTEE0m+}irpksqev^@Yt@^0e3)nTch_;TpfSMHZIw z#Ds-=85SqQ?=4%ok52acfNMdp7~j_GTi<3iJntJpSqTZ)nl+ew)GG2kDfL^A1)=Ul zedv(Wx(t>6@4^$F=1$+YRaw2mmeuLHe`a+=WePbDdT_?;7^=*@6fd2!CO^o=pj~T) zY(ep-7^uC=@TWFCmJI$q+YiMO%Nl~}4tE*9FM)^na|oqDFM z<%Icqi{s|bkt|_BFCT4avU~mLsG3*VCiVBt_RCoN@f^COjc`IA9h@!UAQfGwCS?^s z7Cv#c`}M0=Z*^`nUd)jCiR_I^LKmu4$$2M4bK{>~@aWHMVur=(M!;uBL?~@{20ZA9+7+ zuKBDP6gl}(ZYoaSZ7O(Lc&x}$H@-{3%lZH>31#;j{2(Rh7=|j`x58frnpb*k%Zt6g zk|i4~B|Ha8o}sp(_rt>ARO`=xobZqpSh*oUUOerzeDXyQZZXwCxQgtJj9OrJkKM1f zaC`j9(r~cw$zeHuad90WNl2AMA>V2G$RPXBw0B6R?@DH`OoOme+UH{nBl)A*x36D! z!N$c_xvJ^sXma9A|E-FOKH=!E3pM(U?0II-23&Iy8}pkZXu7w#YT6Q2JO}yy-0Xm>y*E7_!M8 z9G`e%k7TF)zNVzyG^twZZ`f(s4a^*X2D@Fni z8i0Y?uC?(PJaG`3ef)886|&*s!|%Mjm#3SIonFSq$%z_22%|I^Nq!0+H~en-M&q*{ z#Hsc^a`A~%E#DgYz(_B2G-fExfvuI+h&`&DGl>(Qn|Tr063vZ9VG{S=z1JaG!;Ukz zaE9U~tV6`rqz}jbr6XLQEN34DtR7mnl_fH2U)t3>uO-!o>2w|Iy?nvYs41eU`%U=dT+cSh=2DT)6XM&|m+b{{4@0)w;0HLwSl@gfCqH z9!wD3&Mnb+KX<-84@G=EpKXrlAAUn77P=jULf2TAIXqI0RpnWHFIH@ykC6V_N04RD ze|ovyX92Fhp%kV4IEo4nWd8AcaP=N!X6YXeJL>5-(qD35Qic2sjqip+xF&IkA_H^| zq?LOIdS8%1o-=!URN43qnVA3QKMI8KH20pn7tRg%re}AP4t%K{#6#Uz+)ECkJ~v2B z?^22T(|a}6r`7>NpK%h1;167qX?3-aj_m`a8{5Fnm)r>!$Lr6e0vm1F9PDa*4*)%gV@WxWFfeai z!)T-dal+-&asNIE#_sTQeYjaSZ-R5po-u&?;P?{SN5wt>cBWpKRCds=gsKxdU(|mB z2gm$bQq&)x+yMc9fwG5boB!%=c0%?-Z03q(%erEEW(&WPBm3Qz= zzsqewtzSHcqc2t!nNO&qpV~8$JaNLHYfhH5xNso`W)5%REiy2|5-lyS(Hn4v;;4f7 z1gCrvonck>_AatSMtFt0>8SyO0AMahoM~2oG#N$j7ITJOs;VD%2)(F9@sCUV*sdny zriDIRr=czCKN&2%#LvbPw0;k|%;U0zOZH-Yd0 zP>GwH8^R|{Vnqe``N`f^a>rl`D?SE6Erte{hd~mG8s2`g@z=CpyAjQbK#1vI0QwLT z5`w*xbXHu$n_Q!Lo~DTiPp9E)AiRN0Om~SNQM6kk>=6 z+>AT_c)(kXusN_5`KBxDD<~aUPfLPy_PlPKata||oUI7Q2=#=)(&CY&?n_DYqLy;% z{$qyuIZ(HmNDG!3L@W+wF%fEp5F;yk`q{_)au>SL%$;vp{kO8Sn`A>iziI%K_WVB!CDwBp^J9FYDb$ZAt}<@e)j+oydZ(Tt>+Jsqv1 zb1)ZfNC$zI^88BB9JgiV-l|=& zCRAP_Vt(ildvzmtOR>Jr=$xy8SMvZ5dKR;Pa9Ub`ht_1Htb)R&tZIC=-_7GsRw>!1 z)d)&RgrfIzZA0i3(6R0e?Wg7|p5g_--H6SG$MXvHF%)6f=@>blR=2FOZnxPVD*}s+ zj0J^e@H5}<*x&Iu?E3AY7g(~GyBe~iSDdp5!OH)f=3a~e5I+^|936edY^+)O@wSHX z3>)CTanpMfWbSVKKEewF7s{PmId0fr{ytsvqN*SGPa^TdKdC~62yx@_{l?`wX;Y+vqLE|N52#4abW%Vw3*H<0lIOD!LyeP>(=npuJMxkPIOU)YcQMtkhyy-ViJ z88VgyTKAHoqEVzV5I;a3IADML-Q$-RHZ?r9hljnJG>MZwrEDQ=tEZDz#5iXXFh)&wq ziL9*O-y(4e*-tJmhJ}{n>c@AJ-#iWie=1qx_$V?cRM#d``U1DghI~aEYd?whL*noK ze*MyU@Zny?_(WF{60+$j!odi@v!BFY#kik6Uf^89ZWI8x@hmPBU*0ThGra>BaE|8W zi{JB_q0Sqby>LR}x(|Vom{D&yZpRj5D?(*bzO_^EkH5l)X#5I&)4!LCsu&7ZAzfpC zKN@%sob(JFp$py7_TppEyrY?X&;__tVQz5Kr{m909m7VqpTva)eB{VDLGJzKP+TjJ z9KM$1CLvHIj<4>Eg}?tN;E+}QNOsP*9GuP)SJ1h9E^g0yC%_jt)}|B*`S^=sU;2pgL;{Cp zrkY36FUeU!vEey51T{8M--pajbiAxSK1>%m$oC12H!UpooWSi`K@!1telW$%qQMaaW#p0>L1e zn_+f!#U1U=RY32P5vtZ@IWqM`?aw;2)sKeDc9T1_8?4G?`Lx=Ceujd;>6 zFkJ9!J2XZd@bd9B+p#0spAjcGk%FmkFo*2%%1lc;{QMnm)~bFKdzh6uF^+=@iH<$? zBeMZCs`odT7*un6fYGH`*?#P#(H(33oh>X9ESGTjNB_d|YB!Sk^DTUYWs@d(8jkbj z;~~&o0jiRgkwNyq;#_L)`15U%^@hC<>ObX`6Al?K7j#xTF_M5U? zO;wd>LPm4m0@4NF^ApT(Q@BKGYe`-{jqg*ujZBOm+cR>3@d*A26LLY!2+q0gIE+!& zIr=W43W*srpc3)%l1KJ?6we0EhRtWF4>YO$dCl03jdK(c@=+&2K7S`CX9rQEcnCY#6ob zZ=Bq*r`s8$T2VRXP^uR3Wt5Mq$Ny62!Aa+gmz2~YuKEvvJztlrYO)Q+Lf}4@4YzME zLSv2HtXhzKK07-*a`E*#{5g}|ueHg~*L2xGkor?wic84xM3X#^q@*OGbAZ)BVe|F! z=gZ|+V2+oN7{M+_$)BFunD-uSw(-uL011IDmK5kn|J~OKA4X*l`1J2w)Lqtv_XBN1 zKQtD+ja1XqyN$PuRsn6OwK1*T=EFcKW7be@Z7o8`XVX|sn1Aj-L@J;wY@3lPkA0<% z`k~CTNw{j@ssDJjBPxYbv=4{;2@NfBS@=IQNRo@o&EDuWZ=#VKgS`&mWgUNo8U#6yeYW%m`Y+R%}T4~$<&6hKl@t5h;T;K4=4a2p5Od6l9t$F#s`?3R&rn;5V z0)Oc18AsM8e6~$|2mg3_;mdD&J^$xN%oWVdUv?ru<8bFcrKTJ1lj+@A}{ z|5r-6J4H#1%+7}oA+`V2HOw@3OXOT^?1SI0XNqf{2R6=|I=u#<@B=L{x>>k}ei>=l z;7@`GLa;&O(yy6KIY#pN`&j^8eTmlmy#k{ce5a#mbXdS2J$wjgyVCu~Rlgt*8OV(T zp`tc33=pESvJ$k9ha<`M=FV7AXT25fE>Ms7O1#M4#%foCuGe=Ue0n^5?bCth^5XxG zwl0Fr)1zq^A~vK@?!Bz){^oun`mczfY#V(ggaD+`Dh9?^h$g!GWYDc4XjddX3#feQVtiw5t(NU{{*lvO^E$;x>faYM<3Dfu z)wnfeA|BD8*1YuMPiWYy`oUz;i=8*!)`#@;sXeIPD_(nd?|x#B=*DNa%Sy}uw)|eo zilqT#wx5I$pw=hs@ZmhYOBh1=Ht5-a5a{GyiK}Gqy9kpS=jA8)US%!!{)? z;V*-T3`nw-00O6|rBp?DXec&3^|I**?(vL8Aknp#Hq{NFE z>8!j^${dsNjAwi%FktUB$dCya9(^AZA0PSB0Ux3<{AYFl0_0R-FOvtdctSE4wyt6; z`W@~$a)YhNAf8K~6kbur|F5OBW0Ii~4+*ZLAKyQ_79WGK%*`#QoT0W41K`=e02OO{ z%dmC;+_vCkLk^Etf9PW?!eaDg?Z@-j34gy zuRol*(0tXZe{GAJOa=3dnI}O>(EIU17<#6uX_)?L>%7d&Vn2g52>!4x62sCg2?$yP zCov8X68{4t3+k{h!Fndnm$`U&u4o@Q2)zZ8+X4bU12I7wdS8v7lj-KmbHRE*)l8wA zeDo+18q{sun&A&YWa%pKW^(IRt&&la`@ad##z)PsftDOnz69kY2B_utr#(X|g{P+{ zC{GEs!)0zANg#$qR<2}-gE1&d{gmiyiwXc5N^c!(5n3NRqz7(3!0WSyw@rk_5FO7ML}_U}XN{-|+v3M{ZC* z4AUp^ufX^n9jq7==WyI?L=yJdvm?vEr)jIJ&*9g#pTq(sWi3Z$ZB08h{X5|U_pj37jrYPFEgybEp{4zBS{dI8gcYx5)zV3(H!@{BrKpP_NuO36R z5gaw+n6oE~Iac!Wj2vpfQ`V=GiR;2XHDlN(V_Gzc#0d4oF9)Xi2HbEdonD`G@U9W0 zA@*ME8)Var+ruRh>%Xaa$Ew_jY7j!;+tcD)`~gTna)DCq zLE0QQcj9N|u#3&&D_7D%jsoLaDY8;TSU3Y?7U7bkDvIg{2KT__@Ef{)YJh_~E#RYi>|I#4(bbe5-3gQ3BOsEHZ@1 z(R#rM6pEiZhEKdU{t&JLSQsL(%<&$eDF?YR6I5@|Rj=O_E4tYQhYzqt>>ju$aQ-vE z1`fuctf&i{50ge)%pQfePbrfU9@Y1pBtqFIK zu-4JhfuJZ^Ik`H>76I^0=j!E;{xEITGBo?wvXsKrdz~@|KFfX*tNo{*2G%Z3-wZFe z*B--S6fwf6Hg8Zbu&{qvu$;k@P*|~qkc5bMN`%XwDRQaPKS%6*{dw{2)}_Rqc>9To zS&8ZC3Y7r8Q&v@b5A{@d%ua!HQ^Z!pF}6ANbL6GMs3?}=IS`Y8@D=3caj0SuZ8)r; zq=dznO8Xu8d-f{v2NsZO&lWdLs2guXf?Z7Ckd$Jb@d#JkE{+bRO`9Hdb`s3Z8-u2= zc>4liOo>&OlmtHs0tLvoX;BbJS0seUEdk#isdxs^5n~mL=~j#|aeR8uV%!r9wTYZ! z@*#Kk?zGKNCcxpBm8JZWY<>JVuGSa_r%1dO;CzP;9wg*4lI;;;ab;4Ir60r>Rt_{I z4z>Oa)ZIP;@w=g`w;;PTe-RWK{^%Za^Gym0Pwija39Z3eo9Ij?q5>c$KU(^de$ss< zt+(;_+@|6=#C1=vuuIW;v9BwJAh-J_p2XCOKJt0JP@IE7$eqXP&0KnIZo(pmC&g(F z7P5lyPwe)FM?LmiL+8Dri<;+v7H)-08dfAoEG#YjcQc)q{ek{9rB6Zaa39~m_Gs55 zSy|a&$PWOW12V7b#}lruy4R;mJbLsi7FJb1pnUz1FK?jV86U#I0+}<}w3{IJHne~- zAP$b9C-$i+DM&J3z+_ci3AuYS7qBI&`tdIn1AdAjFPRHRTtCE=;~TZLErrge>SGpw zOGwBM&~~t*yq=Y@{o9s+Ok7cYJxxGhq*OyH8=qWW7i;o`WWjB6S4 z<3dnz*-wJDhZy6&9b<_h%hR_8a+42=uRjOF_#+dCwKEKjKGv6PC4K|X(b{ra;H)>D z*~`$7GIis&=FQw(m;;de-eu8cKRFzBNx}A>mE8EE;!|X&g?#8d(i6zjh1l`lyu=|p zx$zG)XS-Cpe{F{Il$gCBgL^aC4}$CCVjiv-k7DExQj?RdL8BlY9jrB^wRoa91X={= zr-v$cdW9o`c6QGQ7J$sPG}^vLxK^YmB(wu_o~-5mQq_-a+RdxvX)G)(R~A2SbaKK` zsaW}?5jiIlEvnB@LEbP!M#K`uI`9tsQK&gZPi@46!u<^$Gb1&R93n9UZv%PwcF2)k zyUyu-eR*~re!5jdLL@kji7&pnj2IUBUUPFzC!@7%)?}uBYrOv+VG?S#T3Fr1bGX5w zrobGS^@+!E1v0EAXX~-vC%%AL9mKpL`Sn7 z`N5F13gKYs*fkH0%P}0K#dEl=lr(cZG58wK8MD>$=+N0{WGz>9R(o`tn`UKWlhVJC z?$=PRSo7f4@V>5FgnU%GLp!%-noN6H(|2Uu!Eycrw2XbVw_MwBjo+r-ClDqR(gwq6 z$7)nh49)WQjC=ueuL9Jgpeg?Ij#4_A+h&!%`{t5F} z!U@dXuqko4jI;23*3spfel0(AiJ0kuhmuc64te(HyL`Bj^fBzmyRRU*?jUHD&%q@Q zb&wTc_AW>!S{4=t9HYB}Pb_{9I5M8{UPPkomi|rb$5^n2U#@GxMpvGT>oIw6^E`!) zgF^6QDXF~W-8Rf%QTQ>fq_6+%OAoBWS+_GWnXSC2XbMFJXy*^$M8Od+{Q7+8nl3is z@&)Hn8Rhls$#@~Cfbe^HJ-r!6yFELEZPK|y1C~8C`{$Sgf^)O;^B1wQ=cr(C+2hPe z;Z^q?Wq7IVkwo575YZOUv!UkDW%#;wH+65i3=NI;X`i8rwnrv~yd63f{ zix-6Oo3}B;s~Nvj$wO7#iY-aW$>X!a^YM|3tq(rV)0`bsJ^pn!SrGbCD40!R(y%;s zjc)&slZo$?{*THi!iIWrm;EH7f9zuzGa9^n-e150jh!}!f}6yHGu*Qn<}}@|z{|;$ zV^BgEtSk0>iQM|ht>onU`8hdvzx-PKTWJ8X@Swvi%DSq4+{P(8{6K+8EoMMDBY^3~ zYJOI7IH>XOWhY}_THCxh4_jlkFf64 zHbXh4HZY6x*{c}q`=DbU6_P_ox~H{~QB>ki1zId+4%!vG+zIQ8>$aBA5x`#>;_T^J zpxb^p)p^U{#Yt>QF|@!9m=(|o#cpwQNQ;j7rm(|2NL zrN2zU=1Jdh1~QAJwQpMk z2%!#>Fm^*qXnBw!Wf@y4#mF*bkFmVh-TQo=)BE<{>C<86zJI^(a$Vo+`Z_QpRyn1! zwN=CUI)VFQu;r`6#_+atUpEX~*<6O&ZX14g(42X%xwT#`6*kz^qH>DY{Ld-H)K5+_ zhQ>QaH*Q|eSv)74i^{}}H%&5TJJuOqs>iS@=E0^UL!q;Zi)&+9ZRI5LnnObuJBCb8 z_7YV9t^r-ZFR1Q3c91h*h@wxkmSi=J5jDoGNij0t_R_vRdn$Bm_pJ%&NS(ubiDM)# zqw!sOj!Bf!#Jakg)hal3on9d8MpIfZH}KWGycj8C%g#(V=Fi_OzHoc) zFWqQS`&_mXHFCy^;v^Wi*59{3!;VX_Y#ku=uZJC?6y3!z-)Tf4H%g6B;j@kYU0Hb5 zd@Vy4_4X}Pw$E@F{hd>K^{a!Z8=Ja>hfv-}4m8wAXBn?hHKEd)6>PZ`2K+adPCr0q z(pVsf4&^vw?BJ-UbfSk~t?1Cd8$32V-%)oLNc$3=CywG!f-xm?7J{l}?mlA0bLUT?!su{#zWx$yjIxz!1p zD;I*yEeR~i-m@Mva?E}4R%FAwc_IiUzM`KBJC7HM4zG2Ft zsr{miDaKlnmvUJ+0pa#nnRn8i9UzIK^rF{_E$F40b|y!my^eJX?MAS2NIzj$+B0@f z=Cb*AU)w0XQSJLzi(cKE$HiMUIufvc0A=!b-||9arHc0aDZkSN>4;BBPhA6Tb1m&o z$d~l4Onf0DjFBLD*-`1AgLU5yyRqa4+INbCYF7S8aQ9+JZ*#u)TZ?BG20tp?w9!DM z(QeMn<34z>Ry$LAh1$x|m0jf&gi4)9t~KfzhfZqJ9aq->pR(6LwdaKn=8Y#7anV6` zi5RL!-MF#9Wb+ofc=&19em{QB&qOVs<4Et~LRA6eL_$Wx0U!Y8&toyP;;)dCyf zA1`h_cUd%#i_r-(m*1F~koUnqJ9(sF$TwKNkoH4+9v6Gm4s63X#CuE)H}6x`&@xkh z2jxHKYJs7aK>{()(DU)R09rP*Fi^SuK-t5w;SyphAUgOXmW-mu+eDY`pNR^tUd>x@ppu}=y?QDcuKuwnTCR{$W z(o30%L*cvEb`niO0|o1ZIApW0qM~HgX*Fg(vP8@?+I%6Q{T)z191!qAblaj6_{0jA za~oh!k2j9?xj?_JkEHDzmyhe4M@)}t<~h9IC6^<@diyCXNt|dN_~I$kxQ>c#lU1h9 zmUQKkd2gm&RBlB*+;YOIlycdYcBT8!$ENANZVXST##Pcsdz3^lx^mKc3|NsvI$T#z zF40BzM9Ms2V#KV6fon&T3%WoIE%CZipJ$~h4Jp*=&rX;MC|J0gK@+p!L`vuMqs3u! z$6SlXV$vvIdQ=qL?xFr0wrYiiW(IrAS3YzIC*)9^&KosN2}${lP)t0fh3?W@67$~BueJ%b^Fg}p~G#Y02vG{h^=5oTqv$=vCQ5)czy?NNtC&XJ299C zExPSJ{b6OvEUW~GSy7p1!*_5zfH++Y*Zso5h+`k^OYy3ynf}-qk>f`u-FWl4o;srL$)T@$NlxwWWm$h2&R<=JxGwe}6Guc?u;eu`vg^hIrA*%>@(%{6S8zRo!0+CdAEI2T(wD`rS|?(nsn^T zoY*pFjKW+H@U;?OV$=@9a+{79JxM)2EO)2;(T@`0vhz8H)wOkXU0&0*pFY@2aB(7K zH8Js7L+nE+lZFg}aPOtU*Ri<~4Yc%73D z$!S5P{%mwiTLm+dHecLL12P22o|V&4{yTeU4~1A|0e@HZR@TUsxi5u4;-!Pcrbx50 z8qqTGcOj`UaCU&672|R-ce!{Q7MfX!cN&e}jDy6z5zw${U)MD|Em`u|2 zwUNsKuT6t->5u;?nr}0CR6kIHizILA{bEXASGRThpXx_)ZF@rBfBSvF-n4p7F9Kf@ z8$}W8M%zx6YeFofl!CsPgl3;l0pmD46KLI?KBI!G#zMH_8|7csD)0Rf()vO=&D?8U zkr}*`Oh}_tcF1m3HnCih;za+_{rw)|T%6t1xpuAFpAl3;^w=>`I z_crOw^x-zBdC?hXy%wZ3n|vMZ$@coC%%sH{xQG&8aD>PeVWb7#4o*YC(ZpGHz+)?BJ^Cd6Zkn+z0z4|xh&c5-bZYa4gxs0&`Yz}YHc)y~ z%Vfq1jY&xSqtQP&)1T1Qt?!k2!Spq^NNfQKP|+`t-dG3+H56&c7k9Om(u0jts^9dE zfcFTmwhH%Qnq#VQhdHUVr(?Cz{PUyHy0Xvu>|f5tSN0B~o6T?h{`a-tVF={vBFH^g zNHT8dyQdR~ls^qKoIW~jjeO}hzC&?lOu$}_q$oMy_Susg91~pX9#w}(Z#}I+#)0nL z>U?9?nmQ1uIz6dB(XmuuF(tNyS{N3_614eza!kiwIlGj~b~Tc5A%vRS3mHqYZyiKo z{juv^f#IClIcaAFmB1x!rJ^5C?-qLUOTH(UM(_QHP5Ugq0*!du`@2e5$Je#48GQ@7 zQI!*$;{qk3h3HUeIc@Df6%ElRP$oxMqbb;R*B*~MK~wz+Psab`CI3%^PB<) z%Q^5j@<(@`cqsaJ35lqpR(lDH^k*x|)qeYJv5C>1f0?uO&faYmnphQ6YF22L3NNkN z*oOPPg_GViu}W=J9Q0|A;rp0|6ztBJYmsQ}3VRS_RI4wZxS4-8@^y+-TI|Kkc8QXS z^4|9P0qh|$e&^U9qxwn+jwF?46Xq9O zM|@lL^u@R6cOc`I+$WivYtZT|Y_4|nQ<#l{Xw@gD72IoFXH0ubs9 z#V%2eb?aXEw`DJRKn~Yc;QXiwoUu*WbEfmzmn5UR{yQlRFITi)CiQQdK{JNRb9gy) zAxbCq;NgfbTqq0W!Sn^-S}(Fcym?J3jk$lnFE_ah(S+HFlm_wPmVp#AV@V+skt$pe zB}K)i;GUz#M)3~C1>tFK$*2Id1$Xpk92|{6ffyNG=lWrnEd$qD{mW}yYieLcaS~{U zFN;^Kun(*+dfgBoW2y0oY5D=z##JR{@QM%%#u4lf7b?GPwYy+M43|=RQ603>yDPsh z;bqG}kcSW61+rsv-{5)0h;iRmwcyO7h0RYp8zYEG?fjvdrCG)WmM!+Ye%#Z>zu^ z4hd}$Vc|QOJnW47*ESs}H?jYwNg{YSNK;+i16oZuK4`uRt z{g^Foc|S4k&Ww015B!?L66C-YOwwweAM?4E{M zc=YD)Vq#)>fsR_4qo?L^5pKHQ@@I;L~i(9Y=rvqNM%2XN(QZj!o?Z zaGsP2kOZUq6erKxuCdl`;A6}|LQnL5|DAFLpPQA{)ocm(hY^zi@A9~yLm0)!K=K$W zl|dB|RJ52dYPNk?BWg*fWj=%qLh`_mqQ9Guf1E`iCYp(hx68kV3DT|onh2~eqyI$C z>Fb3QZ7xB7`HgI0h@cM`*l$wq&5W3$ctY_*A4`q!_@P6v68~<;Q;GD0K-8zN=`62G zy8FZAjyX=q6*g<9P+P>vg2wU%sLKhBaUIVC-B`QJQ=%CI%cxb2J`CtnF;ZSf3B=0j z`J=-J5VdpGb-DEs8l&&7z`G3B zas9A8-OJlMzw@IXzPCsBZg!@P=UtULgmnfZ*7P=VGMfFEK1b4=UaYxszHsR}A2lui zRS_lsn?09rbp~zYPkQWL@ZLp;7!hJHQbZA&HMtAw*0{F5xH z4>G+tt!TWrqZ0*c1!Zg1F$HEx(mCg_S&x88JhZBX)s7K+%y}DA>^#e+^;Di0EZ=-w z#Q~NBFa%S?g<@>Gh8fB5hC-kZo+h$=t>d!~PU^yfw)-Xh+|vFOE+O(v+9q zb?GgNegaTNeuK{BF^hG7%gIZp-+@8`NC3G;W?!-S1@yTnE{XlJ5cS97H{oL{J=8t> z<%-gNjHxesgMSMrJQ|#HhFFW7vXULhkmc%JDlXT6%E8y?20HXx@g`b*0s6+W* z&sT0*xgvY{vSq8ZJvzWL0XFf4mQGv=2LZ%=o8R>cUuGw9v@d+ziHF zz93)r?)^6o+R~*_krx9xku;39ybzN#&EaI)XT^9m4A&bZqz)SWEWVgmMOfJ5IAGy^c=qiFB49_2x5*{BQ!fv6WC!jI2Biiunl~;qZ(9oFwQjL zGlyW=(LqO`m_0H8VFR-of@d&fAV~p5Y0RGIKJ{|y_MFDk)pX)kq3px|hXZNCPzgI; zh1muz&dRz>J*wBwoIrD5aJT(sT#1I5(OiqmmqhVjjF?ncuXUIBfOANYdan_ke7mNVF}2?B?;YfHZlR)od? z5Hy$rtsux%$>NF|cphNFS86zfP8hA9UVLlyi1x{1>xfp|5KS<`z@Got){R)>;bGdn z)uF^Qs?g=oBOC!jVd$BQO~Kd&=~7ZvM}lZg2C}L`9)K9*Yf{qE8{)`pu3_c+NJvF= zH8co9p!Sx@(=hasb#mn!nb5%yY8)eW$QV4Ps^axp!^^mzf7(8v-s|)uVP9j>6eKi| zsE#J;*uPyhoO?_sPe$Nib;NHAmSES{er+(TYKU)XjNI9V;E<(nei?`{L`sCWQq?#K z!zyDUYy3awzQmc985fo$u49v1!1~@|h;D~G|2k{H=7P}YWxQsjZ}*R@Tp0#b51ip= zgaxQuFj63I+0VIK{ZX-CU_XA?K@*d^o4RUe@bWiN@<82H)Sh8Rr0HQa0N;9>enU-y z{-=k#MXoDMi?HBg0Ew#~Of?X<@t_o$X!mYDH+ma|jFGVtIRdh#Dtb(0WCN}==poJ*aLrO-5*m%Uzox-&M4SAIB#O>l%FT@9Ny&urwQ;gFT z92`t!V=jeYP;AIgtzpx}!iNUy6Ta3d{XbSso^NDKpXqR}T)i3qSXy1@;yr!*N`zg; z+4Ft3M8(9Ql&O022FE|l-WJ3b7F$;nKHO?mLuKzHK zM=u|}XI9w&hF(aBHD7q?KUu>q#~^(nxs!T+M?Csjbo@W*WRC2KiiZnCMF|`Y8@VtQ zRI~O~lQGa>%&EvqN!dy7PKQy^&S70EHAO{5SQ&wYhCVQx^w6&Roy%qw6WWvxD*L;ul2dQLA-QCycK(0sp?G*YiLcu34m{1U(L?%X zb6_cjq?-^}h(3honUjpqIy_6iR`?c#CivDrbabA!_=o`wf21d8ibo2IWO(1edX?P3 z<{wJUiE?bs=i{Qxmyny{TDvuaOELm_1y6*(6BDsII8AwY7ek=W5fT>m!X9EQAP1m4 zV_K}~Z~z)nI8?~3Su@bpXLFf$;)OCCp$MpY>`Ui!4I9nNFaLS@zt^i?q{w7(RbRJo zTQDC3LQU}eV8`64)Qs2+7y1zw`qa5Q>E3NmtqlxQu~uyq_gq;&4P$XoZz zl%oNE3A{n{%D-G?6I-;(TrPnt#6r(Vq{o1M=(3MDojLBa*X<0VCT8n0SuB4f@!_H6rwPw;CL6|ll>gZ9&?-$@t7K)7#w}Rnhir_6 z4e6gr+bBS~D*CR|drrclkXiMs2T%WV-i-prvh~z-C6OMP>uW`#yTG5`V)VqrHdrWbC3|`X9X&VD10_ literal 0 HcmV?d00001 diff --git a/static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png b/static/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png new file mode 100644 index 0000000000000000000000000000000000000000..538cbeeba5671febbe1cc6d3b8261dd71fc54147 GIT binary patch literal 303818 zcma&NbyQSu-##jWfYPAST_W8%v~)-af`m#-Ga$`S5>mquLk`lQgp@RhAPn6g-3&eS z00Ybq*Yo{7&-}5xSUcH(^ErHeoVW@O!oC1x=#~OaD*=%ggIQqfi zkbAjGTdm=XF`mjV$ewUVeSUQR(R=SK_s#U3Sg#B(+g5ZlALo2lCeyQY=IV=ppqy}j z)QlapZd%f${nf4Rw~CwJZw06bal<FdohipaHvfw! zR&4dw;s5yu6=6LgE;J5ue3WE8yIQC(=#_dJ-;aNZo_gDm{r11xI-KIqlz3%ZUY1lc zp>^rd%y$d|O)dV9UPZjY#RRIVZqGO85K@b(aPXTxKax~Zx^26AK+gZi1O8}zZ#^(D zBLuwEa9F5PjSBhoUkzc&wbsP2S#R_bqEL<+3J&n4dfd)^9M)l?e~*7IwVPcf6}7Zu z#ei9xV$j%m1?k#xOEiS_6tObm|DMJO+u%gRznkoM zYyVK-c>Zm}qyN6qe+>kc>HWVgRII8h!0NXd3L{F-16K=_a(rGiFIJ8HP_3OGb(U8{ zPaOzXtQ2FK-@5Vc)9mI~eblaV9?xI7n3bNN;WjcAAN|kB)du|6;6$X}ySshFH~Xq} z$u41Q&x9?{=wX`YpEh0zd0cx^n;URHp|I`y-bbTOZ}z-wS`X{`s?Gj7z3UFE8`4HbF;&q{%Uz@0=r)$fGnGS7x@s%^QPuS zyv%7@7HC8Ca(TKvaq0GetTBP`|N^Es6!Wu{$Lh1x-A4;(UA1m zbD#Nv1{2bYR$;B|pD|`zF!fKnB>FQ37@pI~uw2@_nvDeU+(3XfznjWumjJ2^sRdc@ zt|SvTqQgGhoI@r(FoNQ3G0=@L%Lm;kZt=UIc(SJ`kFRcoU z_CI?um;EOk7`F1;pXN$YXVea-(Kg4%&{nVLTq~$$=w-Fshg3ovMt1ES3RB zg#kBb(;7xd>FY0qbY@>2CRxexdOOewz2Zv&yA?Mvdw=|2vyOzJU8kjWz7`J}3*A~` zuAtW^Ihf?~W)h8@hBDky#IF*O@UDBy$JSi_Q|!|Wadkt=^$*IwewaDAAbxiaK%O)&S&(#C5Wzd?G$Di zcvA=2z4FDCZZ=5x(Nd=wg-xVABp7`d^g zE1Trnep8$kaPUHVX|ck%EZJDwU^N>=a|v)y^(53SLAL9f714o?sW0c2FnpFlXDN`q zcJxgoMB2#{8v@|r>t9$)J6FnVijyw1dfh_4g}fW~gBnmT^7b&7eB(y{xBzGkp&2Qu z(Yf;Z0JD1>KVt8Lq5F4DQtFTR->S*yr|Ph!(YT47DhDgMS zEJgl%-7EEf`v2;vB?(i3aAYiGoIiU!c?WcTI4tDvo*gvJDP|yc@Q0f*@VHbZ6*=$y z`if&?N7UI**@5kfU+%)v7ZYabXnk2Yy)-qkM>{X7{9_!C&G)jQ^cayKt9N{Tnhqq- zHu|!3_E{ZDg1N3*M0Lu63Q#OGvJxB3|6ait?rTfxFwE&tbAfleL~kE|YW3({*_+3C z;b41MZ)Y@ZnF7Gvma|2)Mu1$*y^1 z4amfRDO-Y!yZ-HM0{{9YrDf{(i#>V9mFQ~EGuzAaIoCQ2uQW=tJ_!3vnq`$_0i@NA z&3Jt!ULA-8Wbn!I)xZ~WPWR(f2WZ{2sF=K)0-f(F!tUHj(%FZHLwImf6A~)%lS7c} zKd9djI5NL}398%qQsDb17>A42oY%453CgJh;D#;+y;;1t9DY*Sc9}{icNB|uiG=I` zSn@V7bq9~fpevgpfgO4wUYp(+lpM}q1WqY5+HdY1p7owwayYYJa8fiH|ABS`i9d zc|?Z&h|&?;QO$DBJ%mC45gNnkKC@ zf)|vDB$(pm;ZPTfB!vFbvwF~@J+DgvU`wg(pd3nc^ViD`lbogG!0nf(# zp=QVgtXFaSf5pl)wMo+RP!g<8Q8hKU3LBLC~0!!JIB|0ruExq9c~_Y zg;71tLkR^^^;-*vFu}3uH1!(M4>vcKw;HmSdqDOb2h64A?Q^IFNRxQp-|#u-kfGuq{L|YE zz?VGRRHqsLm+=5!MOeT=t<2dq!SO{YVk5UF2hxmewB(rLgSgyalik+S9PDJWjZ^YL z30zy4dvsy?c=tfG|6q6Le%L14@Eh+vzL^KV?y_@oYPWIQ z^!A{9!$wTxZw|~`rI`eJDh2=ZgrxbgM2K)exM$48O84(KSftN<3-b$pxWw_yw!C`k z^oI>NT>(AL;2P`O#s{G{Lsgs_&#sY-9T|%{%hUo)>;@b|mRp{^X3-%9fFEvO_VQA1->Q5C@4efABpWcTxc7S4K^LWPXIJE<9XCFhM;5-R@1G6Cd;HME zSMI`9=U+#6_w`1Vz#d!pYExVW@y%{!dm;(uG_l3)Jd~~hS@QRgh$VQtMZvjY+41-ZYMI zVk$w=*ZZy*w>4S*R|W_TD0kh2t1+pz?8iPQ-_;q!`rdEN7!4Ihr|O258}KLKg)f zCmIEIo>#n?TtELsAaWZ=&DJkZ%9rZH3G2rgc!UZ2yuF*OWI)iAlwdY{l&hD6B5huFKE#&DLi7 zX1xqXUw1!+RIX`9NujIO;)xd)GE#a&<1%fgzE18$9VTI3o`1@G5H+mq-7T2KH(UFH z^*in+kzQkzHv+Sy?v$vW=WQ5zy2Dsms(iCV>!MBCnz{lbI}vx%XCcN*$}HDp5|HvF zfvTZe?&vU*gyk|+q3`_N8VCGywl}x0PQjPP{5fTwQ8gzd^%) zvuV~xZDC%YXT8A!`$@#Rx%hhT(5GC4(W#DLvvTStH39q!16Ci8uSC2d?pT-f7)H~D zd*t~~ue5e9F)Rabxp^JbakbwSIFi_$V+jeT~$iNQ+_?0 zEVNbEGq2XYsiZ2Xjen$ktL~aCe<9t12RK!hgOBuUvC)mIRM;Qn$*W-Yv1P;86tCpUX!I{ z@Nc{_*hs&Fb^iA2=ixH0yI0gkCaiYDmWKM*(G6)8pc8uI-gG}bS@RKGxzm6?T<;#1YShZ zNnIThSz!|Mg}iP(I5FzXf&UQ7d18CwV>&Bs1T0a=YLP{Il9)PZXMp;SZ`BnL_d%tR zI}1O*bRzegPDhcc6Xi)KP<}@osnMM}0;tB^5J2g`y=L@>CVwxD>2qPDiQcTbUD4{& z@9(p$DWMLJ55HGYBvTD}U@#^G@jma%nX6T`(<{PtI^d(??9S3RMU+pu6{K7jRHD`U zr-IbiO1O}&MQxyI$R8#Rx5)#5zP@>Wy5_Yrp*UrMD90GZeeTLKmUZ+d&ah-P+j20}#4u0! zYA|&&_5Kb`(hHmIAM`?gvpn16VoU{tabFxS?!~>0^wH#}D-zwmaVCtwu!Z)XZ1)?h z7cT}*hLDP+92Etk+`JyG&_9fvn(=a+SovO6_GVfMMzL=$^{zTLP2?55Q3YZlQ>>*J z1g#+MK(U|kZz_zdJGKOvZhW9tR9r~&BO<0%010nAP z+`pyURz8MxI68t|A3Z)SJ6?hmUkcKVh%K4&8}Ha>Dj$E;x3O(doI;06LNK|%teQuS zxU+v>y1C@Hu|6a8y{IY{okS-U!;FtpgJv834Z=Oj=4VHS{^Jl#7PO1#^N>edCO^wnvuoyh_I<=_KOYxB{|4%j@1zeJjH<`0oK{U9_IxB71=q0|^DO@izK%@faphwI@(Z=w#*(4JVCF(Kun{Z;`5;A8?x5 zzO|w&=mPiL)%ykr9Q=hleAE>rd`HPe)f6U9<8@x<8)Xi)7VMk1ib~Nc8BQOTWHOvx zAaSgaqx>pyuFAo8`irMQ>4*C>zl715E{i6fl9Mf6>;(z{Q_|l3PUA~OJmVc z*v+c-&Z^A^Dh}+9Gb)cvY$_82g3ps%Z}dg=X1!`=$9W$pJFe|v`w|OcaUon!l!EEL z?Djhr?EskgGFYTs9%rrf#-#RX6jRnjq&S$r2GMp7zQ{>q(}uR(E;E-C?P1y^N1F_E zd$}t|jvp5YZ(?r6>ai!(hg*eH40R2U1Gdt3822Q`#09mB_$|4-no(-X*FT@$FxXzW z=VAG_ITgwpUpt3}5E?{S-(C^z`1CcCH(`fncz1x$mlR;Y1D{PW7s09bRiz-gER~RMSv3luIwCFu^o~(Xd=EAG^ zJs(gjaZCqs@5?IGkOJMQSRPjclF@_D2py(#bzA;Yt-9Z(!i!YC#G?%X`%ff1R07zE5`BREz+BHwoERxu{TAoiaCh`hI({1!9W42j@KcQlPSE$BKn^kKlMf z=HSup(zBJHx1@Sqv*Euv9IgU4M_ zPmFO)QG453QmhjU_zJpY4p z-V3xJwHNv)0v5ZteT7P}94>B#1+LL~sEpETFC?f1PO*NU9(`_n`%ygHC?trwvhNRl z$n1dk$Nh3CeCoH_brw4*j%k)ZeV-}#HHpEUtg5LAhs+Y%s-AmG(B~SwroedZJpn!% zb#@`VUdPOP|7!VE{kGFKPU37T8QIX5GN*J70q$C=b`SZk5W2ydBZx+TE3RQ3Ikk-^ z)2@()*!keBCSO0lX#ZScD(>B4jQ6qPXYW%ghmW+Z-J_Mt3QA&KMTl^??ID$CN_?_WVdOynki)V_bO8z*}5ps%wt5;vI>KZVr1)z0Y>fYJ%a1t4DJZ|Us$os-exupP9g5M53RXU*vc7XNh^|qa+g}Ryk?V303VB9Dqr*hN_6t- zTV-o5*Sy@#9?-4i?GbiVU><^bzjWqywV$%!dy7=T!ah!3 z0e|bw4>O-N?V0buoKxB~O2yaWxbV%iQ$>@ucbPJmcPho{HsuB^lEK}_@UL)@<%Y3r zwSkgvN$`%qOy}_!WjIiBR}DO-t$IHW0P7Y03PTo6Y{M;{Dp!T;8Y-h|zNhP4l1INZ z^<{%h)gDWZu61^6dF&7OwUWqf-j2|5(RWf51wU|hlkLfw1FZ?Dz0|vTm zp_6mh_*yhLD!ZL2ENDI9(}+pz5%s_%7Uv`?9n7K`e6<+dolRnRrt3lbZ3GkCJ!H!p znw6ixt8uG0sExBPnD&c4ZXrK?=X?{Kf0S?+JP|x!os2UZM=|&}*S5mZwZEe4VM+L| zn8^EJzva=jtF~@5s_!hbz#O{ByzlZfia0-FY0Pp}jLZRX21 zOZ^O*#?pSxzSWFRF3G3lzxZQ>8OBY#;$R2esfskSVRKedVFPsff<`?e&fAYf^&vabeGAjVxMQxEbx*F3{=XUoYHK*l zm~+@SA2+lz=2|0IB*qp*lnr_Xe9!^EQw;3f7Tdm0lV*oSr1Yjy*EKQhiAsG648b|C>^TedC9N}A z?hYPMeEhqJL_d;pii}#`MewKaVmyGAjTck`bM<_Ph!m7Re?YOL-zwi}iF z+t_m?c1DN)6?e#9Ii54Kh9o+R0{rwJncadrl~aBAh4npF2*VjI{UF1%LQ zg-h1cF)X0JsSo@ymQ7jxnaLmvn%u&ChT~vnTd~;Q*E`#1-fVL@Wd2+gj&-uW=vNuP zklp6)ob7@SbX0oeu=73PqurzIgtku?Ojmq90HG|GaJ5eZ@|2feSQtCr(Fa@XlhHaH zUnRLk4eE5^o6C-`<>0dGj{$FLnB27GDGen6GnH+;ej&G`s(V;Mcn$IR=&5Xf(XD2| z5j;Dh?76{SsbP_9^b>m7cZ>In9aS78Q&eLpo(-ay6*;Ws&clQNi=k6_4_ z)@i6L$*6+P$0#Ee;!KuH<~i))g|BqEp`QWH_~McEd!xb!A#L(+CI;Ru+cR*+=m=z)oz{wsra&=w?G z2TH0f?SCc}=sd;Kb8YQ*#E$3+d5d*=H#g&X(Q}PkZTFdbF zoUN(uMa1dNCxo@H&CN`{Yab7xr_?Xt;ak8Te|^XC`h!;K9JnBQGC3@3<-9Wvf@nEe z`Q9%as`W(T+u?zv>prf!*YTcrY|B&M_)cczT=GpwtYw>3ai1^(pFY{#Y}P)NYX}z^ zEa5X~_KNbj70>e-(piFivcj}{`2C$r5qjtMD4pc~PDGH#bgJq?axkmLG!fu5gsltznbsWB) z-za2ijMsVBaZXpoxOKEIlX>} zodCanvRh!iXI<>m&n{`#@HqL^U%#i;R(g9~?rPm&#%o!&eEcA1cONHe*&Gk~3VS(* z(&CPR9HfjI*m5Th!o*(`pgs?d;2$9@WjRUcK#KWO>eD#~=U&$X4?Hle30&&hIG5&$ zy&ACMO=xde0)Pv_*IvZ=hT!YS>UW}JNv<7-u|LVwNR{q ze+D;-3bYX?ZnJO>AWsxflZs%s7eW87UxZlcW`oOe^J{rMl9PqCe(Xs;v*WjbMCQnH z2}8>utD^@bqluG30RwC(V2g5-|Jc)2Sd{bByU9&@eTlr;*0R(OgQ!=T-hJCZ%Jpcw z93aSNeYfc}dKf~~eBOJ9Yh0Q0y45syxr+~SVGBs5=NuoBTmE9?srGiLGn&LvraP@01ij(LM2}>eEzAzf`p{IPOGINaL<}Ng}j_Yz*_hMVH1? zl<0l%=@w_0s$t)r=x#01XgpQMrFtzLjz|G#SFkscHF-q&*bWWXSE}OmpP#dvtzpfK ziE&hgf;h$NPfI(9J7*xgbG4hRr#9Ro?}f|V{-iX1Olql4q+ur3Q(2;FW^Uo+wud*witgA%A5@%DN>Cim}L#s2m4qP+zr!l4`jlg>H2jCeA zJONHM`Sul@e*mZiWGAO%#`2&hV5tpef zztD#!KcgNw5V>RN5q7BNkdjcDM|&3NTE~_YOFB;P`n|XR6r4|_{NUtp{ZQp6mB4bf zE1Y~g$wVo(5su>!z2W5_|8eNg0OqXE-}jPNr}Awx>N0scyIunD$@{{e=5y=P2|41a8EBmA#wNQ zkRIOst1k8MOWygG7jugD_;s|CLc>v)qYrdwGpl6IP3m&PVE&{b@(s!6!`?WT2&S|isXHV>f35+;a`VXJQZwsa@@*-Cqk_#2hh!0-YGl5- z)Wv2C&SmcxYQ&F=>qzbqpsTvJ^Y6rWY>%BFA<(&N{P-y8S8%Ze^>+@5x66l$eWspb zh;TG30?xP}R-;EqZ(e>9;z=+izKFW>-(I<-N-ZSFhCD0aMGvKcxz+5E8Nr?UWjlHr zqyk3W(R?LdQdu>fs9N@#ds0~^B%{A%aq&+JtGo8`PVUWo`KO&4kQb>$H5cc>;~F))Sa-Yalz$s(G9w1@E0S3dC#zs8yeO zKDZ(Q?6y+)G(7c{Ef_OLz#htoH^})#ExE5_Nm#FlY%=zoK@c*WK9q0v7g-%*l!!pX zZa3c9&Ln@6DT9$uRbfh9XPQGP+316mVOdNJWQKGEJUI-LT??{EMaf;HTpx#7_OssQ z99Vl8l9n(lO6x)xF&J;H9)xLV(#87@y$86Cr=5;qD$b!kM(6q>~r^4Ei=DRHDR zQGTGS^zpHAD1_#iLh**tGJx4K=(6y7rBkk#7THp4eVGodyi@R&lI4Of?6fb3E*L>q zCUH`zvCm5tZ!HauBl^#=QGJ?~7RknoK1w<*ZYt*sm@fk+Z7LoS#v1jv*oDquC2YrR z$y!AQpFQe@cOq0<({O7boK~MGiPEWmt7(&@De!WEn~a zbT@wmMq6;sqGR8LwW0tej;oUHBk}ltESu5fG=f1*DnC_=Df`G!Gnf6)LJxgswd8C1 z3QLULGyxrD)OXH|n={lR0rLQ$h5A%l<#pb2&BCI9EGXxn++00DMwkq8gjY*!zvazR zvxNjmp=e@gT#Xad?D+0H&*lv2d^%Z(nzV1qB*K4K?%O8uGV`gDo69!*s&+K!cpSU? z6~0mIx-uVm>24ujWrnC~TN!ZDH#+LJs~uBABT*(YE9LWw|_6w`Px#@IfC34I9)a#rN$zr zkW->8DnIVh)x}jkUp_go@lB*Urb;1#Z1C5OQ4=ysHsa4usveqU`{3L-HM%K07Tjuf z(RMY~+dFW(ryhkx$I6HklCHp0wW*Bfui{DwQ|{LGw!25375L69w$bwa*D^0h zk}8G4SF=hqV|EL$z%y&{HI^M=_WGBs{>bOcAG@x#7Z{8NzZbLD3(&8wMnQhlKtB=N34g<9n_%O76@T zw?pPgfA96gZg9VnwHmAT;u%OkQjAdgq0qF^ctbLD{lz+4S}r!JjXJ?wmU*T+GG+I& zD=khhbIUj{UC_yKe^?qh)~LiHY`dfoZSwb( z4*Wg(yhwqyoxY#Wg6XoW@^pH7Z^d&}yjAe{f z8P=f(@%PLUYVdW4OLZ{dTg;G4U^0(Ny`s@DBdf?L<%@Z%`p6vY_D zjdV$a8Dgl3-5RkQFk_Y14A94$d128IA^ceD{E%+#GJ){W$(HdCuuZ9Zh9Jrg)~+<) z7_ZU$D4A|Wt3i1Dho&oU>Afl(hkZ#+`R8k-pLq0yumQn~A)Yoa@C5H4%tKVubufpe z2Xx=Tb}S~1cy`d*!PB^)iJE=b=s?nNi>Ml$nACJjHOrjo8&^WGQ)CZEDd7pBMT2{c z6qYYa$wi_VJV?6H`ITdf+Dd1S6}2|7_Wj8GyL^_?YR%LZ1@b1U?o%vV{yii< zENHVaxZ<6q??)xunrRtf1CPipsWMLlfnC9raaQ1n+?KSEn0?X1dJF6Y3pakcha+#c zUN3%2!>jJUh8O&5JE6Z(B^mgLmijU}F&QyOA+88JajA^2lK%L0?b$TTw8YGcPj@rJ zF(r@XG|Mp8lR{$GHMbuNxl#lyu;e{!s9t0`_mIteWDT~P@(T5EGm%dEA|kjApvqdI zbnebzvH_cksgO`PB3uD37Ps&6{EZ@uFq+c&5(_BWJc)w@MlvR&K$qN?7fHHxI+w*c zfkmU6;C0#`$duY^k6v0?qXWs~?`91?CR46+`xf6H(VRZulKps0CmFcea=ilrakpPA zPgj5F{J2e@N#?PXVG^@mC%%+ui8)6dpZg>Ge{JlZT`h}|X1zO_@f+@UZQ}-AtmTNb zqL=0`ZjTc9PZ57Z+{oK&N{_=^DL#RdI zD+4J1+;?r(zBv>zX>~ShQmxTtsW5;4idD}7foHmByjmF^&e>gLP&7rFYb41dQk-cx z2Y{~DN~@EfyX^TTTl6Hgf0Pq8CYWrbYrrx?i})@m>T!F#>#>y^DM}=hBmM%mwpV4l zyPz@Ux&%s4NkbzM@8mzU?Me9L6s=^{IK@rbk-!Y@(*HAoNNkUz{d$}8>Pd}bl=e{K z-((rG*<+lzd}VRtGevMa1{r1{xE-=}3q(|sZz9{5wveQI1sR+>jKx?nQvXUUx=H^f za{6VJb4^N1%|48GwCXoe(vo96F3U9Yu(;C_dv+ihl{hVF8tHwiLIDh^*%9nQ&{dA^91E}6g7XtAbAQkaGYpW(UwoZ?w zBozTeI>PO(qJW6cK}OAXxXP^I@x|?;1d*9&8cvLi?rVbXFdbFb(sAR354`VwH5~%{ z`+aS%%cR%jZ6%gqba`vk^-z_Zvs}w5Iy>$xX^CBMXGh&Aa|bhUbnu;E^qC(a9!->$ zn#qvW;kAf{EUWPbL+kzModtsQ@bF)WpMb+A4^tDC>muz{?G!Eyp5y2PE@ZbZoa%`) z9Nup_0Vw14%?~k^l&76FRb$(PS=A~!85KJC76lRo)azoyWLZ=a&5K?VyY6k^r7B@Z zFHo!JC}v|SK`kIo)_dAdnZEk7)zcy4Emt8I9O>_F#FCX;pWodh;2w7{E+!o(BK6q>|>LP*n5G_>ca;m85oDH{W(td`A`?S?~)i<^$c< zg9WJU=%qwRVqr%+>h?x>ezC#gu2BC{fOrS3Q>xO+s#dsv`IY`CQV|C27=qs6$(bK& zpaAF55JQXb$@Mu@bJ@x6Ny=9W-esOc@n~B-4@5tj&MhLWKh8(TL}F9GWr!7NmF&%Q z94N1$(%P908v6U9PVVIhOD|BKe&bK9V!DJ?NZCxa=J}Gp;oX}FoWh&@(`qhbE=nGX ze8dCXuOL!FT!-oHm9kaoLY@;ir5R$n&mC>KIu6JRJD4Di)_adjVX~IbIffvI;RTaUxA4tC@Nw(DybFX)Cw09!I*}K27A6jQC#TK-D^71&#UC|aP z8=yS?fzPDbL`qd8NDgcwqCt-p#lt!1Ke+dsXjAyzlwzC@Xg}NAdhv6e->}X~w7zH{ zEX!v*mVDm-AW*(&$dY5`cS?pBK*AJ*4i-+z5;wau^`22$fBMO^c{D=$! zsz`xl@|#!s0KMpY+MqV)<|#tk=m{dDsN2w|mbO$c;*v;4lP8xu55ipUhypB`l# zj8$;6w@HDNQC%|`G5I#V=qka2;OpZH>zj&+ip!NmKS~VlU{s_IW3?{A_+>p7ISxE| znvKon=MY9l{!2aIbD=?iP4RsdG%#sQq*X3LDU14gJqdCraNRG37?tjg{j6) zA(?b$thx!^ak+3y2@;0@H@hc+3UXBDg5Oyjv%W1OTZOUSX?HOt>X8lx3`szMkz0rY z;s~Y=J)F+HKH!& z4b!|A*lgRAUXPphM#1gl(juZdMq=k5>`D?&HrhOC0!h4h*%XYNQ9Z{`XWhy28zTh{ zJPZZAoO;{X)c`mvR>icv9Uzt8q99x(&Di?342a|*lxS=K^umeMPaX*d(}diyEdYG5&gDUuX%k|nFUbTv&B_(JOADd`oy z2g~b71tvs_@pHlGBpjT)n-aFHHel^5AAi zNnHt|FC<_u%%YF{@XG678A>wwpow=K+a`3GBi;oY98dg_Tj3nh#6PdVKWI2pFYm;~ zRFpLD%3er(!Q<$9MsSUUG84CikwOU$lGkfU{U%{=ZDh5wc+fQW4Rh#BWz@w_I=9au z-#1!Tw7l?!GmMOTLK&2TMhD?yISLF41Aap$5}G&(j1anGh8uZYY9SYar`1HuuW_oh za1eE;tU39{Ca!cyw;g-1`??*LWbZS&3SufS;OHP=<$ByONBll7U~MW)pE{O0kTxqm z$L}U}H4L0T{rH*(}|5_H*QCMnYjm#>OA?XYsj=A(?uQ6(_##E?m;s3 z$dvTQvOcg2K|LR`=mx*@JAZ~4T>|vqoDks-Y0DV6xvi#SQ~Q;lN6gVqYr^vLee0DL zxf(;PpeXv`=UJ{2(jCZx0B>P9@RU4WJpdVECxGbk$3w|;JX&lH^?Hot+k&6|dLLNd z?Epp@1uM~B;Ve*_t#*X{V=6%qUqqE1ts$rJgA-ee-!FK!$HXPWdBU%oPiVYqE?dlz zujB3^M>uaY3K;j9xE&CfA(HOVa?8)*J$8=eXIR*HyTcZO)I?)3w2B?D?Gm~ zO_#vK#ltz!*+Q=|E*8D3E}Q4Rf#<7^8Z=0`WAI)hsspzHv6b(!*PB!wu=X)-r;Y5G zu92$}#=&VRb*Iw#>Ew-Ln-Kh>z;h6h=K2$9G&Rxx%4%H_rAcj;mxu&ofY8cT+oF)W zjF^-d-2Fp-QqzJQ=NMehV>Q`OD`R6_PzT8fOXI!c9NRQmJ}c~!UguevsE6#Blm6}| ze~yUxT|Wg(HY@K7{lip~&3*rbzn$mFWUg0!CP5k2Xna9x_fH%$Y(W9X zNg8%G8KuWKI4W=Q8U#5k&{ZsKtVB-di1Ix(7@l)nb5gJdCbxmteChZ{oh{y4C`3Sro>?m3~0{CVy_JduW+PB}U zsD0`bneE+um>wku-qUb1An({`Pn--w5KUowJoa$D%Z@NRj8b!#>N2POVa4Y_Hb*sB zr|_4LW=pdvtIQam)7&Rk)ZX}JnZ>sG`0EPwU82`O{hL@wlv5Pj2TLVV#esa)Ezb4% zd8TBukJWIRDhATr+Y`o;-8Ri#9RIqSuy)RTTt5=PgAkP*Q3JSPnE#K1l4BB+ z-!zFIbB0ZTED4fJ;Cy8d^mM#f!QKB@RI`Y((xX`7)Q5?g8Qgis?5DGvMJC1kvhbb8 zoRXrkI&Vd$DeD>m0QDm5y!aV^eBUuayacRe!tl08@NcYN$S++de4s9Sc1!h}B>;2S8DVr~A4S+~(9f2W4iXPNGM8yK;z!>JH@e=|JZ+BLMsL`41@x9Atu z1OrI+7!h1}SEtWy3Xfj%BiZ>zbJiHsmWlMB^76V!+62yk@KUxWj4KQ1)!h_U(&uCH z3+TAuH(3}oW0O5YFs14ki>AY}A2#`978=gdm#+1sU}%$v%omwB-!c7?FQiJbNZ#l~ z5*-PANm4^;cm*sK>Msl&(u7hSODC${Q@g56TRlS`r@}Z?g3HN@a%C*=-$~b zwO6p)uyJ?W`Z=ocFDo9a!RZ38b6zNFsAiV?- z3^jB@4DMS{1t7DJhmfvb}Ba< zjWB#E7~99PeuE>AC6K>l(R$}~P00G;MwW6*RpD-GwTfL9jXmWBL~7 zc#j^!&Qm`R7dcSmI&%{O%aAwGavu9v3UFlLr9W_m3Q%%5KuB(jDW0t})&Bl1Y_Lmp zUQlYbzEmfczvNoh>!-p56RdBVU`FpVkX`FP_3ECb3MdZ6{L&&pVr4f3w+`i-^je4`qm@cb_7D>1`gjpRT1uJ&%8a&-{0@azITtmj zOdpBzJxgvK1*7nrS**4aliM*eP$Cxn$ampqs2j;7ZR z@HWpKA~gk%$XS&X-`s0=*q(*QJ!E$0n&wR9nvhhe&{KE487D*+lDpC^v;9Z_8uWuF zO-cE+aANl8go7Kk46Jy3o@=g2vM;VjO92H*$NPDNV=I4n<>S2rzlH`RnjT+20X*EY_}Hm^yjNd#I9j> zbx80@Pvlkc zu=M))sIJSXyA0%5LYy;p{fZ|Iel=cw2O9c)dmrNMZm*h+m`iP#$k6IF>VL?IYCtjM z+ugZq%UfsWZ&{E!amZ$k>EJyz*-w0VCG+uJ>b3T&x1(#$zUbWTFkM^et4Hh_6tK54 z%U?ieTUun4>_kR-?CU@eRz8tlKP&1R_hnw+F3-xjt7ACV+j#AiZ^7yL7yLkVb2h8o zVEN@NamQy+IB`_qU^^!z30iMmlMo{f4Qv#ak&Z&(IIR!^7Hv>+6w%x;@a#OHFBB(g zmFTXI3=_$~fUK$PzUpVTg`(i#4+$)8) zyJDp3if!8Or-A6+mkt4H*vO(}V!*tTW0V3z<&Wp%o8d%gG{VF0tH@eZz>300TM3ls z{#x4!MoHvPy-A1;zb5fj%j-M64Nd-m3|J^v%VXj<0z#1bvi=wE*nRijYB4`A9^H&y z95$=aUPtmnbVi#Q|>M#&|{ovtL4Vr`Zf~%UP z_EM3fV1Q;+7a(z~G|8+YSco+Gt22DWb32KxP3w!fHteRMZc=9tM>43a`5XPZ&!Mv~ z3_*OVE69%+KN&|*-GxV*y{`4IRm)>Ek@&mONas%I8Yi5{3*Msx6V_9>Y?2d$K@ML` zRa^$$Og0Rq6ACIgSj_3}-&GA?T}pS|#Lo<*3il;VLv(H=4aK}9vVe{!**ln&!W8u6 zj-ujd<~pPtqI#97cE{DjMZ-heBXoW~ax-;^P|#7@oz`cVavfa4i%HmiuMjHpe13jP zC}1yDx9+{FX(}>NL>~2;5I(sVFYhxX7d=A#S(Q%IO2tk2YD!GmH6z0ozG*$4$8#oy zwY%s}+9)^s_rVS2bP^^EPEW%MxOt&p!Z5|Y;qMj{MVtP}7mS`&qvZ)#yHJUH^QG+5 z168?0Dw&(J6t4i;&$Rk>rW+Q-KK(({7^dAGCbV)*{7Xy1M)E^^wRJ9VwEYAPQ}NsptIQ%?Yg&vdFu5_ z*Z0+fs0^QcE=7F<4<6GQsittfbW`RU5?|@oM8Fh!?5}21vAl^Yb|<6~JJa^X@~DK2 zkCy77*VUv^kF;Gxz2SD0ky?GEyU(iL#LGQA#!+Dy-en!~m^!#ga^p0}nAyY!&1zb1 z_*1%P7{26koMr!R19u*;e_!yrR5gaHM#ix7J0Bql+%5V!?^WsqzGK$iYXVxKgi=Ns zW#t?8X^UHKcJ|j_)1F=L`>43jb|kel6;1oOYrh>}{e+|{C63d#`pYZCCr8ecc=@Qa z082=aNmc=h)B-8E@x?x?jLssePwCbdT}oYZhur;aUWZRTqom97qstf2#U{~-J__+3 zmI9-?SL!n7*SNIjTn>uO7?Rf5>hJWYpMQN&HLvo%IVeh9F|wa0xz0RrIZ9O$C?1wW z1+p@iy;M70Vh*JmO~E@ppw9;t_&gr3$jWGICtjlW2!&!<{JDv(_}Wik8S(^IES z5bcL5YT|{XAZ>Mi_s7~^R&jBmU zF)5m&C5*x);x#XVN#w=(cB_bOmMvRluf(1(pDmrP%tJ8|3huyD@DWpZUbTzRc4BSm zdID+s13}sEM#}HjbkdP7mvN2`RH{MP&nY6GTc4sx?YH@o&qsU@bT8sxz(91E=#sL0 zkh=*NmJI&HiexSbc2wKS!PzO>slLu)kT9un2c)Odj7BPPjzFP_H=bxy_lb3()B7en{UFJ901|7K5izx)*PmDf3AH*6B+w6}Zn zOug@HAer_)tn z`1a3hnUS^wgDN+jXU2mpKni{4<~!1-z8bwuA>Hq}~+9e(bQa zwGIW!)t4>3(lMWsexPr&7XElUNYnmTvw3fItpw*6a;m|^96iPRDDyzm-!Aj_6|#Rx zar@h?szEUIzdV)>KE5uyXyk^uCIRzXx0kL%4t`c)mITWPaX^w2hI|-^HlK@n`hhwO zE4#7ERQ$T^*#e<30<2KD+sd@gkz8~~@{rHxc7u5&?%T8(rP_x?huce+OUAB(i$gMJ zJ1#dfI~c51LRkzqOIwWrUKepQY}sp543|=W=b{LveFXM!GSFw<8Tei2GiYcwfvce0 zRX7>V*VCTszAqlI`GiGZPWwauy*%IkCq`9}^lv`6v1(#=`P7Hk@!aPZnQeO{Qy=0f zJME3f?u8v164R4eQ>igmiMq^9RoAk1hiE;!@>y4n^(TitIE1>Un1Re`Ed2&G>E|u_ zn{@uUVqM1{zh?5sEd<#2Op2yi@?-NI4vBLl58sI;tR}}$iLtA4KeSVc8FjS`Q&Otq z>wd^i$eljOUfXK*fuas0iP)~f1I@GP3RfSmNQO!|K5UH}q>ELnufaTGH5aL4B73m* z@Q!7hOnrsol7yq(=Y)HNVjBK>Y9lpI;IpWJ93Fp_TPjInfNZZb=uYs>&fw>8p8@__ z;ceb6iC;SmYF?r$%B^XvQ`Ik)R<)ns@Ah3GQxzf!37mUJPp|ZJ)x9reNBa!aI$R3q zZa*t~)KrCJ_Qb34t)ClVF=>U->Nqoz2%3x;lZK?-9t6wL}#-Z)*tcr#5lF-`k|GFM#-(bnzz@;ko0N-H22JmW|17ts1hyh_S6-=%XJoZ3b%L40Wt%{oj!dT0JVbUwvdo#Sp{JmKc=>l1am2U zKWuXz46zG-8+u-uD7hZM$A0=HMe--zlsqT?9G52&;gvxplPL4Hyfy9)5Pil=E|Zwc z0&DaRIbt(OAkpAeRK?*l^#Z0jP-$O_^4*;|a8g{J>XM+I`euF!C69S+rN|lbUas53@_rrQ*{KXqLVXwx z&#%LA_6QI3PeN+6gDls9jIL7e z)cwaWb0T!6%-$DAuxpRegJb9)mFjD%AGzQ&-e?iZ@UXs>Cbx~TM%P|kt@t!sNPAU(uXRz}#i~`*Fu&+16=)S?Z(yq6E#2->4%VI8P$T8rgrSFPg2* zeWNPxJb4;RNOs_@^^Dq<1DC01X8A4h7N#lc@w9LyTaHm$Zy`fiujw>^#y%S)BDYd| z_v^N~wR#19du~W;!-yS{T<3f=go`}pRnd4 z2>0`EB!RK(QDkcJqo?>-_K76ymiKnEO~dgP={fF$PU^|`-}XP1!lpr~B3m#*UliHg z=0AIcb?!v1al?t!(+hq|8r26RHh~uz{Y7{f<GHqyEg)mHVg^jXelpWEt|*+E>%k<14;_@&n3z@(^>7#`krcvG{>778FNU6;bas-_USFFRVICQzwPN2ycYRiI(T zx+8jb$%+VK@E{=JhEY3$zV8VM0RN=OJ{HU+$H1fX^U)v8n{uu9kVi2e3?X+;U;M%} z)T?}wmTKt^T1@3P%?@+(T))MlxkW8Y&2=>x6$^B}xeGJp$w_wOd3W!Dgk z@wZjllFN(=w+pf7AeTv{qn~&f+Gc@s;fQ+5)oDAfu9$Bn?#8grV1?F?>v8J_Od`z* ztc^7n@|uEH*KX|ko~~TT#E2S>3FFIx>pIqlSyiv%TWzyM%T+ZOm4Dh28l-@z1H$4? zSrbZzhq6FdP9pcS0!Ab0GG9y7#~<_|rx_+_3f0L-$D~!+;~ubB+|rvDE}N~99Y1?aiEhQ5RqUy*TKc=`@T6RTBt8PmL-mGljM)sw=0TH$1D-L=-X z*VL)|MTco0w(^-aOwV(xTMis5>8P<|lTjcukjHQrp>YYijk(rmEaPB8e1mO7X@Om( z-+@P+wTp0NDtljvvOYOJzKjA%_V;mZ?7Kj2*6I++d0F^L6a- zRkj&=U9BGZ8m(CDlhG(0V{D>fe>21QIL3D~>4oxx5x_;mf!RFrlB>@x>{X3(zRerE zCvZu!;RmSj^R)}JdAjPjR+*I(+;l8~zvmijH_-CzOZ>7b&lMbl5>pI?NqX*;s*)$l zhL5pcG(SWv6?aJd0(~pugU_A!lF@JTx1p1+DeuO+!_E=Du?qni_WfpV09cQiJ$=iT z6sNAlMM21y8KwRXFkB^LZsfj9&9Olz7N#CX+Z_>={TSRL20#XKa5 zrNr?eyz!Xr0}7VZ+ro4BgXG_HsxwG!yP4)JtDhRZ?GN+>)zilRw0P4>4! ze^SI8CyAcyU?@&dYb_t3MR={Uy^(-`ZO&D>>H#h*uIen`3sTttj1j>QWm-7_p^|0z zFwm*f`(r%!sxRbv*WS@jU`7)8Boo#S>M1=2&jS{M!+-q7@#rb#5WIU*>34(_x1jN; zl7_?74_eh}Md#H%$aqkhskeyN1lLX)H=HX2s-8cP1Q}TQoH}g6f`UYU3OpMJ$r)}I zF*PV^^&qRwKM0vMP7$hyAX9T&8^1&xnsNR(Kr&BH*7#O|Jj~oIuW(%J@M1mE_5j2^ z&{=WPD6PT-3jd=El=Sff{Uj59QvlkbE6r#TB z`PPn`)=?VF0b&6aG;xQ_h6GZpSP7D16kEdVo#Z{@&4+*mXzGvy78UiymQ}lif$J ze80qo4z`QCItP8>XP0`N<04kT@Is5)Eh>aAF?a1q6V}*4&cC6V3nY`i^iRKvNN~6% zKJ{_e+FaCE^+JUvfh^tbiIiG5!wc2|NQ@=18c8S%^obG)v&r5*AKlDFg}uo)ADi=yPgC2`y~B$^o@+Bd{87=u{5Ucw z=dQ^4ThaF;L6TNO5fQ)Y8zG#OGSImFD>+Kb+zrW|BXL~0lh)C88liVT({S!{8OhKd z=$wM1s(Z_oc-<`DWCA^+PAkpS9NgHrwYZ!(r8zaVO13k(G~ZHaMED&g9DaykE1(}7 z6r&p**RWQAJPrOW5mtc8h{=$g$gKSyN`@~aDYc3DB&aZ(CSFEFEV1QycbWN$Sm#t* zACZ3KJ|Ri)nZ`A4)hS}`XFL|fE9n}ibY0Q(jbclTbky>Ip3ly`hM$L{Wp9{jZQ3~4 zB{~Nm(r!Ces!LSD8#s2c<)5sK8j{DZLM5YR&oK(aK)S{8J4g9lxR8DETPq>sYjFeZ z^HsKS0dZq-Kbudbg{ImL$Em`9cJ%Jj+o+Ak!o(h5(rUE$_d0CD=rzmyp>zktM~R3k*--aDhKpFI#to(XNYdJIj{AX7_0?k*EyHC4EmB5^C-jLy!G#kX)}uQM49&>D$)kWX&+{zuXVtLIISw*ob|H8ZwwO~F@arF zbbUc6V1=2CL!8jOOi>0-;259~EqdrX=*8u}^Zocj@(aU?E;UiSI3nHXXE^4!q!3j#9D1vbIBIu$|!| zHpoR)S+SHqM0^;ORsW^+e2a-{n3P4V$%9k|=%%L)c`BawyIF5>p_6sbZy$*rNhEYV zGi1p#2{Y7hC$3Cdwi@})vMY;Hc_Q(5A`=o;UPc!j6>Q7kh?Zse!e=DEHdGbrW#MvrmV<^+c@I*LA!`tGqlQDf z3korZKzPEi0)F$<9Y&sORZvAfbt~@eumlYiGGL#H?c+@ON7GyxXWBq96a6W9!1f02w<&G~wP$X|_t?>04Ch~Mrixa_7K z9-_6^ZH7yg1|kDzn|Sb#t4WCc-z%!RdOUzAt(o@y%4gB7-2-YUN#MFO9!LM#Tp|oe zu4WYrxG7LJ6diYI$|0bqdo9&{ivGq_Z5cGUQ7Wezy{^3i3EW`(t=GZL*^Srok&fkf z>wjpgBz-j9K|#0)KysoIf(?JsU$fdo_;ys@xEKc z{PDKMR0EEqB+*RV1>^O|iV(#c<9F#jBi?PGd3%k^IU+@bMfwq{9J}yVt51+%KZ!A1 zGkd9^;|);E4m}o-)?GWAlz{j3N6+xey4JD2YOZ-f2<5$%Mkk77PSjtSR@$GnpYp>I zvBag74PRNjHdWbS8e2s)4+VFd`mC>a>H~cfexPqNi6a45xfdIc0Q}9%;u9xrn;CyK zXba7*?*j!9k@A2xzg+#JEgL&@Nei;pmAY~h9cMdyuQ1?}r^kBngIuKEP>Q#yua>&m zGyi5!ii5<`^T{;(u@=WHx{yH}Y1o|zdJ6uiW@h`yn7h+#Jd)2+n7?ou=lmj*A~=gb z5Ip&T$~?csem=aK@bG5N<;tVSn|EMlfR=@k6JN9Vrp%oSPs z@P(oJqmajEft*6J*n$!lflHS1SQ0`LdE70gXVXV4Q%b3 ztuiPIk}2Djjk_P68urRYrCvxK5bIxg7m?J&s<35sUq-u^c%z&}M2$v@r@6qMDkP4{ zZ@5KS%_2k1h4aHY!x)K0*uKgcw?>?4G27@i9SsOkVmPI)$O8H!6)nv&VEgtI8K(ee zBo{r`h9qXP&jru1s1Njh2YcT~NqjhYzKl~v^!3SGiR4DK-Yp$2p*|F=@>FW_MOxa7 z7Zt;cD8~Ud+iI+192%6v*DzNdwq9;YWf)ShMGTQ{N!#~tEfnZcHPQ{|-n9YBXqCGZ z3vB4K2$ESkSXz#l>YmylYyoRg=EFbuv5PZMbqf2$&Nv29m<+h4j?S;)!)%7VKCcmBze=J?wre%Hk&G7@Q zODpK0Zd|g=wD~2RHrK=Lhn)YY64pXQ*3cw&+so5Y`7V`Jh{PZ}7xI{9EbjbH!>xL2 z^t((8H?_AH#eBBcQB;*0%H!{QzMTag4HE;4?!E17N4~7BTQ&DVRBufinrbPVHd?sy zXLF9J#37AuP=q3LnKt+%4$Q8G(%m35`6JPMsHMqCcSD!#v_A}S z4K!1aN?}1YqS*0m;lH#SN(x27Za1MlgzQVRzKVx<@x59^Q4m8=DuOAAsM}?XyG(#e zi?UlyOrj%}uDAU}MoenA4}%co0FTY_veps{r zxTn0W>kKE7$+Wg87=p*>uDvc9Ensnwa^BdyhaB%zh|QpS2SK`nE~tW1XZ?M%^d>|lE>73qG^_9o zxB}@zJ^^Q~`P_sN`zeRgmhWG5V`G=wB-n*E@G(su3Z~#;w1J;y9HNE6DXTR6Ix=LE zR31>UbiSbg62?w16U#L9>_h2{!B{AX@hACtwv!QI0QZV`#xObMRMmC(yo7?4kKD;r z7m@>vMoG18Wo0Rx0T<8XXut1Ws({N@JC)9aFec$0AOVrgynDd>OW#ddzaw?hL1#0L z&;BN+UqqaYICK<`y4gftwi&=*{%qE}X};?bw`u{t*cD`ax^)|1fc)Uie&CBf;haX0 z%U^9?G@f9KRrFtdkD`iVu~5QqhOd3}yFGL6TXNU=S@T;;Ea~@eJ7XDhSx_=m3>66Q z{rIk#U+wjK2)hW8EvZMzucMv5Z|9r)GSLD-9!B0L!@}KD9r$)zGr3i{IqUV|Ly0ET zca&%Up_{xO(EM8L=fkQObtq5o!(kW1)ZAI)3&)w4dvL=dr{8@idfr33#ao|Kk)1PH z{?SEVE!FsgXVtz1y(dGbnQduh#f{fp{hS(I%U-s!6iqnVY(=j&&*!k~YYXHhVFH&g zxq&__w9z<1up~+X&IyAMVLo&eKLON7@egEQDX0PKqx9a0>5Y70Y5|AE!+3 z&62IRJC1y@7o5?CgXT~?fuYvlqx2kqtNs!;HC6h_6k1AU%zVpls=R_)MPVyU>CV2! z@0WAXN$bf+!hu%KyG(D4cP>zQ($_oR>@w7Lx_v0=JI?V>)C~aRuw_olouyw`9=gq{ z*a|sa6mB0-^I7N`=9XCp3{G_mfo`E{OLY0m6pD`K^Gg%iEXKXZ>h9B2!= zKXk0x1U;`5a^f8T;F;|*1aiGow;;RKK2K+oBE8xGi}m+8jLE#%ebU~B8t5V_AVlU; zE-u7k9FKnk2D+t(t+P-@s!H|## zx1CJO8{gpUsRUTAh?2Zzeh7h89L}U?`MD7D+GbLbO*$mv+X4KtBG4xb0fDbdQk(_g zmfarvM8vz~fu1WD)Bd~d%cK2aSBo_ax8}7U>{ZX94Dk)Ga9zB8&gB7h7qP8HrrXo* z^U`4jesLYU#sEZ%JJdPr1*V~?$URR@u!>8RCe53se$&mC=?R^!g1k3Bdc8k-fBUtv#SJf zSoDPmi_V_aRFzRDPEC)3E;Du{*w;SIIYaBO-Y?DSB;Wf~WqrR>Jf(etwTA2UG5I8C za!)AgKFT_1#S;gIOnpeZKL*a2;;zlzDz!6JvWA;B0@fzMM@#MEdnVSJ2R5tlkGE}W zvRD0sapW~}AoPdNhzI!S0Si)73 zunMYE=7T_+&7T{0VU;wKo1ai}h<29|h^~A)4^*fe$AV0y92N*O1(JhfJz8;ue2frr z#i#|}_1sKq2A1BcVHR(>MH4ot-iD*KsatDZmM(a^*MG&oS!E`s<~+wrl<64xyk$Xo+)y$zjh*(-6ru(2FNiJGw_8I>u!@JJCsA+gmNEdqRPc^_AhU24 zrTOsII1>OSW3bnQO~LmnhjJ-|JVKeZ`=9>M^@p0w!WGEU{FoyK+$laJgx&Zm#1|gC za0o{lUdQ#!`fSxWS%wQ$;0$YDel`gU774LNElMbE|Kw1eeE7+$A3kb0>+Q02 z{z52S45my~MYSzM=B&mhaPn*&LQr%=DThG_!6Fq)_Gz<7;XaJFqAt&EGnjjh|;TyWEq&kC^oY zobUL@%uCBW0HSkVut`>Sey>xK?Bvjsp#>DX0X940@0%-$;Yg6u<`CR6H_jc|D0{2f z5gc@JG<||ecimA{aMBqd9|NE19gbu**6iME*o{BJ_SV#)Z9?3Ci^Wow=^gF1hku<- zwU$Ln%;%)Jx9*^LRPo4ynQxiRunah@1~gU_aSiIle>S^_vZ7VA`}NySUkN($bVEEy3U~e{F=?#XQL*s%Ui@4Ds+_cr+ z^Q`{0&KWT9D>Z!TdM1%?r%lN)vk9!kiMj*eJE{n~u|*OL`6=!kvLw4NsSl`-djj`tFrgT05%evz=5e@W2_DYP7N&)Pyy1iqA1CnlMdX%3PGr>1_`$i&^xU=k8Yy+8TDw2hX z-}Uq>xdu|eYy_RHk!3hv`IRyxBjIc{3e#n=9miC49PRu%|R;N0l3B#?jNY zizY60(VBAjc>mtc$>swAS5}A~TVEC8jzTZ;_$X zWwF9HoA^kK$RxSNY%C>eS#fWX2FW++T;H6cbH2w4T3Zg-KD#XPlJU9$Ob&)XFN3aR&B za~MJt5JR93w3p>L5|}~pW7rm~XLghWU=UvHqe#*YzttG)BUbEBIu{;Y-o9b{qU8tf zRSml>(*f{O`(C;M4HO`L*bzY0Y;Im|#J;K@y>>zWZckty6igTPUqWMMjfrmDGvq6q_zB@JD%xluv_yvzQNdKhGP57`B+K*_H4$}OJ29S@A~5Mv~xusp)@8j7#_@caqs z%&%S8bE<)kE0Nw6*0pIwlFM9JCtdl%X0DLWNFq_mvG}- z24!u@t5(Q04;%(XZ25va15&Vl%f12*noNI z$^w)C5VJ}%fHZ!1k?KQoy)puV`*hpmoi)mRAz+V)2bEEddS9=Hk!J(c37kbq;`^ZW zs1ZSZecWCKt@FOhjerOG_)Ah!RAuuTkfZLN7HZln4B_T7wxCQce zOqU6$%LD{IK8>%rBWu1W?$a3Hd6>}y3JCMJO+)*6_Io^7J+2ngH&3mLLN%0tGA{!Q zLC6f9sbOm+-CDD-4J!MogxEPZGtJbj{3=Zdd};PR5DphL7ejm8H{L$)*eG?*@`hm# zOpZncIt$mV!2_ixvOj`s^+;R44cD(2AE@tVGFAuesIDukY_`O*tzV??J;XY?n7`0xP_R%RW{#QyQg$oZyE| z{J>lJuew3a`DuFEn;bCgS3#QCXC==J{6@>g%zU>xEwJVu@Pka$RnOAQU;e`wBJQI$ z4CGwBhGTMz7C@yB7GCYD9S=jr&?=p?7dKBsS zfbE6yJgcJJMWl35(_%DA@N}U)U)YQi-2=1HM2B<(QhF9%BR|b&zjJ;&{4^stphc{{ zQ95kmsGG0MuzW6k4z~gV<5#bLLUfR`HJ*I2wgvz_NWG_o|KzpNfd?&Iz?d}Pb@Q!n z@XTC}{+^F8@t8EYtk_<=gJo(s>yRCLml#;`hXwHKydEc4w1ree5_H#hv!L&YoDviv5>us}Kc_lAN~Up*91B3qbC0oie)a0C9=0M{wB2$OHD z0YTQZ2Gj^f%>EGIQty%9L7Jf2{KT)BFa%7cfhT{J_NaX^Sh1I7!BTa%+2521Uuwo! zu26neA6jSTb6_QSbp7c30Ld6P)k8jYK2#av=XTb(v}MDDfetnVxL|}@0%pkJK;uzw zDYO|fALdV^@co6d+YF`v3!~*|de`$eb&8E%6R1;EN|cwxWUiQN(G%2+-)@Aaq`pL6 zPUb!6vHjh=H`ZTK;`qQZ@~-LG(?X94!J-yf7(SJnHQgUqyPRphJ5SyJfof!DvbzDL zZRBwNeMh*t0ERWu^8MY_RYdz9r4Re@tjq}Z+r3gF>Qk+2K2`)|4Llawz4Uo+ zu^RU;S<*K%B1r$AkJ?78&;P|lKk%+Vqk73gk<(Y0nr>JR{6wxrzTPjHay1oQk)zy#f zK&~~RH<(7hnDYIg+m0WHui3OAQ^z$g=FC4R-V_^mJ4fl~}1=r2|w*3DhR1&6gf7M5+gU=kH@=G|NV5Uf+zx7C^;<7av zqpkn%m785f2P{E=7qe^y(+kFuTVssQz6WZA{`t9olV+g1Cb)UD(%5I*d4i_QAH&N5 zUhA^B3=~{V&i#kbYU0X)KRUL+8|U}DXX|{5ekc4*5fQfei>A9~WhT_fvcMp67^HCW z5}^&rEBrUH|0cAyz$#K~?0E&9vi`a~dw?-84y){erPf~nu4 zjS8DaG=D?+TjmN7!>0CxooaMCR{>&5sB!A-*3n9R?!sfg^}y+|Zr9Om`62I*sgM4d z=8vb!9~-1xp>)>o2+aXW$!R%$1{0GD7s|@Y_V!MO_~R?%Q+;mo>1ON+eV}_tL3r`+ zAGJc&#d^AjN%R+ni?VYG85YKwsFk&)<FQfe~pm|`~$oi%c4mq zwzybG$fS<08H!W4v$Lz%$T#={f&YBHpY_j~-AWfv-jPy1EqN{Vywrr{w!F6*7!0m} z=;r9^KVmd~`G0T~{|nb|dw#gxxd$?3 z#&h}J2ccvLfM@RFWgG>LHh}P79Q16+#j42ofgT+i$+2#bC6FxOPd|P%7 z4ySc@cklY0^M3|0{xc{gCG5)=&L^M6SBs-#V?}B>{+Ap4-~W=GjSlR`oxJOCT7W|E z@eh+H*Ys4wAzH^4enR(RqaLo($LpIt3y_mzJ^bd}7MEN|F`e|jnLIYlaTfbQ)(4!` zJ1|hyDPFGeVSN0#(@c#{KPUl39e#BY@)Ns}`qACv`afdh&hjr|>8GF)(RKs}oc$Eh z9Flt%Fp#kC^oC;BuJLN^1LJp&tinHE95QJ_9{H*+cO8SiTZ797?)gb6-r7p2iuc(* zQ`I*64#n+utqr9+!*7U*iA8+)@X*RCHwkqI&IVHg#wH{r#3d(tJyWFo4^cj@{!=PL z$5>$i`F+Z4pS3U3bYwSWl!-*R&~B_tT8 zrlyWsxK`_X;Y5Gn1Xd)}mHQMiHvA*X{D*%s;GsPg{BwoVuuLdiLu0WhjM5!6lU!G; zM{=PcLl#RF)lN)o;hrT$gF8sA+=iCLqvYZEaI)^DeHAgV#Z$?Pb-)KlR+09(+b`dA zG-TG)hE=VukLEm(iH(k~E(vA%B(dUYRW7_*zaDT=zU2)w^b-vsl7mbC!xk=W!9Ruc zDJx<7sh+I2fZ0l2eo)lso~HINrsw`>X~>H!SAwo+ zsi{0)J6to{*EgfT=*~d5N}$en)(&G7OF7Rgf=BCWd*nWe)p84O8 ze*RRGe<@XFj%z#HTkO<_rS7Wu>gvWQw1Qz3lsE>y;SwK`|3FWe>CfH!^~ZZuPt4`J zJhrMm#F3zz|3~X%4Y(UnhVxe721B^9Z})OT8#rkz{)&h|+0(KINW0RId0b47625Ec zYrwN(|5T~E>Zu#!SvO*5n|_%?oA}t}JShnKvFn>odbuqe*0GR?Hh}Neg^VQs)|~%N>Rf1-Hq8BHy{o4@HcLBxL4W z{N+7BAj%M3IOp?z?8LQ^sZ*eo)C9~Gf3#z;^&i^#*}Z?TKu-peOmufGoPK+}kAN-K z<(pHv?67?Fm|I?8e6xK^BrkW(w@z4eSkc5oFN=z$ve9c}y>V?B= z?+;HUKWKFFxvE>EoQkp7^jsXqMlHjzP0O8Z|Do}d9{j_9_eKwQ+i=daZ&Ug}InL{Z z+?(jFhMuzm+`JHW;FF%V0X^*v7g><$>`l7p##5QOnk;|tuqX(PY(BbZ`|=Cxn}T_` zN?_|p)aUF&QS6d`BfOJdk06{(ztjK@FD2eeejy?}Z z5hQ1ZBYuzIB(d&0)mUzLw|BPdf&##>r$fWTDVS`6t^fZRNbsiUq%H-w(ASkfNt(`h z7FXFw%Y^m&xlbQ0PP@b(L(vqod3z>iHuwIM^46q(QGRIgtF;O(~1o^Mk__dWVLp5$KC^d|p!lrfKP<3;X0h4|)C}lUs`? z?IGcDHS)w8jS}uFz4Y)tg7gl*8zYc$zD@)fPfJO$Y6fp>yo_G(n^vBr|Cf~C6; zT4t7>IfU(;`{R($qiNo&h*OPE4cA(~>}8aMDulo*t<=@YYMiDlbaZuV@H?`xO~)9J z%74vGNthe*WR{EyAVHMVk`?MKP$=WR6p9i`y-B7^wFC0-I8@D|%O2M!@Cfe==AaRo zKV(t>UPcGrH-8NV-eXC^9v_R$ydQlF9kJin1zN;Ryx<*#cG-7g z4^yqd!=()d1_l)ntc_sE7UB4_AZWL)wdN!bD8baCprTSEkXOuMoZt8s{A8soHH-2O ziTZovs~Ig+QoxynsUHrJ={&1-f2^j~mEl7NYO%5fw2d)-7qdCDx|5<9$EVgucoj z8^Wc51b0Z&Q$qB1Z}SFCneNu&#D0g>$be`qCh7hBwOGLtl;@f8n%SgX)GREro$Ys<;BGj0imcNc#p7)FW1w=rF<(6SpLjOf(`?WvFEMyd1w(0-z z^}J-ysW}%=Pirn1cHYv7lKTJnde5jPw{C4%MFFK-DAEKJRH~x1P()NL6qPN#w@^Y2 zB(#7?7X{oXJwZT0H-r`lCG-|Gkf_uUAQU0A5FijrAbfewc;9o*^NjI+W8_DE+~dC2 zSZkJR&UwwbmgR2gc7bcz@yMCS;IQZbTHrycooI_rQg?YtP4`};L-uxDI(+ZFMjoMVZrNl9xT1F+U$xoND@V^)E}tNz+;mZ}7>%CU^>r3q+cIchFBo0x! zny%#3 zHqrLr{(a*5B%9MAOHlmZu)A0(&?bq(`)aqF^s8i#5zwVF30E1}Vas!;QBwcG6(}7a z_~)Uc6o23p2UkXBeyT1#0O$}OqX3%VZI^;Qn=g;xb`a!$tHvaIYW4bA zRA$ness>}{f3STjvwI1@Y2T-k4;D10o3GhD(!xpHWPYaxlg%&uwQ*>a#4iT`V1G1g zJBaK~V7qGnG>^cV_YwrNH9u?XwkOSmkMal*#MBHn98Nq)1iC`~+nSO%xY8wzp4;+g zVM|>st*>YGe3DTKHK-M6TANJTOrI-jRGJ{ul)4vru9C0mrG`$;iliLx^wuhfIRQYW zcSWv$R-_I1y}A+n;(D;@FSKLbZ;v6!W9x;pa6WKd6UbdS(yws64U!5+bZVfd$+qbhM`3MQ?h zwey>7*t^ju9x(Q^K)G#izgNV`8DF5SF|lpGaMyij-6)Z_Gm_St#1;bL^e9Z+tUTn0 z3JE?kT1LN%3)9DebLlqt!AT61RJp8Rnit<0$L03t+q+#BlXk;z{MJb>F48f0k#;wm zr~D*arT6_c3pM`I2J8|x zu_Yz{?`qcuv2*O{b1)9U;7@0!B*JuUnPJ1!b1XTn=fy9o_u^5G^-~r3O46e?48lq6Y1B#ZbQv z0$mp-60A`gya4EDk-c{r%#5ry@s+L;kkvX9g`)aC^b+;V^@!f}&T`ODlD!zo@Vvl- zY%?sIr^h9CvmSVWpp&A^mV6McML~ZhsjUIUfzFVl>>uYJk@+M5SgNt zk!j&R8Fkg!%C^~oMgR-v)b{t8(0@djP87t)CGm^o9=+0j&%bJVjM2l<1Tcx!i#E}# z=KJ=BqqgA!%!xb5?EUh3iy;rXooBO^y^g-k8t!rA2I#kLF~5_ID6+1 zY)IOxc>>CK^I&gAe4SQwde`la`;0~v^?aH^evE;R8vj)Hxo0l!vIt4^ae=MW-he~7 zf&H%352ylSUqIC8Tw7aBSYHJJ_{mqB1*;-X-HuBB=scC*9M~q>auafHg!-+X`QcrD z`f{em%nj5)JLdyi2KxtXu>wdGr}Ezq4b!%fFgpqlhUF9SWPhI%`EG$e`Z z)`=P#qB=#~n^`uyzp|h~4jkHdQP=ns9#8RXue?ELl>m9AG_ijkq1xK>4re0qwH1Tp zedDpYj-}NpmT-G=*AjI5HJV1xVGT$;@n@=*g3K=rom*jDIX%WG&`?u&vmA;pymID@~A;AXHVuDEqIpC$`v-nsry5uVdSS#OPxLI z%0d45^J!0SFVPt>s4dOD_rZ6jD*^q2nIF)WLNGXz08{Y2``asiPw|_9k*j$4O+l4w zx|B$E+*VI==kfE;tUAynLj#c1_wO7AjUPN158u_~@aEiG6^)s8^_(5XpQ{aOH&+sQ^^IZmATS?lqHY3{QA#b_$Nb=x&^e<#zLyuCU%Jj}~&;ci}*q+{a$g0m`dbo=n+>i7psQ7>uzsFQ}}cl#L_m!6!{8XeNk=;0-a^B~5T;m9+*4gsAbWZMRgK@-CXDPe9JPk#V+{I=d6LU6X=8ggdO)HHi~&2c3#)u0>f#tf~r#8DQ9 zCA*$RTc3D^o*z@`NzY}@Ra8^MZaZuHSE}>O`wYZd!JF3qxh<|agh;{Ks&*t_$5Rr&|d$MFi9sTnLUAY0CzbjdXBZk?<=~Jo*sN2zNwT( z^0UjNdkk_#kR+wWvf_K|)cNIqHtB+hjn5Bj31c*<>*)_E3O&{&A; zk2ZTf9@pn-x3isEUldgnp)Ufc?7rZ+DSoH$!W5)! zeXVld%FtK0?hwu;-c-G8u)D8U;z-GZx3!ZM>hS?%)iQaveG~mBn~5K@W3}Vv-@iXE zzN_IkeqK6ySH=;x{X+aFolGS)VBd(TgT7Ix{FoiF9DWUJc!m~yMkCG_-MkP*j%Fe! z<(-;%<2*R=X540C@70qRw(xKO$#RN!KjvV=n0--awg>lQkr9^(LYJ7xn8;#A-<n z^77@EJUEm`U(k*DBEPB+8=1KO1vFwqBW11Ah=jzw%$NhBk|sH;$|KU?udmOf9U>VG z9znK+(z;>Xsac#wFN|Y`yeH1WIUV%K;GHJ-~Uqv)^ z(-(Q(L<)&MliJ;IyX37Axaw|xJjZG9_S-6U&7$;Q>4B{2O2kSaD&}g7&P7tf_&USB zwFa!Uj2}yU4-1DJdNcWprg8!g(BEa>`_`UtRJJ4u4>RhQ&e%^nAHN=A^IFGfbT$s5 zxbDon0wJ;U!$u#zKN@Q?5NHa=3ZeBxIN}DYl@$qzJ>n4QyY*l5gDlF_4xsl=JV^3z z#!bnMMXE_Z^5zn3@=*O-J^0Ut4&S=3^E*pZN(s+%n1<7_ft<5WM*CI;a+e2W`iij`AL>V_Zd zB=LWotv+erT|MsCCS81{8Ds!Cac=a}EFNDTWJC$wb$@(T*bo1})=ZyYLOZRWBfK)= zyT0oYy!q&=r~GO+b53)nnZGi0$}_86JM`Sy-}cH=?5hfsMvZ%Nn|XQ*F-Yvsj9aVU z2e9uB&Ml&SL|7AXhUjvM(VWnhBbY42L%M!Yu7NVW=dy7XSemUqXv|5gqfv7F2#+Sx z*AyYLQxZ?I5;_gil~=sh$0v7v#x46z)wVrHK4cS?7EpVOLGhpdeBAsq(1`xn!n-Xk zEt)h{{#z(JT9hfkW_X1Ot_)~bdygKs7%6$!uluG_5keH+w~8E^)FyVWx`|i_Od6@? zuNNFw$4UY-2JS>I{w&f^0;=J%W-lA%HaOoY=CEE_d?$Uo?`EC<l?chGuxBG=sdQ@WmhnYv>Vm0=$Px?_-&(!SkV``@V`oV$+dT{ohJAe z!2_VA=LN`$9V!7m`7vO;{AAeeatVqO(~l&i*8#bccaMWe zS+AsET*RAO#_MB~_GWt63 zqZ86hZc9vSXyj4EgVOR<&po)O4Frk5F|4h+(;9?hLJH;c)JfXJb?ac0~sr#(m zm~V#GFSMI9iR8@L`M`YMaT+Z3XYJ(8uGq=Gg1<&@-Xagti+Jb!={ts{iJr&ZqSL}v z`M!|?vItNlMU--zE;%uuXO(>O@e)75C@Ax2oSo7NSUH>KXb5;0`Yn{21nTceh20Hu zSQ%dTL7^894xl*ai7_vkXr};S|4M+ki$tLcpoY)&r zG)tHjd|^RPFRxK0-#RKnWt1sb`K9&J5^i{Z0k|sbFV~+?R=9WRmbU=^f~0X^Fu#65 z_34HX$f*(4w+}nUhwT3p?w(=>Z>mN23jZa2{!JApmz$|;qTdp1YgaV~UMZYc%cj#`CKR}x^0&>n z$6sy1lBiniv4)s+LdR9{tC1rvLUu{h1`bxTE?aWEF6$vHsN(U&+l4u~r{IU&I;L;CeCRZzaBM~<WLUMP{N905GP+*e9 z4QzPeLmk$X-fw^U(Fs{l6^FRG0LeqHVeNkY>6Hx^!i8PWzpiHGfn|3B_0gX5MSXQh zwB5e?2=gn5c7|{if|$`ZGEI9jpn?|jbo#>UN37SuY5;Qv?u4=PcN&47PZrYk_rE2Q zc(GzpktzfgJn?Rxj5ft>#gDT^LIEzLX5`gI$jG0Af+cy2h!=D@yb>uBeaS*{t6SC8 zYIddUf-uQj$$~mwPlYt5i(zU&iGW6o9&+S((glo1xV4VYCm>3m@UyS{tfs*QQ=UVOE@j9O7#u zDtAEa+%Nq#g`N2SUsbd5Yqw|ipmKe(sx+^6!oPT73MRQ}s=hI9cBN%yK(-;^EnI<- zexNldIpj55IQk!cNAbi3Kg`%iz%AY4Z@0M0QbWwGM>aBJ>Trs%7HH zsUS8m;D3@Uxy-`!3^ zHHGwc=S22gXPmU)f|-E-`(KW4dMyzQrmfP^m*5hD%M!DAv+;dEyXeNDc>WVxDKBjCX++>*%8~h z_Hs=g+Lmp~e9)3s#OleB%hDaG9NK3KMy}>57nXkITv4KMs&^v(_Syd|$ z?-g@cd!Gi_=+i)>t6{Yl94piK*(lj>xb>)7QR~U~{$7RFxb-9*gBcBrDZoYiJ?e~a z4RJ_}V8gkuEb^(?p80fQiZ8&`b^Gl z*SfBYu?7opxo<=p2XJ2!X6obef-VV6MyFvX-+Im#^2Iz@*$`qJ=a!-wErf=;%C||> zaYBdkV?{7Zef>ZGlDI)rzf{;D!$ZiM>!J?u5YuXVdvRN1#3etI~uF`xu6>5v|1 z`uVfqoE;RCD3o)t9t-Itz&0uVFwMip1hC@YMvRc_N3u!}@R2C3jiaI1_KcJ&(lL34DV^0#)P zwwx5M z*-qn@#0mq5IboXU?SdbfJ>1_$$=qnbb>z!5o?BLb2$T%e&PsPd585JV;7IKS%r)FV zXf+d>xD%!IGIY*n6j*YOJv^A958EHBq{D?5l>pe=z#RO@=6hyeZB0T8r5SJhZ`tsQ zr#|dO6)@tFEkqa~+wWIps^7qX^ej;aPPZzbDOyjT8*Ag>E`CXQX# z3&onbd#?4qYbimOgA@Nqn8TIZQX<2qkfX-{_(s5i;SJgA!ph^|i7kkAZPaJUa##%G zIO(=-Hx)k_;nv+Za6b#2Zm z^;gt0l+5xiAMb1@5h)wdeW$Z*i^I&0ZK{ubMmP>`6riFLFiqlH5pv7P+~Ui^*Zk;9zQCZ0pl2oEOSZg9JGFX}3wzXWqmN zwpd+w=tOb6Df8p}XB}OEmi6@mfoPYUv+rDtOm2EisvIb)Nmvzn!;nVy!UrThT?NdJ z^aIlOXS)t@+KtVu6ZXS_LaZ?vQB$CTTcn}LMrTTHSUuU+eo*IIh3^G_|8w!G4=Rl( z7>61aa4C$4Ip3R_<1}g8Q!{Py;Pt>kn}tvfe?R-5$d--NqM0%1&+Vs!m51SB0U<5Q zq}u~l#f;stuJx$0aAmSw59iTnopnMPSFdxYy#I=bgjUG;RuTD`<Uk^Wh+pP!#Vdsr~OKgBvzJd;Y9~K z<(WZtsUD30wG*i(HXy6Tg~R*5mKy6>5@EL;J$GvCORnrT1PCrXSiOmpZ0OvK|7LQ4 zjO@`)pv3le(d+Mb)2Ub3s>75Uh#)UxIIV75CzrR|oD4X+L>fHyUArJXsQm9B^Oet6 zcLvTL)XzT;fxmOMV}gk<>=<+NGeM0tCB7{?Idw*8|FQ8S82F&lr)FX#vPoERee-3g zH7dLcne?f@dqG0W21;4=5Jsr0d0PG9jVC_OcZf%A-!$^_Pgll^JqHYmq5-&IiC~B{ z-kY~Xo1tNsrmh7PyGEI@o&p_83JJLwc@Wt3-9dteF1aNf<>$?QZk~9e07-=VZGIrR zuLR(`FBQawfk2>lSrj}(ECih2B7Kva(tRw7`z5n`jYx#);`%S;l0v({oZ|)A^Rp3f zDp;?nsR_-}UD*eD9ID>|LU$#c;L6j)A1&-k1q8GmvAtYmny-Nd9z$zE4dg}s`8UUG zd3AecZ!4G1wu|w|uWskg{_^b*jB;pC@#`I^{Ke#x^2Znhp{t;4i-D!<6X<*Dq%&vu z4GAnVQRR=hynaJdYmT3Y~!5`F2=ha&Cg z@#T%{cuoFlPqyCyIt|}{dCh1f+G-R7D$);Jy!uzw;8>anL50l0O6W8T%5Dh*Y+nOAm(g! zb9ruAac+Z9hTF0?_PXkMZa%X`ke=t1&2B>v{W&mp&Y9B#6I6jqCgqesp2rB z>0yKn#4rbz#%jaqys^tpj0>;q@U;EouR9JnYgNdeVgQnUef^xKa3Upt5d18qGYs`= zWU23nE)YllP5q|7dd5dAtco0c>`8-E$Jx1BBr*9m6Pq=CdF+5WEVR*xIPR5sVb`L( zch_}HXUneov_;q82H0NMGhi$!d58m@p!tXtb5c6}HOj+vWp6xF<5KHK6N&WJsKRZ? zIA3W?z#Ef}=JyhSx5|VDs&@R9M@*2-V%~cWjX(p>(i;_P)n0%~V3=bU3u9L~d{K8o zn|tN&T2T2)9fvVB?$l^KIDwekRC#DljuEYI*9@K1<-X9KYS_a+3Y6*Ir)7bH>erwn zz9ZpPG29P0ugrb?DjhAMI?*UV;7#uHlN=QmASCt9UI+6Y&59(x^PT7dr-e@`w*vy>}xG%wCxN5b8vO{R>39}P7-gs*csFZ z2D^X&!VY+T!CTI*o-H_jiz9>+{p{JZ1ROm8{mN4@Fma%*tqsi+mRMbYzq#qyM|=EU zZ+%^l6SX=5pR5PSXvBrK5#}GCA!x$~$F3ukhbJ+SK;IBx{;exldLdwZxV1kgUP533 zMS`FwJjZCd$sM&fOn~8y2E02!d_f93D@gQ65|H-Ccv2#W&kG%Xp+LO9iLwRC)8PdL z(-WvR!hI<=^2*2%Bt;aul>=mSutt=z^{DBW9&~%g-H<*#3gh1b_TMJCyaeKp^y3jf z7;uO#n6sNNA8q(H!W1O3G$()fQW5~Y05Oj)T`zDeInsq5&#PN=AK5@ETaY{D)SSOP z3|S#chl)4UAu+0t!`c8Gari5`G;knTVI3b0Q5vE*gOxM_h>$54>lCe-z4RCM%6+5TTzxIv2kXZ(q&WLov+iN| zti`XmmyZH+JXkAd#u~Ve+ld@|Vt1?r!SyOC_R#KS{CCf`hS_RsnTB(5^sG; zaGeVG)P;ONd~@?}l`w+3Zu%2kuLZ38S><+KSNPhZa9YPaAn?n_)ZLbYxv~7rq)7>@ zmXq0+-DihgYBY3Xnw`$K2|Ve}>NoPtEqO>TOnR@!Rii8?BIcRT>y^SsFzy?pHg(bC z&vw)Ebt-xz-DZWF4qw9i=7|yGDBd&)W=rpZyFFoJ{>hh2S+s4OL7h!XG+mPvdh?Ee zHJYP=nOHRb-V_oQ8vQ;ofPE=AHGAfwg)wiI8_z4Rv|m$hYn~XXooyD%PLxyG{KkrV z7>maeG?nx^4$mg)q#B2@-T`rRT5)OuB`VC>zu#YXjQWRVFs6QDp%60i$KF;4ZNH)e6Gm0S`wu$~b zb1UJd7y_Vn*)tiQ)kcDN>w|{0#6vVeBf>nws@-h*)oK**4B?!H|6OdQo*&N|JNM&a z_xK-4KveoWmz{4kg_*(SP`1G1N7z>-nAYNgc(;2@AF+`jh0V9%FGO{`bohzwQE)zZ@9p*bpAw`r#E9Uba&c{DFIMtSgq9jcSnY?nHSu%~ zOJ8kkDRU{U)~T- z3N8=Roa8jY%e@jX@{Fi-)RY^81b&adYVm$*J?h5Vs?aNOOhv2}AhE*N-epBQ!h$4t zD>v^Y{FKWAx;5#k$8%YKPpkDS{pPf#CaL;plddOn=K=$q*`H?b7~<$Tf0T9)1rx@NUOijE%tntvZ?CONydmi%x_Uj-sTGj!zCjoJ zPz^I^HcRaV!YqrQcRW97l3pWoyJluD*$dO^J0Aa$(Q~rFpG02=(EsoTIFuDi0pa46-1k{y!NAIhuz3Koko0~@X_2kC)g|}`|8C!|89EcFNluaPc|Sy zx!nwmX$Vc2cL|ef=vLNsYLRqQb{O>6*d{sed#-9PN+Gb*46`CH4Y|`KMk8JmLYj*7 zjH+pd>>TxLuf5+X@bgWx!QO6c_2p(QV3w`eZTzj;SYF`Me!;QR`Ff$HH`EqHXZ}@^ z#-VV*1_te2Srm2(7mX~MZ0NjH-~o9RWcxe}zlMc}7c5;#pn!;u^*%2EZ{4y`x3#*1 zcl>HA$KWl4cMFsk#57RMP(3^*4j6!euknq33e9Lq$W{70}=+70I7e7v`}H`eBsU5?po zetQElE1;nzFS9a2s=&E(xj&iym>RtzDUBQc@Esb_@ACtd(Un?1jRmi1?sSxW=GLl?mc2T>w#{fZ?g< zDGBz?Irmjx4I`i7jQ{Z2@8wcqb8}4Lu6*_HQBY)(acBHhyZ0mO2YvFX8|hLqjWZ{@ z>Atq7=H}aqxXD=_MrM#0ec4-*J6mr$UVxfbZXD47NyXh==Py&WU-5}0-H4d0q5{<#ior_OZF36+r0Jxpc^u^Y8k3*L1M#GoJc%n6m1i?c_>?bE2s+ISJTAA zKT)zwBoBEK?dLJ9{DgI%rec@{MYmPygFLqVo!?XoLI~p_)?1UGLn`^{Z2h z9?PY~$Kt^T=LKJX$Vb_?cI4~<@3BjIMGAJIu4)SU->lS6UY4JhdK?jO3r4JMJ6k>zA(6u)VgwGrS-X*!Eqtp#n27!=)Edvno4ko|&GVdYxouT9jyyTA(W1~0nflm*?rO09rAx=yJu~;az=_MCaH)$IZQY4~H~T?f`EcU!V+;7PRnS{})`Gn% z_Ay1H{I9RBm_Vo#E#ce#G{o9?LLhO;%)tQ!47M>|#kUPv42RDNpX8(C}xN zXFMi3Hz!x!Y~M*MHkmwlQ2JHeN)!>fF8CV?9Dj+d_V4#C{mP^Nw4;*B#Vh7vH{^6n zw;Q~61>@i}bW(JQda-0ePnUe*gb7nr^u@2?;DZj0>BZxtyNeR213~G()mF4>1(GGX zOu{&JS*NV5Xo@<=F>p84mTIQm*arR~dk=|!!A9yx5n@=+_>aFew~9sRURcYUm~ zi4?D;+gZKfSB&7)HiWI~I2-K{2jztBeZ+mG(ifeq5uANLI-MOv9 zq1j*;CqA%seNOYd0AcdSwzviUdWoChhZX`J70&LxLWm}<?7MZ4k!-e&2-XQH!x@Wp!M*vop9BIC+iCI9YDA2 zFXzX)`jTP^kq37JUz`pNyB^?sAs#X~3(DS=P(PrI=|Mn&XPd)S{e4@i1E9QWoN(=nE~la+5)BM)KILQOeJ@iIR1AX1sVF7g1+F;x}jdy6#1*+~)Sy zLE9~x+5t#FswLjYWRR4zzPHS?buPl5|i4<=NHI-!F`N?GQ=ASxcQAf$3S4 zuL?Mlyy$k!#9wtJNY>Zh>n)(P&ds-xesFL<-BMj+7OHG@eu(+pg3$TJ6CWFqIk&!+ zvAqjpI2eGb9^XDHee$58?c= zTKiwrrl$+mbLSMy>fB<~Z?O*Ogd4E^5RAXS+_h`pA)5G_A+dWXTo5M-p-gDfHCjcT zCTJt}zC0@d*@(VEDT2|$KpSoNyhF8fEHxT|Lw(S(OHq$%%V4{%WunL7SdE*XZwTjN zi@62o@`S3sbqCwYX_aDtaHr&obCewWf2P@n=YKT0;?KC0LT> z3~E8S=Zy`0Rdjy4%`ZV732^OQzO8Xd;VS@Ayz{c?%kz>pc;8-Px)d;Oi%pHH4x$T( zEREnG15+cK$1Gy<5{Pt>IcADMhe5UU@-6?MU(J3B+Mbf?K-D{~eaNS+W^N%{JUtoj z$Zq|+*F^UY?~(3D#(z~{UBwIfZl&J7+n1(O?0jgtm> z?PAX_`4a6OM$dIY&d!BwFRu1qe%_rrC3!Uw*I_Y~o?Z%?+Jlp8on- z{!{6(GE0j!QAyM_X4zQlL%fbFc71_*IS8oXNEM0?HjTqy*KIHDy{=1CK6%cvZ2IZc z4GFwKS^~SdYW~l?FoEQrvN;WBA9v2@{TgTcH{Zrgy)uCC_sSjj70Q}jcGP2Yyer1) z2x%LK-*?2-#@;Lb>c>5Wd*}r2ik=X;GRJ=u+E|0UHuyypL|(b*ACD|2CTL@y#WYeP z4w~kChQvkQ`eDFSBWsZ46mSpHqH~K=l|SVH^VE0t48;A|614r1m|#LVp$JjsgFf+r zGhSBZHi`4w>c!SebHl0p-@ktIENg0yI;$wj-|&zB z`6H*5c9l?k8kxn=PdX$=7JeqBQmZ(Tyl7PsLV&7Taw&x)w2 zE=UkkC)QKR{Kl*I{`AKu|6+3@70J)^-M?k1$-mFanz|2Rm+>b7+}#F8djuovG;@U* zNS@q9xd)m2ueR#E2SFRNy^Q^pDz0Y;x!+Cc>n3{Nq+|2p<0`MxrSx=?r;~|8La{n3 z2qr$H$1Tm2axv(d0A^08DO<(Chs~X@H-qaW+}aZc7=N*NFe_V8c9t*)d2|JV5`9^_ zAsMzgr@t3?UKh}LO(2!~pb;u88mQ_`A2&;(yt>5w-@m&mGL#-L#4|v3!Xqvz4ZUwl z8b_IqTbBq+Rv6MDmQ^lKR(>L@V7Ckic!Zu4ffL6g^Uj)p9fbFD;mT(?{uwe-2h#9;P@i|{N_u6L(DM|^bl)kIhhD!U6PFX_?;*Rh zg!DY+&_8t`{p;|^UqfaeCO7-3FwYN$G129s?BvcvuRn2n91=h?K@lo*B^`YjGwPmm zM_nLqTCLm^6+g!~B{P~aFuP|US@)xDsLbS2#h~|nL3wqTNq*ew3Q}Y&ol~_wT>N#H zX1$n-;6GIpjUFN;?W#^kN~>#a+ey{-SPf~VnnD3Pe<#120Co?C*XRqzA3y>$hOX;a zd39n=GHa}=FAAgxMV!IrI;H(4M=tz>Ga>SKK4;JSp< zYTT2BM8u#Kh+cz(eEehkMKg4wuDu5#Q9mvBV$SS%5?O?0Z|yH4xfT z_3m_yN+Id%`0Cl=671v1gQgdZvcoK(5`qp}{ucs=agQhe`eZD2mB%}f zH_~HP+g(8mFi8AmhL9LYb2io5e9)ud)Ml(7!#a2xU4{MjPEP$Sf!L%UwY#1n;xX$5 z5tgp=8UB%{;dhZG?*RHezOas(0Ekm-^{^|FOyA@;{%X8a^`PB2O^OFF0*tBqnWf_7 z!DL9+(kZH(wMOugUk>+V`LRJ#x*C1?9C={3?6FgoupdBd zo;;I<YHM`}4OGU-nC2~iS+wa{$Y2@3Wrm@w2e-xN&KtGknda#$g22Wdq zPlek}xAC3a!YTBdfo_rXCPCQCC=Q;IKvI8Y7Hm{=8nxm97&h8x;!cMDZVe)?9?UH6 zS1v{e$vPQ+Gj&IXO);iYDAoS?R~TM%L~38V*A6HA-COVf>?K;8%W_uYp=^XIMJ}(h7KH)f6&RNl-is zw{FVVX6%iuQQuldF##*dN8GI?*&;+}1hTfO&er{??a+QjBdFwpl%A7i)>WY%yS9=x zcU+LKn&*eZK@Iw<4CHlLwS&h&@cFXtKYm3B)=79Pqgrn$s?))Rn8KoZN{qPcWIo6J{nw=1OQy~y9^;x|s*oWI;RX-sTs*#?4TI)?=QoaaLSY z4fs48KKi>D6^DBwS|IyF*>@ASj>jx%S#snmSrg0X(}mWtiy zfyhe;I@M2vu!m{BtlV}wjr+~RH(sU@-viC=y=Vik0m@f~%Ypv_+#Fo|Cu=@zRybfZ zqU7MTjB|Yn40ekH2}5fWPqR}f$QBC zFy1PKC85fGjl$9jx}KrhPh>3_GrjYCvK=NMbBGcbPK%(>06yn)=oc9~3dv|2IBvCKipCZqT#t(Pc$r5L%GFM7Pn``Ua{`WP z1|=ekSxm+vO@Y{L9^)KpJD-ic|K~w6%d%TwmBd#%t|?BaP7=z`3}-jZ)b*oO^#I*% zQu7;CcKvm<9UJ!}wM%4H?TRjQP+oe>&D^vmNjysQhXo>sRJ%s7>tU#JE zfHLsa#I}uur0EU$e;xM53`zMMywf{7zo+o5>8NGSn6bklOxj^8*Tsx37%>-6G?GfMVmbouZC4sieXssRc8!HKO)(j}>K3a`=o}Y=6E4CYo8?NATy&0(Ano)%a5AUCA80;JYngh$Xa>eh>0A^(l&eEV375IRFB;WeowJklRos0eG=Jpta$N9R& zEwb!qhnH`$zL;pGrw;!a%~-d%Yjf3Vl5Fgf->;~sK2w+{{6qAISkGZ^VYcF#C;Kwe zKgI$r&(k21$>dvS=tEpZ`1$QC6DW10S?c}V(-eFuT@KUYa2&+jzD&OQo$KOAa0a>PSPi`wv;_ewUaGb&FhyxcF(QE9xfysGdX5wtI8(%0@&V4zJp!HJ_4AOsQVu;rTmsK=^m* zf1b|E9x(fCBn%;`6h-QcBM!+=j`xJB+?{K^JS%@!<%FWULio*o2YXxj2i=&p&zDX> zzPSbETK^z_R|`UaHfcCT#&oGDE6SNHNS|bo_MTS#Iowp+D;&*kd7gla_s?AZWXyWs zy@i^Mxt%k}2-Z_hs``$UMzG$AsveJ4|JF|4zjuz^Z7q zZ+gMxJqOa_i3(}O!j{5MNl+R6a0AIcdc`B1;Ue1a2T|q?mGy9cW2dK={sTLCKGYRC zY4}aZy*!NdkaVBXdR%c;riYHlwTW)npGUzZg$UWi(S2k!ZTq|4hJC-|<9|K>-`Gnq z`1F5mQ}9Wm65=S6Vr4yvXy^sUckQluK06;5=2%9$8CgL-!}KrwJS>TAbB!cK{ z>U?n+VR2R)`;#88iSn;Q8;Mk^QIX z*ncfV`xWQ{p?%!*-68ET+YY`aVG6Um3WBS?b}gV5NB;^>(fbYbe%k|%&YUf~Iuif? zSU_wK_*uUTd#inK%(1Tg{~cNWeFmKM1ZmcAqI^)QD6nSO`^xKCNl?C%JYvVuhfHn# zG(M}1i8F`mT!>D6{hGb@U%lWR)&JL~DfZH(SOatq%LZ|BXjYiY>uW>}SIPkiK;XH| z-mt{~$9fIXDHMR2eQnHOKmzRZz+EP!{WBvF+C%TPEDJ>pdtZC)s%>{4ln)FAP)H@P zGpptie@N#fh3!U(k$+Dv^G~#d5Aw6Sn|r_~&3a4?(0bQizIH8rlS(-d-#GYLWb_}O zt%(0W=fwUj6K`L0r0oNTeciqrSuuJx@)P!K`Jmx!)77b7w#_hd*re7>Jy|Vx5GEPW zYK9V30R^j;&|_>dVmEujA?y(vfbdQCrm*O(#b?i26hby6jy8KQ?!y1|h5r$B8F86^ zoq~*UhTrU7O6n-G_r0W{ZpyEhL9X|F{?|&9vnhnahW~0ocfkB;kCYoOZ>E*?@KQ%D?NI&FTNyl+NM& zhSlx?1uB{0=;`F$@KcIbGuB1~kfo^a0tBEAAhy`qOqvM+RBCmKcaI4=Rgq3FocdKT zDB3oqNJb>@T5s)~liaN61du9rXWzO*AD@O44gza#I$4Galv(_4s1bxk|xl(c&&r?&@f)BB~jPt5_6mu#K4rzTwU)9fEYN*)_ZD_^@- zqZi2-W6y}_;-A?KA{FPHKHPf6o&~_wK8}G_!Sy21eT4rlGbarj3p2u0xL;++eY4ys z;Co%BGFjcAuToq1hU9frJ9?98IeLZh|FQSx?@<2j|9C|q#Z1bQeQ6}JmTm0DR+gka z`pqXBaAE+VHBc8SwohQF^ICajCGj7puvoFF!)^Euh;#0zu)(LeE);*am;bx zhjCrc=XE{L;fY zBg!K0uezExgwK^A61ITjSOXj#y5C)bHOj{J8;Z$G(&)&i%d_4U_p<%(Lt|J+`k#9c zkvQmBzx(FBE_^(GTDE7Y(Dr5oEnX!jJKG~V32d{q{xm~)H z+qB*y+jb36;Ymt^EdF?Rec@ZSnoEZ=v=7`rbY9`n4@o3NR&c_GdDRMG_rFf7A?Pk( zt^o5q#aVe%Cv^1Dd^8$aKK1Cm&b=N_e_and4gMrDTZec4@sUP{Q#BO$o4ijCTJ@TO z<%b+p54p8<=V-LTTl^=1WhT0QM3W};HT*{(-vPGse}KZN<9}b_zTyA%i2r&Fbxwb; z{9ix*=Ue~!50AC}{_20YkbnJmcb>oA>jVGSME`Z=?7y9x!JlLJzi#Hw??yBK|0T8V z#=(&*Ggb``HXk`)m~UWN+Xo0`^4CAyntS!td*%M|T>uucf}G;SM;ZaXqt<_{60ki( zz?)@ad;C8>OHiLUIL?c-KWBH+Y<)8KU5p;Qm-bW)HC;M{z z^N+t>-$U1`yKjfOD-hfdhcuOq)D3xdJ2-dP9G4BC>JW7u>w;3(gwK7)dJoL-=>4dB z^26u5DobyyXiapZeq9Vig8&}tE=Q%6PBU_+UQ`F(w@w?9{?98Gz)3gd2`dMao()%Z z58o=hChA9-c`wARl1-kips3#+{r$5%EMlbG`{Hs)k2Fn3-@*0b)AyBgRWnb6$Wy$f zxAPya|1Qb7G1AjSTdN&!pzPvVudY2{!Bn*jL5qCA)x9QrfR3R*C-pxT3;el}cdS>M zHr_xw)ToKaB5JZD8(r*{*V*>7ujXjzfBdcCpPAvnEVG7pAccoK`pWxjk;0RV)W@LSc+3x>>mJT==T6Z*yevo3meTA6t&~y3q5K z%1t~K#xzfRf3sF)N;lL62Y~uNG5n&u8|1n;bnx_}6ZtwGuUIpggF>? zu*P+*!*-O<^U27KXAlB8PxCK;B zlfWOeHG^hKo7QI{*4=kJ*U9K-3!h&~hmLmOO01s|6Ulax`HxT!x=gYLZEw&XP})Hk5kVRADX75nC*xq8(#dzm%y^yRW%M>Umf)?aP;Gx zH|NVwOYop(MQw!=Qjn(zZ%Cziy25cKxP|)Fn)f52Pl4R`qwkNCzfTOsZx>lbR(O|) z?fBWb4_aP&0>_2!8490#Qf9L4uO2pUDKDoL+7bDgXH8PNJx)70a^d}JE2ZHNV-bFz zFG|Pi{_A>{HtTd40oJ8E^FzZ3sykR0{7GrzdRg<19(E@dI~=<|`UWlbl@5@LCJK|jZ_=YAls zH-|AXT1E@u0H#@vZ55NQ@^d1%SjMxz8 zoHi_dFFrjc)m0VsAaaWMYoQJ3#aan*Ft*2W<9w%|FN@Az|6u=dC`ce~S+Y_eFhjay zMZvFm3lP7?+34boa=dBcCG5a-{cFG#S_8B>lPuO8vZk6ASY{E`FCrHv@#( zk!hn`$4Xy-s-PLy0p!`pZ9Y=!{5Ix1hGxN1&x<(pvZ4O<1)b*mYj3UKUghCtg7_W>wCMak_OUg?cH4l~_pvv+PmgmR)#iV8IP@rVPQT%- zZ`BBn`TY%jAsF!zpvTq+&=u=54NDxtKLPVe%vPObn;XQ z+FMtD6PxmnrXn{Mp_e8XqYS@5fK2&6T!e}3x7?f3*s2koGUqr4-2L;TZY9bc{s1>p z9r9aELL?Ofcc=@hT@yd2fd0)n#W{qjEbJ>IRTO3yHiIvNKg(|cl}Q;kSZ%+2WC)dK zTH2)J{T_E~jJd8W&1dp;$OKvImvD`5KtR(5P88~Bk82fb@t#i}GKBm8`WUL&HFj_m zpeTtiBnyqgh}pOq%2C4yA?2+*;a7I=jc%s7rKKE-GdWn)(c`bb_=h0*O?)2!DshU6 zxco8n`t?)T*e5=EHR^fcha{mkvx7>J-)pMdTBOZd&kOW=Yw%0-VR(j- zUu%qhkN*C2Y4K&DBNuWXF|SuK67lNxlmPA4M%!VLGMnAZ*s*n=UjawL0!yQ)j=c#E zGek{)wq>&BMvm=C?N1J04!Re@k&B~R@^W(E|JupMrEiBBBKx;K){b8!sLD4z7=l>u zO3<#DDxX!JcWSHN_qar@2Ee@lCII-d{8>QUgI5d-zx%&de$| zaK=M33RxFh*XrPl&{@h0T$SFfJ2qzC_rGspn>eHzSds{*$nWxec%z3Fc3$kaJAQ}H z()%UagD$j;#wEoP#X{(zOTwE*W#ycbn||SjUrw3pC{f}%E|^n=e-$O($!&h{O7yI@KY-d_{Isp@z{stm z-O~%*jmguqbEOtKg`jM(>T2kzru?XW#bP1%}v`wWW;%;BBy+6AW=kq4g5L79| zTMtLhe9M-|S{kYCtqCCc->&hku+iHj0BS5U|LMjGZk(!e)XCTLaYX+nYt`Lfd;WeG zZp=igRFEr^ zKfKw{*l5&}5JZ@+gZEhbm%nN1$sPCY$H({*Xi%tv#L zJMGi8I-gXBdPIp5r;$3v5P;iy0OZ*f8baeN2y=090rm5WXgUM(@c$X=n%I`6Ysq2v zBstBm`;K57*2gpFCL1OZp*arUbhnNkcBRb?o|<{?4Ln7XnNz*%leO^Ow({6dfRp>_ zH39mq0nZ9v*lGL`Fs`1-a70r@A~4D}jezTnFZ?2Z06OO+s6LD>C5_$+UGz3K#Z;xa zIbF1vx7_++SWnif!D@!xB1Y=1f2=iziYV!Mx3)K)!~{`iH{s)RXlldyV%4JUZAonJ zqplvzp6)g;N_ymzDHQU17U4$Dq>nwodGLU~)cPtN@t_TR8PE?7%DzKSt&1L?G(DnsT6xuL)w{}sQHnz)49015;80S2HD{Qv<)d(eNKw$e;s zW1!{?A!4@;T;oN02qm|*^sG#}ZH~j`g?E>kuX8zC zd)BTr-kWs&7m^wnCaW z4M3Hmyl>%{OE@6KEvNi}*d|ZLXiY1eJ$tnA!Cd-(hk~!Zc{4^Iz5wFRqfj^#eN=qq z?VrG|N!R->(_d=>y@Ao)dX`B2y~N-94A9cRu$j@uuGI}PKZJ|ed420)=1?sdtl_oa zl1(IyJ7C;q%4vuG+D}7Zeed~i^zKmpL-eytkMN!`588rg5zA9@!i;6+J%rA5KE21? zXW*?J6S)lpqKlm30=>UrH8zI>IHzgXs3X%>w?ZpLp&KY=lMb63Btzp;7a$n*YtJoT z!PUPg{Uz;0{1g$OIg{6RoRcAPUI}o)99MPp5zKPGvHBlCg5*|Wlg}gH3jgeqB;OxI z24c*I3R(YcbW71%vRG*`Mw1`G2X=$t{-Gs^s>rUAHTEsQI`{If=;E5NMgpY~+$%7x z#9g3ZhKyjbJmlYdY%dzp&W?V9W~Q!!kUdUycgmV|JCQ-DxG)~j%E*Jo6BwPSAhNr> zc8CsoFEg%j=&IF_-cF_{DgVYhj4mES3rkv#y|geL%ej*oSNP*y60kq_rsuEbG~5fk z;okRgW8d#%;G7DXK0P*ZB>?k7O;|dp7vn`MOQc{c;4UWhIvaUt!a#1 zGKcZv5f$+Zu`<(b2=nUD!hcGdPTv1m#o)i}NQLT5E*!(zu=iOq2k@()m!S(o!n@-Q zOVN?l5=yig|Dl%=pyx_s{Fxgx{X4M0e3+_)0^(r6D}a>x;0Yc z=5^f5=k>_=>u=xJXrF@AJz2uGfGSn-#WD%@fDzo_y2-`{ZA6^i6DQPD5ItX@O%7vP zZ(<{V0?zD`7r#{FIoDEAz5A;l#^`|zRyO4Q!t{005ollN;<($N*rtT-yAM%Sf%8Vq zT`Le_zq+)4iQ`VJU%5}q@()Q;6JVUirbGnx`dK<7O`%cf>vcXh-?0Y`3Dn`;*P@^u zAfX3gmI4Ofl$^c&38G*QLiu*Yp204SHT=@soZ^EbS#ZsQys;U?qy7E4Iuah=t}T~) zO72^8b%_=?QTofV{S@0eU_EYJwXOdIHSh%CisE{G#rc^_&J&}X z-0PzgE{3P4nM*P;2D$;Zpqml8dj%&1@()8X8vf&+ksOWgN1xYnp$Nb z^XAakWuSAfAiqu!|G+^!03lD}=RG$-~@i0Ux@|Fth z%uo;JJDZ^omVDGayN}j{N*qy@taz|_=J(nt_fO@!HpAQ_cA@+yzX$wOEc(RAcyFj_ zY)OUEs&&`z0CjiSN-&qr$fX^j5S`_mqR)T{K?5d)S{FhH!bMu|WLH0!%&qKiu1d?y ztIy2LykTpb_6K|kSs3<+i;Md!qjNn=1|;5Ep>r>d6Yw$CS=~zo|D=1oXZ~PK z5lj1Xw1aA_^3VUmi0Xq69|C<8m^#aaSL(=+eRv7y*WsU|?Pjg1D)RDa9lGx8_4glZ zWt##D9mB@rlSRCY&6-`yEa4q9)Du8LBU^Ycs!sZB$fqhcp$aAtR0E%!y8x7P#5)br z&;ppLkJn`zMR5x|%gow>W#sV?wWXm@34}&Of4ax?0nG+{C~ep+qqw^|W?KIJl>JaY zg=fIMgf$ws=^}6r+6E36K|3Qdno2G0{Af&#R-Tn#q;MsaK7U&uh`7Th4W(d;<1kV3*g*U#!X~YUTu03J| z7rBi;Ki(Zyb`keG@t-I7@fm>R?MWF-GefivoWn0yP}(MQhMpwq5L{}d3RD{EH=#T* zL7bc;V!1j`_xidYL+NDZL(1r;BlLhC{*uY&Wj#!aS*AXKp{(YwWlS>bpgxSYeReJx za)Wx+fHci8Qv+f;%|uATL;@Lzh5D42dWhyc=~=Ba*u19}BeE@#xX>Q*q2b!sv+(1! zgbxnQ$m@|@UoG|_DA<5J4yU>sw_b`Kl)U}0O#cgcvok|>N%PEL+=?%o2M(m`lLenP z4~IBbgduFB|6Xy!7yAQC1wdrPYqSMO(zhC!TibkX554?vgdfFmLQ5XY9VMb{5uMd) z6nA#y@WLm7_=Lj#p}45I`$OP$i=scg-D?2t1|kDX`^&|?c||Z8ugVjXBHQ%Zs0N3> z>{scS!uB5HDl~Zh3cSSFJTK`os^%wwZf9Is#b7~H(I_prOE%Ir+)KB0~ehySH zMOWnLoB%*{yZ~DwRW`OSw;SL(r?sBvEqSU=UU0~YKJiyPSCy1|vSXuH3%>)Pl5#d4 zeE(1-cg2CPpT`%lJbu3Mit_3I@b0|-kN$A|z0071i%PG?gLw{zE>z%R<8SPEF8H<< zcMEX`#7K+++!IjEnDHbVO|`sp8N%)p!jX3E%>Dd(PJ{|N?eg$PpJ$@l4^Fe7T3qsf zS#3{_Fs~^)9U%BVLNfZb_z$^=q5)+EFHMP+ehDd2D-Oac>*s;DSFXmtwOFF~D&%yY zPNX(^(XP1nFiHN`VT56w#veXM^ym3W095u-!17_o3~7gr>P$Z?DkxL~;*-yUfO@M* zZITofI)!Lf9jw2O0%E*4gd7Lohrf-{diusij30HUqPqIr=Dh4bM8bC?0OGidx*f%g z9jgngvMhI%^*rf%B8ER^)641e2P;GP*7BV~#{QdqutI_}3U{M?(|3|X5R@4@1s#;_G%KsU~ialwds#|9SO)w(G27vQcE z(P?4S*48!|`NX8~A4bDyA>l6??IB@hK9M@=bveUfYNW=m3TRRsN)Sih325G_X#zxl z*SqA)M%x&aj*;5|O($ThEEZyi@&alqg|?Qp*W9n~_aG3H#UaFsf*dvWpR@cgP!K0Eo^!$Wd`K zW#(xI-z7#`*J+HyrBOg^sD;UJ(~|$HCR&~&%E+xw=d1sFzt=h!1IE)fr+B8A4`ic) zc#rWwD8Dk%)LaEZjMcx76Z^~cf5@G%EwYx``zkj?Yy45eeO?-yzyK)!CET{Z=F>>v zhFmnXNt&{eNbpB+m)1|wK+%_ZaDuF82wz3R>q^AC7{baoIFmlvXAN);+zWIQO9WM_ zf)5uDpC~X?{U)9h_)vNf`B|{dOOVc?_waEeudcYc-rV=Rdy0)ke7i_h$L3+jpU)rH ze_}p4yS7^M#=2QLdD~HY2ftmD6f-xKma@utr@i1b#yBeRAqXX9O~TL$$r9 z&@8psZ*cfR*HYxe@IS+^L~<~XyP<^{p%(Mqky!CXn~lodk0v@)HREdi;NlN+(Hp_o zSAe3>(O?Uz?lWa#|AG~^F+bx_4aCYItj<(({N+FN>OY*G47NlTct92g7<(WJ3;}%} zvgirzd8Tml7{Lb>4B0^p490Oo_JIeu4<9yylkva^E#lL&w->WO-`Ewg4MfReGwc0= ze?-cJRAs6q^`0M*9=AYzEHCf7R^}+`Tk~LUpj;DW!!gRcJ-M>#!it*idh#+A(1$&P zK5F^4;d34jP)UK8152AGb4H#)_O?l4dQU~pce@JhLh{{jp*AQX6(^?2e7p)-E!UDy z{g4|Gu%emoX!s4ge|@IJTU46BBYN1wD&D3IU~or(u|KU@Pc4NF{ebvdvir)ESkg_< z+w{5Oi(Q{Wm&Pw~3@a(RYk@xrZhJ#n&(tTi48}4jRDc9&t;8ir1*{H=DLgl`sb zAYyino+pEe=t>pE<*y`r>R2ijH2sirEZry8d0rdjGT-)(GUSQj7S{6Fa3U}nAT~fcT80v7OCW)pW^DjnkjCKo?{ek9K!o+tpx(I3t(d3o zqgRT=*B8e|kbB!Cj*4$(eEAP%dh-JSI*JE52viw8{so8!x1#hmD3d~bdRw!FAnbB= z07TP_8s)L%C`{SKLd<0IO0cMFwdLY zL!%ffnT+Q(7oMKCCKwS2?qQ23__$&LP7`CKf z`xBi$f1Ht|NTd*{%zD@2ve@{q*L({Xhd_CAY&rqtsgv%d6PxH%TYoa?-LBEzheUYK zw(QdT#-PfH46obTQX$703oGFQod>S z>2g~zGwqorUe%&^QDI>6ndoiHTe5K*4@HkUlHvo@6>i`~{RoV_<8V1*+Y={?a)<1E8V8i?Bj#Q`14@Fn zz{DMc^O0Wrz}hi~X?%ybI`Nm*n-tYRFv>}(sPO4Stg*1p@V z*G6sE0_L>|x+wiZ;@jS0Xmb_t!g9E<_iI_BLfNqo4pzxjXdO`VVD_gXkF!64->N=J z7TE-7cFk%okAe@?TDwU$YEQ#O*x&sT!v51|A|WOF^$0N5QqVU;p|?rGFShT($R#Bf zuQ%&&;Jh6PYPBDZrv1Ua;DhsCMCH zAB;aLd}ss`NA{I++n2=lT0gjGkHIqeUg$TM!GuaFES(`3f~JwNhmOW9FyzP*xoR`c z!ib*&PeG}nhxr92ZP4F{x$F31Vp?Oxj`+&WSaDlS{vM0^ayL_duH*3-PETx-kCU3r zm0UR5!GU;qM7_mqHk3hcW=$)_>JEzjqF>(<+CiJ#NCro3ghmO_XuCIs-Uu&Usg~_! zT7%eF;%!z{-Ul9nz3ob_v zHMUZCITvza|2C#E_w@f&Fe+&#!Dzh8$koY)nMk-D$nLxcA!qlw$DDKM&$(xL>wYzT z^i$B+=BS?j51bFPk=vtfLB5t-uHlxCsGR#cm0g9hnVv?%E<%zRuDe@286!-whDUo0 z@jVsGZ_-CjQKG=^36ieruBV{`AeC9ORSpPk$gsy6C~QB(M_0tEsozV1)67kk4NiOQ z-_OUKYW;Afo$bTQ*YkK_#HAwfa7D6ccQQb}9dWrXN*)ACJVYT(Fg#6Wfe2th~E zMqzM|Ut&{r4k!b50hkF4?~eFleS3Jn3y6_BR>Ih3avCWVXzwI5cqL`5VX|H6*3eUl z&5{y>KdU!Mdpo3rgFNVUu|p(hGX?U=nA3g*3{0|+kr4+a-Rg2yvF z;t6{NtUNF7=@x%9mGk3Nf|BGB2Fqae)!1q7z}I|w2~m>il@m!mM1J9*rED)OSCWh- zkC4kFTuNUOy>sRCVwp%%GN+B6!t(MG%8RdwcW}3PXCWX~`6~(ih{&gJrr8lX+hO0q zaYK29R@HlGf3}Hca@eR_u0b-P9bLmz+jRCK2jw+tCbou3^Btbc)dh_$NSE^N9}Lln(u`1qPeM=;N^h!-J&}^D-f}G zr0I}_Yz9NYV#~jnthJP@@W@%DB=QD(OU%hG>|(?_N~$E5qJi>A6T;0bwX&AQ#GH!$ zG?F-AcP&rOiJevYcS3Tn?C+w>&#)^%wT|Vo)R*rl%bKfZI$o7&l$Y7F9?y1@Dh@dZKAq)mhALdh2rP$oj7#kUV;v%Ck5?U-VwlcAm{`bfLzD%IV>d-2 z;k5}A^fR^HjLvu&@TAqg2;gdWh@~AL@Z>gFRp} zhUT!>rjL_;rBphr0c?~G>?>2(*MG*O5_)^zHwaUMI61Y%*DenBUrLBl&4ZJ zrB1X};87c@jIRXf0d;a}(s!)hbgKsV4n1I4?4>-;AiCmVIZ{Lge+0U52bdDd6qdDM zhTdeT5DpX65a{}jk;5(rx2tl5#7&ZuMh>T?EW1jRr9)JH%J^M5A*bPi7oe8_g5T`m z*zmUP{VtkhY+m(h4pVnN*2Y!1E3b=M&JlA&N*XUuJ_k&XK0*-Dw zTgWZx#tJmsNV>RNazY^w?XC&EP`qvD8FfF-^b=5F?I2Z}3lc5fqb?s0t5h0AkBfkc2(R9p4O>W3KU=LPv)Ig%VXxY@;OytELDv*)=to+nDV4yJXHJ__O4OMGq(&E{=$;67u zQTx__*+Dcg>mhItXeWv>tV7s-BK zh-OVl7Db$6r1*l8HCZ)Y(lsa->ISQ9c6Muz@U_eLLNbyrMbX;8B<7LbR?Yb4 zHopT5aoV>VJy1}uEGv)dgzzmz>m;$h^R@3A=8pRC7W2kjv_jsaXoq5?8Ts64cimwY zHN=4~r3q4p+Ja7`VK)x8??b2niR^|W?Sf8MS6w?^ay+i!OBrW5xa7Pz9GIP z5hZQM;zl3zI3fCd8y)(waw1YVcBkj`Uo{RNnmF;eYb`f7LG< zVGPJ&;XnlR(6U^O>x;WYMsPOmxGXPBj$p-Z!Y&Ju0)IJuJ-7{iuZ$1`Nyy&kg%^3r z)q?801E~UYfw;3uOL9>qR@r@POF~t~>E-Bt%L3_Y)lZAuY&zl)wd`+RwSs(fl&m;R ziQR9{QN_hu9MWo$G?vOKWLX;F$axfLd8WY)jL~s`>{;ZLXZAMR{+_M2eG{8H6#ifGU;^TEmrPHEk#9YwEPOw?X(v|fIJ-s;R zC9(hxPZoe(AzYqE`4Iy#Wn{i0ixD^+LS;y9$@^Vs0;z9W)u>Dx!6o&OORmist8_J3 zBtcP0U{2z-q_4AEnZ^#rtwtzf-~@^uh!D*$$!qGC3^4({BwfBnIP7b1yKW!2%*n&; zFf#Uz#n!H9=vL%qosu3}OnW@1ioeD(x%MsYDmc+>cvTdE4Vfmb=GDQiHDSD%ATL`s ziieJ9*TZ~Q`F2f#t~b03u)U(Q6$f1;_pG1hGF2X|?Ieqw59W^Y^2Y-DO3Qzug6s=h00IxO06w3rxOO>D6y zp}r4ppP^jRrWEt#OX#-Ci2rU92drI+%4r;D{FZ8u)bslMj_u?q@u7S>oh5J##oGX7 zGkgSE59a}pIFa^qD3ZJGY^_cb9OvvcP}JL(@P$L)yha7+NO~haz5crzT%cw4HYmS> z%PBCb$d_@@Vydk!<-LwVbMiBm>bQ#Ow$zC&&R%{(5P!}m-s@8x_Egqb4%>(+C!6Jak83ONOF=J8r`pRS@v030?zmyukF?O^cirdkJeRLJ%u4{IODj z5`|q#lOm#CFkbzHd+M<&+;LR)11ZRMY(&y7-3tQv(_$uc_j4UO>T(Fd+iCSNB5FFI zeII^sovG;(c5N~t*%)ZeR2}$rlhC5s%cLY(GPs6co%wT&5s&UQDk(3j)THrnYn8u> zx^N&T1}J0LRtly#aO4*yJKFHq(s@z^tu^tNOwAF?lbdn8EI{@%O&Cr zvFNiIHUGe;_x^4kGS{)O(UH}G7Dk#0^0nKSDH4CQ%AC!BbLLupQ_Jp{65&Ig2lqOl zyHEs=*g+5huq=o_3*qIO+i8hcd-LssC?N!cGC5~_&$%+Q^CCb!=T&{yB(E{hS3d2E&nm3^WF|K1MimQ_R4lJbsRqku z6Wz&qTn}&>3(&07Y=PO2+j{hbdffF6?3aS>&A?sRkDxTWc6XMd^heE51eOOe zLj(a@obuJH5fU}h$`bT+4etF}pI;?&s@hY;jM(|w6>$dmI{N-ijf_FG^Y^^U394m? zwDTn;NF?|L9SUjR85zORrEcOhhpxt#U)*BK>m^uyX-)MjDW%QaZ+T>L%+ft5kLxbt zU_XZ>pRi=LGNapFcQX5Nd+llcU9*u&=Y5ibBbNeT{*d|3Sf52FtZ6PfX1D1s0W=&t z=8LxERAior5jEmpH>JxoCQ*wFb+vGU!UO8xpEwP(k6h+Q<-Z+1>FP!}zr|T&mVB?d zz)wzgCpld5et$a+Yg|Xzt8h~7mZ^~RDA?<-1)T$Zku+l8B>!ga_v`rJ=-Yy^nK!j3 z$j&G9r7(@^ilQVGCMc_swDucain;>56qgEKTx&4lc*uS`v4Ii{5v4Ow;gZYT7vkE& zhNM9kz;j?z^TE~u&txB_sNZu-4*eG-H?`gFV8n^QOl?;I^b$GsyXeDjsh;Z(1Z(Zj z5tJBCh?7vtFXG9OYv-{BTRPq<{4TXqkqEIsv>kZXjF_rT`L*j10K4)bO?)BR{HtJ> zU~+F7Pp`xl*y!e4BrGqI5f}A5`kszB{o()~LQGPh)-F(UliX3vc06;LAInBe2+Vze zD19M@lNWIc)R*KzVs7*V3FYJSUrXZ-MP7%S5X!vGXy)3S?m7u!<14Js`9fytyS-3= zYxg=DBi{SMeK>HyX|_4}Y_dzy(l4QB-o}VS`Yt*JVw_n_>E8Xci*UrZ=65}C^M*kV z`ptGnUmNj`z0TI(38a-=H(E(P`B~>KHCTqXy`~P1E#>CESgsTqOBKX*)je!_ExnT$ z@_9EiZ+Su=wZX@mt{$`N7cwr%`u_Y%5h+b?w?i=DPPf#V;n>QKZIAloBuX+^r~vKLV-EnDwP!vXGHx&^^&d;yNT0>Q30Cg<&lKH5+x zXQ=Gd)NQraQ6MHd+R>?vED(TI`3$!31aB2*3|-2VSptoG(tR&0U7InNwp!(=(scwncc(Cz^tz2`X9rvdRo55HJ#X; zlXFBsHcS%Z0wo;Pznl|cS+C}jq!~S%I%X~oJ{hrXt!k4LB^|S=(u!JJVoBYS_)gwH zz1mX_#5jPoKx$;~S5g~b@6+51?J)J7efcfdI8&>I3LgH-0m`_Hva^qe?iFD;AK`@4 zD({KDJODmo8_yLe@QN$dDm%JlVNQc7ZX~eqsBEie0v?tvV+M!Wd!*;Cp5|7+-_q8) zjk$-H-j&b}7X>YYMdr3S<5t;dpJUj&Y;GyepIVri65EvatsKk=k>ss~x^b_~)aQTy z9L(Bhn7A7g1sSwT*P#cZ4CYhe6AhMgWkqeS`?h#DSi(e31tNAgMbKLORo+|%`o2X} zaUMf$OWlk_wo}9o0?_ZEzC}m#hNUohBa=<|eyX1J=3BM+{$H~inI}jvcf*N$jM30v z#{Aq&{4xESkwhhMKGNtm-_JH{{xRo^%P#gWM>B!*Xf_9(x~NTzR^f^&iXuZD#k*o2 z!=j3toin;v2h$Z@PwRCOPu3m{^wwv|guM&fw=a611XfSYvW zlXIsv)@Q;fvYO5j^Jp8{ZSdTux{KSJJT6@}x{Unn<1;vOCSBtM( zsNjx~MqYT%iDH6!4Tp2b1d+6+KzyS~dhxwP3<-YF#sA zEkAr6+3nDDVQ%KL`f`acBgz)-HwZ!PavL9IEco<4dkH0ZM=S%HPbXeE2?-@tD231i zawt?ZbJRF_J=DV&ro|LaZ(h1#BR1#wqw3=s)sQRKb*Q>eEBi_Wxli*h)iy>A@at^c zR5o=W2bOE?KJ4WqH7E6KtArFswYLR@c%g9YMW>+MIS+gud5;Hh5?xS%c`7~&6_7Tq*<&1Mnad3=PmIO?} zQ;KAA!I$7Jh379gvU$lGl9ZhbOExU`#EU$P-zGjvybMtX<0Us5$3SMF;;e5ED#m0z zxstNFr*JkR()^EiNvv)Xdmwj3PiUkq&oa#30Obk9377KWvNot&kTh>7WGB1NGyMek zTU!uJh1A5G7<<|Ti}oj694`ISjmpeSx@XO<81gt>f8FI6_C8%h%FDj|r8Or6t%f%I zK9YLNi|TrL^pz;68eE$-Wz!J~`kiHI>g%xq;123A!e?H(mq3)nn#<+qmC_2AOa&mI zu-C$?V~#{|!-Twy?~+&z?|!d40styY;-{eS=GQ`k!XVlt=E_*GnSd? zBzG*1TNQmdkdce*At?+yYl!Hjd}-scW*uR0z3dBZmi3$Q4oW?EgqW-@i{rkZnZ0YO z$9XYOI9zB}-m&bvi=J zp)umFl?fYhLY&IzR;KDs&-O`=_NT-{fd_3LYpdyj`Bl=})D^WaEBYp-QdW=zHHxhI zDP1x_!2AlVD>{#%>q~%N)Rzz+c$GIhg3eW_f1y~^N$e0zEK_kD;ALv8eQu7dNZ1J5 zk}`sNxRti4j~3=Jw1zQ(*@0ahS7E9H=Iee~6Jji~lQG3v*BSmwN-s5q6+vOF-gq3{ z#L&Q6__J+Q=A$NYa|GO~P*$!}g>Vc1Bb4JJeS_^e5xIGW6BPWM8Z48Gd?gY_CH-$$f?#^4U|t z{YBHP5Av8*I#Bis1W~CD^rKO{KthVV1{87EgF8|kLK&kb_daBB9ME1$;{3qbf`}T) z*V^@6fio3@&9^**vjG5j2wtNWzio|(s%@(U7cls+XOVeM*fIqyUL$>};Cn`#cpyV0 zhY>&sT24x$H2!ww2nitBFmlhKjfWVKAEar6@#C1$jy;Ri4b_mRcKq!Z_z?`tiRb^u zgpIWS?lb&i%E+}i4{5lIpMONPys)j&n~gF~mUMheeYX}a@F|_|>D=eG zAORay$&7dCEK76g(`pa9r?mK9Syo>5g1mKjusrsh^CagtloviDubuOtWta4=twldb zo5QGkx=>(dXVeEbyJdqS37t*8X9z@zXpG351a8x4|Dv521PZ=hmuk@BuCmY-iI)to z1GUO%22W#?sAwb0x9QJDV#B4_FKHmb7t__>$z>BvzDq?Cgvv7bCKFVpFoB2nZGd^~ zor`rjJAxriU^S4L{iVs*Uif}R&V7vP#8}ZgX1?Z;V8ik}zQCT;$l`@N9K+~UkORXB z@;3Y4j5hQq-K|zB?6DFu#yL^K=*IB#HlAv^-B`}Wg;^47&25PyOz#*^eiHn{w_)!Zt{L@ZfjGGlMrQSh)z$XCc0Fj)nQibSKuMdgh!s zQ*QFLya#?yFoZ-W2KzR!ks5xHwH9Ev`>xOS zCg&%cB(GR+aUe~cW&(zvRQBc1HfQQ)WEB6qb_%Zie_S}1EhIJjyGP~YL@mheck&@s zgK>%8kRRD-Pkc_IrGTAk#YF&*$1B;T^SB<1Igrsu`(k>-G}oo_`EQ(0n!16Y;i{3d z>4b;tXmbiTST6FHV$2b~ZB$`#2Xj~0zZfH8hV9s(5 zsr`O2;+L&ikGb5fkGv3LrKGxPzyu&s4yEtW$V_>ob7X0E>gy%{siJ+fxW`5f+eYk! zLnu86_6rc>bNl>U0K$Aj=xuSFL-% zY7~;mmWHaYsLI~X%qu6JY>^!AEte}0n`>JMY&J2MLg?^1nWOwb#$a4|LR7q@H6kg* zu+i*JNdO?Gf_1dlk+o^I#&X7@pH!(SkgJl)GKIu#YPj?;@Px8U zA*?5KqPTd+Zcz%_YKt=)Jx25I5NB}H@O4I5M)+3R$dvJO+AQoGK2%P_Rq4PPHVfGE=~v*dvmHc?LwAvl10x8g*hQ-`RQ4?O1n=9m(8KL}RFR7` zyD4FmyWz)gJ*lS0Z)el4L_1z4y9Tit7EXJnUSjX4KR@NFgV&n6*{NyOpnPq@2bmes z7}8Jh+&wg4|-?ZBtI6AIQLMC{S$c4j=SRzNx$m=%pPivr_`=q+)~oREJ*CF$DL!I!X19i zhgVyOU4pbIM!>uTdbv`Y=-_3k-lEUR-LvwXq>YQfRmzyhBcH9#2?q}FGISA~Q4qf_ zfRmc@TK509UI9M#Z>}XG;lVHBdC%_T^3brDGd3GX-EqSD|fP6{a7npR$hwo=C zU`mk7Lr2@#p+=f~b4774V*PgD&cMF#g}9V}Hl?K0D!xutKg;F{ASl8fGbHVL>~^R2 zy`u~sXgKhmo5RM;^|SRiEb=wc>{v)M=UdK43NnWG)@{^ew&lp^5lr4o#XH4PA+EI_ zYSS`mijRVilxwWPX3g7EGjJxtl&DO;muH3IYhL=Ig}!$*M_GxmE1)=K3*-)nL}r=r z6{Vccu~f6*r#$BOOFdcv#yu{T_DeEoJpiVqz|jlftBpsJudjOXQzaU=4)86vgfJmH znDEIZB07As8QBdsaFY37={;8s0b3mhXKc&FfUm>0Y6&+Eo2a zfhftPIvXaCo2=**k2Uq^$a~B0VyK4C^+M>Q%WX)DdU#Kn(w%z)Aqr$e2``3aBWWdo zW+hr@hS;)D-RVN}zS6GAyMIZXaAo`nFrY9m4Peg-e}MZO;Hlyxjze%P&pQC7$5`C% z?+q(4vNv_dWE|Ax>?HE#lbiUK#}kzLu4fnk)b&}w2dg8pa0WSmY?x5&616ADiPg0& zx7HpSrf+UCoKhjN+Z{v*TJ-q;JeHQv(_{fAIiB-`WtN)OSD4S$0uAqU^^8=@15)Em zMj4uc{$ZJZEi>wB-F32-%+dL-qSJhdXidv@>B|-Gj`!zu2B3~YB*7e-ig}3-Xb(Xx zDDQMdf8`1*ILv2&G4|S0M{P2Zc@3~N%Uk|B116;=nT*beA5%54c+kwOQLwwUH7yMA1kuT=|k@ntkO)cHOumuG~ zilQJ=VgvNhM5PLeh>D1G73m@(A|OqA5(Gh11kRzVK%&w_rH2logMbjEHwh3S(i2KZ zg?EB_&Ux;Ae)oRf_uYT!-ek|LHM3Xwt~GnF84Df@m3zgE%#5^#mxvWFDQ9XXQ$MT; zOXZx9t{zY@7pCtFkz5b{tUWaM- zB*WZV7qLy%E!)RySi!1SGmdVdA9;M|;M0Zg;Crtundw79gz%OChDy_(y<|nh*!9mB zG$X`F&NG(=V_fIFQzJhMlaG{&Oy^RSp>tjr%%}cCiIoSmVOX@Ko{rChW21Jh!{F7^C* z!n6j4=AAv;7Pn9(DOxx6Gi*h&R8|k!3gcLz`xll`bGOxknj0TpupE*I@*;}`2I4PI zT^GNhqd{c-rNKH&(CB`@ee!v%`jJrZvBnoIccOTLqvE28>@D38Uq7L94biU6kb7oA zw&bgmcO*I2$h@Q33q%!MmM~eQnF}|;-&(sN8S`Cy1V8yt-KalL{QZWvnHqt7MLRcSYRtjU``MYG z+__NS!RN}ciBFihnmpSo>&LE5MB$d^MO8rwn<_e6zGA9`7ODCK!Wy>#C8`FS3ti95#I92)OZ z_d=gtzAOe?7O6ko(#ZUU=rr&|mmm(tSPC!CCJheD83I_jq-4kx%DQ@Xti zNf;;VM6F#6&A|VuTqkgZE5b1*O*~(aLU0?3JwwuL+D(`5pP0V_v-jXb?15$OCj02_ zJ!`UXXVifCr2ME$SG7ROQ|$_4-;KEoU3W2ZWP!-zg6mH@ue>@iOYjL!3$F<`YCf_D z&V$xInJ%v(eYv^*=#BhykDz+hHBYY7IM&G8tzQltC`sa3r;6J@?e8`mA1mjXA)j%A z3CSInH!)*A#y%E^sJ<29=hv0~?VcG#`$r&k_rwVn*qO(J!!8m#KjB|squV7x6^3?{ z3PCt^qkU2pjPKGzqPw~7#03{G3yLR)+JDKMW9o~|{NT~@Lteidv~YN5SJQWhg5cVF zVHS!Uiz!_o@XZI~jdr$PA1vH*IC?z{f%x)Fu$nBm$qDT zN&bhlkbBA+Y{8=)9h64k>5lQg9-j6p$FJk;u?R);k@FjOWxcgjb>R_>e}K0pcddT^ z7E5vbbazhMD0(~o9j~X*9|E;O8~buLg2VG&V00lFv$_e z1Cer7NM*m!{WI)5^{c!Z_%7r_qt}?vdDT06`N+?7Eu>}yf?gW=ffPmyEA0kzA}wtz6qXS+zE*ckH6^JHg1~+9qlDE?$wmE=OSn?+~Jr zwmeg*W?n}2sO>R4QY!Dq2g=*}!pBbCvNG`WEi>Kg;9z2!8dC;paVEUPDs z+htpCrFRpw9z_U6Ty%f7e*PA^Jgvm!Rh+b62lo5C*U;lr9NV&F5YNLo24EZ|ToZZX zD~Che2Lx5m8{Ud6CQxsQs`9 zuDDBhP+};?fg-Y+v75Mue#BMVJ)U*BCTji;WN+y^blTV%;$gZ#ttN$MYcdmVA6%S@ z@GDIAI@)|_SgXQ=X^LOTVyrxI4*2saqDC#5lWV|WrWTrRvvQj57;rCQ-VTz+DgUE6 zZXDi3t|xy^8gk)Wolw0kE+)g#-j%PP7lUoi*Q>fG+7Zqn;LM2g6ib{_p>JVH?l@US zaqnmrW;~^*HbW5c*sFnsHY+_zgxAX_h^UpZ?ZHl?7I(#05XI)vI_DS#h@J@LyOvzC zh0)!jjbiv@8PY}A<>nn4^YQCiICZDf)|{6asO1BUXXP0|!I>qG)r1Z}@{r5|af~GE z5liQr-&i;$l{K9-jqd4*UREEjE0bk)JPP>QZ2Bm1y|yUWONF6o?MTvvu5#1&yS91M zlj}Viv@J15DVVL^TDX88!Y+Crg!^wx42{3Ed*x~eF^Tfgo3En@`1GByP zAS+aHOMgKNA8Psf7W!f0o(}Pz$KElX$R?EibIP6J1B}A<7m+`RR62B3xkLSWV4^KPZ?zTH!1Hj&t1Wd()0~o`3ZCoJ#HaZ6_g$6%3RF271yW0 z8QYIhsO1mQ=CFVHCKJo?p^NyD`%5SGQ&P#(e223GZZ5)FuJi-^=rqGx!E1NAUL!tx zv|Knl$z^dHcSm#Ug_$qp5Z+bEyuT#6-td~kSg{(bhpF|7FF6&s{4z`KPAB!C&u!hR zKl8b`#zTxaY~UVsOLg|}7X1{i`}wYz9nE~RhwsI#7a6$!h147C-o`JSMpv{m6i7OE zXjoT#(9A`J^(XQo;|J1sEMkjrGcWSEp|z9F(CHls%L;l%oD|;Xlax zy1}MHlB*Z}&Q+(&sYy+9&dF?(7lXwePPg?MeV6UeNU=Y*s@GxvDV$;#WQ5MyqONsq z=9~z9A7h6t=J0H|o^4kRijpdBTJviC>PANHSHnTH*jjRIx3;9%M~ytKlL^IkjIR6| zpYAo)l5{VD*@<99racaO?EOWN;%H|pI#U_jm3a8!@ZR^Qx7xK9K#TVT%|T|q_$RHm zP%^!AcD>J_WaJjOE!ppc4H&!S_-Bs@&AHJ8AGV}ymuk`9n5o9)vzc*?>MUhzfY$vXJ;M5M5 zNJ-wP#`_KrltkL)tv=LBtAJUtAZuc~JZ2|Oq~a9cccy`*QS^|iXrRQI@vSGD-f=|a zJ669V41KQu6C-VZjlPd~GU_yivraWWARls~UX?=>EH&pi&3Q%H^(hV3dWC%VXQ>%{ z_S&ee=TG*^CsyiemgSsYxS%7hM(MVk{vXu7;W2T()=LVP=kPABw06l~y*vR49Nl}Q zv(-l+^w^J5rKr!^0rz^ z7mkb4DSwszf%jI@fFlQHKq%xGHADZn;kn{;{H=QA9oU}tEr@(%)Yff8jP@VeZ-Pr| z#s>Lagwe738(XDrYNN1?$20}UvS52gjvi=G-x7TJ;q?5%HR&0{VMSvD>n`FmaVY^fcUsGgogua zDn8w>5vpBnZ`q~ZmDyxT5v+kwR-}t1Y41|hggv5KhQl0&o*z2D>t=>B@#7Y1>zXt7 z9E`5OIMZ+HMp4;n7R5{7$~xTp%xBDRyz&#YdB^M_>Sx;1W%G`|h+0zHVzc+j6H2>SQYNzYxV7Hlb z0#A`C6yDWwzdZU5A`d9r6VG}T#Zw&bWv|{pHOXc0))8;ljD5Sn5$E^kg*Dz)pr+yj z8TLtsl$y5?w+)G)ynJ?@N<4=rmX51t^j$GUlke+MVk7_au8lNfVNUOb3H`b1mt_Mxcx zTeegE84t5lH{7AO1H0GLZ9Q$I2NEHK7496#7b&kFThVOQ#3R_IYr?X{QcQ24|% zj(lW~;_OxpKbe#6swPybY~NONs)ZlM6j>h181|stl`?0 zxVIaEcZf{03KjR?&K&&T()gGt4~~jwexyoGzo?U2o72PvBd-=KzHJtryjrLD^>e8B zA^w|}&|VxfFVhUQR#xuvi3v^+u))x4B(YA3tIWM|ca9{Kgpt4-p8>p=w)N>HWdsPo zFQa9PrT+Y?*z)Cq`f#gff&Ij54MzAgCuW^u@RvK#*tGQd3x@uEN$ZiVX{SM1lSxt- zZpHulYGR-6&?yo05Fcg$_-VKPJw2&V%vsl%8S7>3lq^UdMeqCv;jQsKQ*$z{&Pfl0 z>3P^5ELG1f_`bMF_2rHC8E)SZ+ZZeUcZXDaCFAQA5CUWMLu)Mw#qZi<MVQ&2QbAez6;ZF4R_2dU%?~#U`HBbc2QI~#p@migw9SVsaQ*?vuatL? zg6eX*WL{J?My^lYL}cjfqF@5z1OK{{f?u&5=3p5K%dWnm^cfWPnuWBkpX1I(KT87 LMmLicFRy#) z5^jfd#~qf8*7UwYn3X@#8}6`j$knW1N10qe(@Le9f#%JQHY3QWkz>Vs8FWDNrREJ; zQ@8Ahe;s;>exwml5y3%a4?ikeSVN~P?O?3e)z^DN$J>=hea0h&M!CMvi-pWo7F|A< zta8L(J6qw@%8j>)GGL#2iyav$l|B0Y&7w*wA@gzr7zIC*O> zF+s^4`khj+-Zw_;tQ#nFW@dF|ez|j5PqW-l_@5&YI#KHdE)&09$hb{P1|S&unoDZ0 zhEATI+px<(-eLMU?+vJ9t2EX>_9ZSHtg=Z#%^d?|!b?iIFoMUWAEBUlRCTbymK4uQ z@%p4wO0PbRmDzh5??Rmw;|s4JAMn4Wak1ugzGc8!)E2ei6^^CpdB3?lNPt-J>7Q!N zR_)D$5W$JFwz9JF5hRcKSC9G2Pshe^eC3YH)HUl`;nwTZ5)eDPZ35mDw}IGujtF^o z@SF}#basH;^5yl(U_FvX75V$igJaG+n1-k^wGGbeqvP(XVtmaVTM4R7*J%Tatk*lf z2h4Sqx?L7X1Ho~@zn*zSTaLo-paGx&lWl|H#CB~siJNZMI8ZCRliCBbJ!@7m#c9wFJM_Ghs!-zWZc6wHDN^gVp|&G|zaLN`5}VtW$gL}=V8y@bN67&Y|*(NtW^D&pN-^xq>B@_|Bbfd>e>P-aoI@dXVKK ze(tInQU^8^Z%kGbtqJG;ksBZ}*4jSL#{73ZTz~mEB}v(pPcK292ZFN}z*-aq)M=r6 zP(T+yew(gS{)Epv3@`B0y-8}B$n%*mhj>?e(mb94W(`PUEwJ$pM>Vwn)T;w$su>Fv z^?C|b`=kv9Rji3enWTih=m%(65SFu3oGnd&n?nau<*m!lMDd7plmDEI5=$yC{nxRQ zXnn*FOP^J;8-FKh_jSUse#))-I$h^*L<0PvduVv?R8fHH?v& zVG*zM-0$eU!f&hxdMl-ej`eavko!8Szqh*g(Go*T@F{z+;y<`I?RKTn?9TG z3E%l^R>dkLKuHXt<{)@ZN{t$S#3`)Tzg(WjtAho495`DzEz-SjN8*zR`3fmn1~Y=x zEG|YG*nek%88-g?G>*S|gz?6G<7l)Yb&8JSl;F-pA?#LO5z=5;gyEs5T=>3X2T&}8EJr!BRj(EH^_S3yRR|r(OjjJ9XpE=_6$S~c>1&YK zf6EPX0BC*1Q^1cc*3;o8=8h9{&A0AoG)$Z#Qbew122{Im)JHcK8;BLl+(7PsY+z~* zxLBtI78Yy3No`f}o*sh^pbN_oVlt;s_?TLq4~e3_VM5$^b+zHt;aRpJVjDy68tEaC zOvPe{ktS;P;(7y1xTc7~BzLEFb%!*2+v z%}NO|7J~-r1WOGjyQTeTF1{ax)%zH$uZt}E%8&@hJO+Od)qbFoHhhl<6l1N_*Zef5 z##A|4p_IA`XRW@@^B|}kWl_A{;(bTa`}s5`N=b#9Yy7!Yw5UNo-LC3DC&P7JTrtx0 z<(6~duJ=n|FNU5`%uf#~#C^-NBPdT6b37i%kz=^Bp7yiC8hmgRVSg9$GdHO7Cq%lW zAJ$azF+H9Zk`#`#!l6*|$aO6JKk*q1LvD9#SoSx*yFKp}K+R$Et7q>yM=BYy7 z5}h{HcG@p#wc+FGW_T{Ntj$5Bg4<2Eb;>Xa)S_*?B@;XcN*u}00M1OQ?j0kfC1h)Hji@Nh`rAAPaPE&bK^$T6fDQMZdxK47=%X=0s{%Bb=m zfryt#-1hO*vSs@!ZWG+AvkDr_hVG;2R7%C%kUPVAEzF+>I(Rq8SS7>@!;p?h2#vSR z-nH2Na(@P%b*|Lq8ru0msgee5j>oPgv#nS|mYnI?QzfJ!{S?%Kc1)#X#TUvK-?}#z zsWhuwdCcpN2ZuxWA<+~$#Ni!9<_hY?XdgzVEmK)zW8&i;1+X#T_pV{OqPyBq6DS-S z){G8Nw$l7?bQnh@*&ggAEGr~7-wNDIR|{_DpMr}m1pdf(r@iM(wDqztb){>L$A9$+ z%7^>I-_$pzU(dKtvf2<sGni;oiBYIUK>Sral)m zuvnK%(LBZ^C?G|Vft`C42a-8;%xwxDnwL&fD(t@@m6yn~2!7r`ad*D~uZ{Y8awbY} zG+>Q&FDZoBv&eKZ3;n3$t-A6`h_Z9;ig^LqHYd>^aSv}2b-C6nIw26jGh?+VPJ2Ww{K>xFsiI-t;#L{Xc2 z8CxG~2!q{^sHn4RMu|~GXy4rKEVothM{V3{_NkSQShtE^#PXQo{hVE*z2wU^tdy22 z77t?@757vR5yyZ6z3=-S{<*h@Jef{gx#SqOt0? zx>{1_do-x>8*(;ft%BfU!t~olM zip}Z!U{ihmpyR48F20(GXOfgKyvS>K-Y1BXm*o13t^qS|F#I7DW}JomQw6 z!+(+bp}U2O@sb}scx|ftTcsxQn|?|2Ifd0%J~XBAjzGw6evPFIrL$Ph`YL|tLA=!^ z6A}}REb|>;yg6DgLbMS>6zWGu*NC<-#67PvU8v%|cnIt|EaxFeQ=cL4q?QTW?t$!7 zU7KjDCm6yYqi`aN^xaG036DaB`kWW(ukkO-nPF-W9G=eYKNRaTY`^%po;wpQI2`_l zWSV-af5gvu_tg!;dj!`Fv}Ye}T%Bsn$Pe;6%MuQh5EzawsfkS+Q{&$~r+B}*yQA#n zh&l^J`hHp=R9c6@crv`OTSvn8z7>&{Xtf_|O);PAw60xl)PwT+jx3)m^|-cP+l<}9N;h@dXB0}bjb%EX;(wEgzxBU8 zc{V=jISCvB3G8gLTNFFE)a15T`5q8=`!Rpqc0T^@MZ`($w}{<)G+>}+b=@34ENbhk zW3zp3FVHJedF41Q9an1zdT$V`-M6?>4N3YszQB;+lc~0-D#E65JnWLLHYG7_g9>d+ zsY}jvA;t*WqIZEcwI#;AR?;jCJ@0?+rB7qgc*LYk@gPNLdKYThZHW_tIJNVUp=ePr zC=WYPJ+TK0fihCdQk6?pvHI=O-84@1^MPGZ9Wmc4yQPR|Z{q1SU_5Je zF1{vMu(FppR>VcL&?|NYJa*iE5c!> zeDOUhGugYg55H5OlvvBN?FUEKGAr0{rpH^hwDkLuJum99q+UWOsPNXH(g~w|kIjSA zmcIJmb{$YHztJhhyz$=@GCcLk2@wsPgI>)FgF2-U+&w9SeeM>x5m2pN-|jmi4g!eD z!hUTIoIfF{CU4iq$0r-T(C~(!06}4?bAiJt9Pw*x;ODXXNq@*U8Vt7>^quUwz7c7J zPI^!wgB)d48Y7?#X4?mt-hFlsnd7@c)eN1g>opoc48n7K78H`RJ}6&JY06uCtJADf z)NT0{BdxVzio+g~W!@p*Ky!I(ti5ZyEx-hJRT-vja1Nz{d*eTQBf}GXRyT=tX)-VY7$fZ$J0)C z@NB3L_0s zXj$PBg7|4!fg5b0HM1L)3zo=*ZIGHfSNsLnLyb4o9Ih>PcT^o0^R9DJ{0w`rsp(BM zXKl|?Y4?J<4g4#%Qsi7)v5b2Q%+f_=w{_(&)UkT%U=Y;58qXH}uQZJfL7WBzwgazn ze0qtnk00e$+m)_I=(d}V^!g9+`ZY6ALRZVi)Wo*Vx8-6LLAW-~1}M8#M$|P&Hb{i= zQBDYN|Hwm2)ro;v-{3J?tNRrzd~hoLn+X1SP74T@3$f8q6vO~ZU{Y!RQ2aXE{Sa;Z z9L`qK>BDx@nP^v*Eu&)>D%Yy$4=dkg@zi42p=7Ud2fNjN=MS}B(D3!8k8V+=-SyaT zG4YP7A4~4dn*5=RONj7|6ts73dcG^20|!CQWtF=5MKJ#|4(e3a8;hNH?qilIz}LPl z)`RjbA~yc>?d~i&--ZqJGDV&L5!(?`cRco3D{PB4)9_pPN2&gihp!M*c|U_c*CI)`^5WbRQ2WFLBp4Q$7dC3uHFIyu9Pm;>JTi4n1;x+@C}x-gYOMe@|)$xqj(s@RJ3rzxz=l4GfEus zsD{=I!VNcN*U>Ituyggd%tl#>#dnvf;mHPw4H%AG|Uu^U4F_amC1G(k+ z&w>}ys$fh%o6@PK;W`upBEhH195~?8j~$yZ*~|YcG55cWaiv*aSVQ`a?zymAFLsRf zAn0X@bugCaqWcPWX7d}by&(&2iMhM`MQWGBB80-baaF22EpjVpl{5RtMBl=y59(N3VKSc)L?=hk@=owRywM840F=yb2N$_QRwpwcrhf#N2@e zt(|k7#%sr7#Wi3HJ%tnXd|C26Ti2^ZAffh`T=m}2u*Q}fb`5U4sOXh&2`>}B_4NKa zm84=&eXIJW1Y-+g9t2tj9-W73`Ud{1-&f3HxI~ur8{-efIBd% zbxBAaadVA-)lpLymqfb3=%D%8eA)m|TMUI5k0;HS;2+o%{*QCCUqhPoxfm90By|crq339P%~M%G=s6XlgDG-q>2&0p@6TL?XQ0 zTl4H$zbMiZWj7;;=gKp?je4Z;0)q6{O`W0SEh663xd;XOKynZnf*deG0s}u6jo^d+ zp#kCvyM++L+kx3PY1dzpYv&0ORB( z8@P^dg@m{S}I(&!21WdzKZdwYm;%=i}ufm~cZ|DFpQw`tE$x?%L&%;5Xysqr=a zspT4L6Ef`d&r`R8>h-<~@>?F^M4?yDYnMUMPL1`{k$?HOzg5xR(LQk(YGYY~5qdQG ztWvTcoo!%?IkgBo(6Uil54VkhwAk2R^JR*c4Q|n%J3n~t77cm$-D6yZ0V&oe!Rwpc z>Z;fhI>S+T0%zIBJk5xa$>kwhJv3+HR*c+Gf(`V5`5{tGs$ql58(OrI7R$-ZN;s!- zVXD-iZ@GhUI+c|pph1+o2Mz1690Z9gE7h?)`}(p}Z|k7h75BE4z}uGkqP>h6(FeSX zSQY14(XX*0AfMqT5jTdX5ThtxC;LYZ$XCrg8MwjG8?CY4LjsPVQ3ZUr7yM{LGSz3j z7aJa0AFU~g+?hg|?4DZ#+B|~e=u;@4n{l6A39RQWxfZx_UKSA1+0d~`j=(9cD8)Lk0jMF zlvGB9qSV(hR39Y=L-l3mfiK8$pyk=!iy*oQH1_-PRhlqd_xXAJ$GVg?=z)#I^*J7e zSLn_u+d;?VKo8Yrrd}PLSG0JB;P@TL)~rV@5ZtlZaS-Wl|JaOz-zc3JN~#a{nz}S> zn23;gP&(h?-bCxJ16#J0S)aUL|0HsJnrQQ4!vPI4D8q0c>~*d~f`4kxCxehc4dS-e z;vT9pC9J?Y1LP`|EJ)P&QZu}a*+1WeEL~5SwuI$IYp|Gl;}~Pc$e9l?w~UV?h>iO3 zog_DDrDvtzZsimop3JLCUbx-tdW6ruCjED0yW7*_`Zd0gqsUsPw5@ z5OM3>q36n#V=*6Z9|{=S-`_Mp862A42IVQi7daqwqTymeGlTh>Toy2pXvvRG(VQ_` zDXc_L(dxEJs$Q5NDw=MbT2M(gupPIZ`FeM`Wc-^*VO?)O{@|LzyibEiW735qi4+qaS2H_m8s3ZqPAv~r>UL!2(cIK0 zscy=_&d}L*$6{_SvfBK<15$o1mZKaI++7x19s6{wsT?X=U#;?_-xz!wMFid5J!3*# zyW7DRktiP#EFWr^p+3oU-Jyt zueakZaF2=7D31p>S2J<5)vVDM15ezr%SiZv5W|#NsIDv4og8!0nUZW=O}jHoSwA_X zOgWA2bPOaZIa74V8w>as#ZI;N#OIoOaGv9DdFf+;r|*W9fL$`+)UP8yWcbCp@B^c3 zj)((6C(c!T?f0v>-J!F58#3Xwmd{@ZyEI$_8$t(?d)X0sgJH4YBjVvuC}UuF%3B8!Z0e9lFxmx5p^q zklP6E&u-jrKJ?~D<|C?E^2NPPk@$uh)#1FOLo)m*Pz>2Wzx}2HzqFCGMCh3FHlkGIdKYm6+|1hk8T#-zkajKGyUJu(C>L^HV6l+iaxsIL3m#n$z64VYFh7Lh8Mst zZv<78K5V_lgF4#t%J$%Yu=N`o&AEW3U{K^zod{NnLd%wgpRfa!Ww2W5{R%{uQM39< z%8lyME+eI1t#Y*H??cbn66n+u({ee{K@qbAomB+!wxqcE*I@lOE8=j58`U&6%=YR?7LVGk?Va^tZ2dm2BE&X$9F3 z>Td=+G*S;F&KC0Zu959Lw5ccLENNIPym#Cj=EaorNGI}201v@kySFf{A5bWlZ;fY@ zN^*hSH~S-%b};imxJ@SC|*)}aEgif zA@|^Yc~5Da!nE9Rm1~;L$Sdy$Pna#|(hiy}rTV0wdY$XBYx4g#CpnAiS9~lkEh{}Z z4wI~!YMx|2-HC~Uq|e~-n1YOWudMwmy)4WAx3b2{qtP1LMW=z^Tz$@9`wvZYsrd(nOW)GJzbMo zIsIKuBE2#{FP$t3K6P4w73oY9Fa^!s@V#^DsT zH#2}eXfO$41W(zIqgUioCO;-j-J6s>+v{|HS{5xiEM&mC>GJ__2(CA|pR)i1HP1>n z4f@dS9Y1%UJq$LjavQx&hikb%FEg|1x|~JWoUtdQ>V9{6iRy1kk_M~(Kl`#M*JwAM zxR$#)txm5~!%R~W@&S9$>zP?OB~2$_3jx;v+ij+LDh99Z#Q+&_RoK_tD6=^pBJ#RK);UarH)4$iU(o;@Z zD6VZ;siW)Hd3R-FR^&`Pnk!Q|{pac8L{COsiE5ivzj@nh3kNv^*DL-JGJ_Av*2=9# z0~pT)n`I%#!Q=h6?X}d}KaNhsx4B;HHyJ%2SvV&fIQg3rQ`IUWa>wME+<*wPoW`aQ zPk(Gf(Tv>Y>1(&~hnfl25P4j`>W<{b*?^yo0Fj%OGoK2SHYu8l_R8QJv-D56c_ipR zGp$;(P9BO|0+hd73FrPg#kiM$QjaiAG8vta6`kzKNGoyq1dlZB=}QMUtR!r**K zAEAHj1c+AFFpkXkg9)m=&RB&^+Tqe06&|YX zCGH>k*}rSMXZrwC0ts<0`snfewX013{Y)c+WrSM0?{tOg#daStgJo;b7gHel-5tv= z%j7ABCy;_`xwONojjF}KE-tn?VZ&sm<$7G2$IsyFvTWFm`D=@YA^|`Apu6|%et!)H zP_(AY8yFXggOMeIX@zn0KlgzDYzwpxG(2&oSQ3ni&8tvOT8U9%HoHBLBQ|9=pX^Bi zY_jL#*I%|N#^ot(-VH#+k2R|s^YO@kr#P4%Zq*fbMq)|ErOCVKrQ;qpmc zo+p3>{b}=j2F=SonEX!N@jtKG|Ng7te3vVfgePriyji_C4hzUb;hKvn(gq|L~`UBYZkqVqCRU^xJw8(h?q6ZEV`hT-#OPS%5|DqujzCr{~Fv9XNMhESX{8?vxfu6g{5QN?Ti_Wsfvuylv^wMjOdNu}R_DE-u- z8`vh^@l4DofK9;d@2(lN-NQB&(X%}Kv1$F2IY8CkTIp%-Z!PLeZ*sCJi?IR7=qEhN z#EitVbtvAn!bGIKUp34V!F-SH)rmAc&c-j6h#t096b+fv4)*sOwivWw*aF_Wf`H$& z`iMSg)*Veq?JaIPab=PXo!EcMq$N1W*b=0M4V}tl@QK}Em|$dV+%=JzRHJ_B5KJSt9Cpe9{P2&wa=z%tOGzBIb%7Km+DeZZJ58D z^txk0>6gZ(WH=3W`O*$_UGgUU*08p+8^D>ldB}Oce6R0R76-GFF7hw(CqWir* z!XM=Ci2wy)+j=O#0b!onv=h@xK+c#K)PDLHD6qok)xbShyxBq>_RAu>c@)iWIwiJ4 z`Q@B^?4{%ny0AV1~iAbwoUP_J) zMlRX{FUh6|96?E`_}9BX%|62uQCyM++{#aOfom#*y51lF?*)3ZEQ?7=`1;#PvOBev zpVVW=9j_Pbi-u0bfj{J2Qz?>(H$MK$?FyL! ze@1OB;Xv1)v&UUYKR(z89^Oo*UM}0I@0v6LmTE~z<)QUMwMjktTc%-8^w28#t^aDj z!*E^JH-Yv2zfGQezxhZhQWgP$2&wFWC8Q)c0duI|5bcoZa3&Xer5K^s%ZA*5C@XG- zK)pvc1<@bxr2Q$@56^a;2dT;Q6CjxCiB2~=o(a9OX(A-6&a15&PyDv3GkUIG|KG-D zq9&J0ivfoE*VwFb3RPE4LC5KE-P}pJ2S3Vxc(x|4fQYpHn&%(Dof zi=m(P_NNJs=8te*Ot;1TRf(lBqamJh>vG46fcCt`xVJ-};+>03AG{ApV7~w9iEjUdvZ5bQ z2xi=h5K!q6fz>(oV_Gup0@%2}1SNSE2oKvdY=UK#70QQAG`lu_<)2C1GzL~#5B|dN1SNAth07v`S^ ziK<^O)_bxOrr4o{ZSfD)%q-*w6II)yfh$DBmXumK#v3#Cnk{8nwOm?#9#hp=)OXQ) zCJAJ3blf}$J?Ty6$R&8lOmSxWXuYT5l-gwayd70(ko}Y%hF$DB;>@) z48FHX%&wdPU?3rSuMV~;OW#b5O^3~C_SHx;Ynj9)5Z=n>WIzCoj-^!p-hS0y%XOu8 z+CfLTe&z{BS|{dOU)O9wLY9X`v9o#iYg=~H7mZmEJ=;GlWdOQyCZ{^=mkiwGvSTTh zD0s~+{Won(Hds$sDg%@wXNr=+$YUb^=1vcwN@XE{^fm|bwQ}aVqntBH_$d6O-3$Ma zbFFVe7WNv~@9bgiprt_mD&g_csDx{ysj-!y1I6X^Ru(Ml!?UGSd~M^2mER zX>MBCbDdVY_`1b=IlVaw{f)o|AdgrH-**5&8n!1VVzEAc#hQKXY{gx4O6gyJV7Rk; z53?AAg7%HxWNj}=1kZhk@Tn_wq5rk(Cl+Pm?Y%}aYxYz3-^Vi)y_9-d=kMh-C9?Gg z4bn%x9@b1YA0ZZlZ_8q^Vfe3ByhWa{yE*^a62yhKZiqQ-@l`%V3Ryd z#L6}f9`H9@71J;j&SKua-Zhyw=gEkdF>9)fpE8m!Y4I-S>-8q5pOLeiZFxKSFz~t~ z&A6)XZNl&PgD8stpNk?+@B#ByRBCO)DR@>ewo?Lw&1Sp2+wo}89ADf=q4~`2ctiGq|@_PvVRlqe+?$u zQ0|)%(i0+-JIf9g3wE7X!8$GMN33UM?Hm7R< zb>4K79qcsUufcegMJ`ENx9flyeMZOoHHa?N^{*5EwsE&f@YR{#Ve7PgAO(2Ba7zGf zX>vS50n62vAjhMZAj=>bh_Ak-;k-PuPpQaN1kAkrD?d&_N&2Y~Y z>}Ytn)h$A&Ze0LWE%`;) zTN=A>1Usz$g)*8;*#Y(@Zh5=u^M8^#<2K!UX<6++#nmQS1v~4{9`5fb^T`K2o=mz0 z$O+w>w3y#qOejKC>$Saxg#Sk{Q|)h~ov1${BCvOC9cH+q=IavC#2<+1za zkWUJx&$sJDiVkJ)$U&uUrv^v_jBD+efRAewg6-MoU1Tu>m^W>P!ub2$A+ON~`|C=E zFmDy&ny0#M)r5>FVnQa*nZWr5e0 zKOI!T=nGrGul8g#6}@Y0Hd3~}M>a0_f9QJicqreu4LB)MqEaD)N};dpS%xtxm6WAY zwk(w-S;iRK6ox1jMn%XnldPpic4L=ivd!4{Y=f~cGh-PB@1?%K-}AiB=Y9W~KACaP zeP8bTJdg7@&hxm;i`*T|3SIBVqjuCgvfkt62rm$qL-4P=0Y-R|uj2AJPA(Uahq*Ib zCz?mZQX1T|Hk!|1y_8;zxlm&GnX}911AZcD)ni?f| zhk$v1v%nAz?7$lXI@x?eG)raS2M;t>t$@9rZ4V?IR}(}O!u(x!w~;P@F1ys+aJtU@ z&rAB<9c+Eom+7F_kIzOe-PNjU)b-1YJ+ z?S`MEh)z3dkN7=l?TE0%DQG0VuJ!=<0Jl&*<)Q7Xr+mY)k=Jzhuyj{BeyHMyS30ncq?(a7nwsSMi#+QFo z!!;6X6BWSj#s57ePTn#5Jqi>`U`FF2qq%Km5L3Q1oOWWWXQb*s^ThfcL+IHDA5bY{ zk+hf6yYoNx|5WoxF#vNjhqKU(@$U{7_XYFgT&X7w6mSK-=7NHY$ej~bmdBdGJOZGe z{&!U>OUU;HX!I|Y465)RYar1n%B8X1mmP+g-jz)QG!zMFecL!45WmL0c;~#rL;$G& z;_iRzT!6hrf<>M@GcHRmP}nA|0n~ruJ7=9^5+`-9rxqBSZBtrH|!mI^55$E|BwrhKf?RILK;eL zgVE0De+z*>$DPiP{vR>}p!R>m-ZuZvVY(OR2n7)hu;PEp_Uo+eDhCb9v%eRx%?4cS z%lNAkzKPG+P3g!V)k(B8)dHycUj=YKjwUEwdgBY(Lh|${gXVaS6AkT(yUSG6U4R@z zJN}8JfzC--I+$yOckP;$o^-!^h}pzwUAEX3lkP84JB^8T3&z*)`j~;-YBjBK2xN!w z{^+UNrM8qD@4q6JtTtA>!*K!T`~C|^^7-vGt$C@kn|iXpAFkc>C1?EQtU$t#t{=Nm zRuepSo%nhc2o`|1y|#HBd=DTVL0Ji8ApkZ+)BpwbC&^oE@DA(vZj;-2Sn&6a@_D=s zKaBsfRNEZ$boS)6yS`>c$PKe?R%=Z5Z_a+X@NQe%1Jt?mAIM_Ce!SoVG|;{tGrw(S z7QmS=m~D&$aMPa>lOWzPHnt6l|0MaP54Re^dmTXH7T@NsWUU-T>!aOIiywo{PwNsp zb8CXP0Z|@7;=&Sj^_TA#7fWp`jSz8?_Lr2nbn!7DRo`mhN2wfx8I}KTrcVLtv+#9i z+qQV9y9E6gpnwy#`X^e41#p-AnB)j)1Za9GGAzKP6GllRq@~14M*o!tus*eVvI%Pl z^Y9cA5{M>6g-7FhHCnT$RKBx*0gOh#|K%^i`7An;ZY@|k>jCbW9o?A(cT4zh{R+Cl z4hzNYDHJ~Ww&~%~3u=%9;5rLDLcby^scuv#Uz0F3=g5bz&rzkV~Asm*0@>Yy30efoYyL~H`x5`q9e|Co%&?N#u4Df&)Z6s>|K%oP5$4;1 zel~zPe-92wG(fKWVM0y=Iq83~-@4#PCWiku5x9cIifZ%G8~nq_{PznS0Nen=Jf&pd1G6nB30{n0?)b@+hKK3K2~J2$z1#eUVq2d zW=VL}prOrS9KPV6Z{jS&*+vnX{>kmJU+eZf7wXULp?B@0W|b(>X1@KB#yd!v@EvV0 zZWD}p5;&%aPCeP)EX{w4iIyTBdBwr>^ zc0HUCwZ@UFsoI_9dymzw@wasX6p|vXpc50+K-9G?_wNoNm|NG?lEq)pi3xuTLs05O zPs?trnw!DP#(%kC2ngIYd;D>1qG1npL+F=IMLCW2pU|B z8-hQ8wg_*sdP3%nFJXj?QJKY1Um=NPN0{x~lF%rsMs&Dnds-^={oKaNV7P6-WE&V( z$NQ?T<<@uln1STd6`s=In_=Bxq3RI-maz3YY2#Kg?YwTBg5h=Bi&m!AQW-8OMIg_# zhyOY6sLp5|BFvy+JNBnVj@w#`+UFpEVGuvc2BO2S#k>LO8vvX3w&b5nB~ND*A*Z0p zpziB3GT7E1w-eNfJa-jskUmtzLG9e?8+`;|8LQ|i#?#gnIx71}Be`KLpNVQHxD)JC zIu9n5VVFTQEn)bDp!KVj7bZRL`Tq#BMNuBfb{πg7FRvn@Gk-t`}*K70JrVSxVo z_VyQj2LLIUi$YqooqRuk4trnxZR#_mYzS_%KAwkklID4X}DRJ+53Ln^_m@E$hLJ<*-1BPd>TkY*NLE=sbqTn_ZuTTot>Ye90MS*zL#33{FW=jOBfl5?$*~M_=kkQM9rKEU*`-65+5zgc5rF zV_o0&%(VhQg#9UNi~b0g{}D6GL3C?yGX*u;4Z^6D?A=O6EX8E&4F3cbQL=cYaN7~V zF52oS8NNQ_+Ut<^0_?7>6_43u4Zjp|*<+-=;T{iZKYkYNk{k1>c! z6qVk3T|R7@B4v5_s}ZYU;HeA*)u=c;zqL7EHMsG6@I|zZA3E+l=1V+~Wwz;OS1^e2 zNz@@O4p9az*;wxDGl1p6*S0g)V|@FgP{G2@G}UJ#ucbzW&W2GAHCsKK%|p)Au7+gcRjxRimjm$z zAmki+l(q9Y{HID(UWZ$(Jn}5c=b%u@*5*1jW+fg)?$IDWDj_TW)H+KoIY;K&WyPn( zY+f+hVjA5q-&o#fq_q^udRs`ytp2aI{r|ch{@c+{{mmlo zTIN_;j2CZD8wZt!;;Zfx{Igj| z^FB|jKRQlB_J>Sl#O$_YFCHzkn8%(jsq%GbT^!zf!0!C6NzL^)x`DFQ=b6uNZ z{Fq_L|D?BYeZO0t=;ivz^u2Z{S2QpQkaVE{6#SgB12Mp!b;(KqeVCfMscsAL`z$W{ zy~~(91Risya6{D=s)sZ@h*OnekG2mll;e~&A*|}w+2^$L_}jTRs5>}!`ZD_e$@KfS ztcR8yih*bh`E&lnY-Db!JfLW;gmM%z-Q9Nzc=I0LqHVG9Em$5{@15!U9f>b;m?_S; z%3j7A)M36i-A42McW2Tw2_vR*$q-D*iP|R7X=?XFx&r>(?Z{;wwLSWB*$LpdnSP%I^}Dmq9Xd<(j%AM=d)k}rY!|=E zbX*RkS@}I#^AUwLUmXhiREzLq#1(>F2@qSJ)9P`)7yZJumS0^E6uV@=f8o$f;n4Ga zKj-~ciiI9xmO4xeSQjG*aV04^PO@?E^bfM%?=n>GHlzj}vteWw#eVPod&7*%ijtfG zx@Fyr^xj#bEAHz)6Xzcs>m5_|56R~U?Z}+z1gWvcj26*av|}Uo_~{k5ZvYE8UcKIf z>}$-k}Q{4Hi=e*{$|JQTy_hzv180`rL zg~6TFBcX}H;;GQ0W1b&}SVbPwRlqA_&x#h)tZ?n+-Kty}!v1J$58p!ImGLt$94t^UgQ9$ieU0#hW19+7bZCSe*J zoHwIqUwE>>e!}9z&Z8sQ-vO^f1g)Tg1I^c z-0jjy@O-#X5)^C0l^YIUdki^yNP(00mXXF}vUgSVVD71}IvpjjAS-b9{Ljur`)DE_@qk!b3 zbn~wLYne%@H?f1q3X*PST;KkWK%PfuVcmIWUti+i*RpihOoIW}lj*pt3njaJ=uxr) z<(4$zOh&WiQ^uJByIat((?X^SC%$d^M;U{OXW#8rBFMK(*#=-XX&9_?Z4k6;>)i5* z8x7tEu=(ojg(?X?=;&MZxd%?QF|Uyl=rWfo-!wuFe=bhPHG;#+#uR+dQbb@G8Cg@s z3*pXHnV2l$ z-Gfa?l5)vY~%P%Mrfr_PaWlop>?W!?_no{;negiALHFvb#R=_ppXi3-rB3+|rmNXWr5W@OR5rFt0?F zovnrjqICVRiTKv833;Y06O8Bh%&sjBSL)*TY|lP{0nwzN+tadq@Ts6zoZ;%KGhV+G zhksEG_lHFLeZB1UretyX*)VX&5ZPZHO^Ptg08RjO*Ja>*wLL-T(sn^tOdL6Mko01M zUsDz0x{4ewG^X%@K^BZz$!J?vAfvdLjco6<$ut-F<5`y_=bqy9oSpJ9U(l4!SX(2%`1(?NZ-HF;Qy)s z>V_4mWS58YOwK-JKWG3OBXQg4%y5#(Ii-ysM{ z_@wi0-=ugExXW>#)mu_7TpuF?lFCt=wjx4ol>U3S{uCR%(Xw;ifA{^5j;9O}$o?L$J4!@`QV+>ZsxhuB- zCqM_j(7z1$_|+BOebCM$RH6n9!CX?Vj-Pd+>?HrUh}kAe;OfP2SsRIaaU8xuZtWkP zuY1^OmF%t|?j6ic%=`Tj0wAiN;4EiWkRoGmATYU$Xj61u3?b-dIs@;NQEjrfu(RX!l)nlNg? zP3e~OO^~uGZdS&9BKUm%z}?eYdhu9Vo2Mln4@#8pOSy9rt)5wiY`2YFXOnVfWOH{> z9E>H$9k|bn+>o!e361-4f98^_Kjz7_*(fP!`wr-iW*+H{we`NTTLBwgDDbh_3PEWK64z=70%Tn z9i<5@-{1G6V;$6seyN9H^ZVFtVo8+L;6&l#G?eYmz8c@tv=h&RcFnb-gElFwcc=#% z7~ts~J+)eX6h>1Y_6uAWm@{q#iN=h31W?ridOOQ_DYe&(gR#zk?w2YIlYGAWb0 z-)1AwRxEIFzb)TP4}xiRt8C^cgxfck`ilCx?DBd5;1zte`44^OoDu0HEuAQhJTE?M zLD{SwPfx!a^O_9JJ5A`gvQ&S! z|6zB#`_b*87mc@RVZ?xkm{A9;C8;TREvpl#A<K&{xtc5K9lWsAJ9m0zpWyBwoO$(jJ=e#$ z76?@vfHcdSvs@>-BvW+8@h(QHp+>iwkMHhoQHD0EyrA4+^xP(hP9CdOXBSNqLLx{k zZ{P<*v-sr^=Xj#)(wrWB_7kKF=`ALtk62YQy!Vab?VlyUp(nOybRe)OSy7wg7{No| zTDcN8RagFCL41V@ZjpcN zV>b<_J1CUb>9`o1-@Dy1a?@)qZ~0l&cWMB&=9rDyQafM>lE#Xi(q(*OvfTET{g`L0 zb1zF!DbMCh)v&aQ3P$oD;zO&Yv!CYUh1Rk+GgZ$X!@rWB14P6ggMzT6o)1I%2JWi^B8MEZ<`A zJ$$lIcc?P;aPxS7^gyhEYP>)jd+& zwL00!s&4RUC(-hmtlGFXq{+e$WkeWYQWl8-tmB$Xa! z=O}Ebuzp|&KD4j^j8sCoLVw%6Paa}Xmix7V6=4(df%zwN#+bU)~FB8!ypj zm&Ny9IP#O}(mG!Uy_y?C{^Acj?0ImSyEfV4`{&?_P@@t_mOj zTIZ=mF0!&rEz++Uf(mMtzBLyl_qvI}jFyVWnRyjl>v-n*-eA1ypADO?$aNQq zby9}@!7wZf(sG?+?SbeJ1qf{-2;QRy1RQu5&{H4l)La%?wb`w4Ki0;f@WL$H8;xOm zFTi$s_WR!t!SaNkduvy^dJS(ha^<<+o_fYE@nzmOAFD?mDf-sG4%oXnv?u7`r)!9C zrNRdaRZofL)lnR(YPypG7#1xyPxQ(E=wsU?;5GExPPNJY#{wm@JJs6y-bBj&NcZMl+p-D17s)#hjqxBfJ04D~%vF!TIQlI-+7t5pew zcuAVMpvS|V!6<>gnq7O;di9^62JY>Qi0eCd^X+rmw-9Ih>{lq>>7i7p!PjQ4W3V)v z9~-%fvlp!vYfc?>Chs?1E4jP2lVd44{Fc;AkNsqS?OKoI@{zG_)#pcDb9t6slXH&* zyv_a1RFRB$tlNuaoN%r~$7!S(bsxOf0w>%s3=f9?odfzq*#37m23h!jn!W+$`-cJs zP2gZA%^u7V{ZAXSgLO1C4dp^(_i7jpAB(}f(DNN{03BqCfwU)%KvV*K09Svi_)iOW z!D~-CKsffqxLS>B)Ox(AIMH=_U~0soGVg_>IRCqYLibwnjt9G7l96qv;M2g3X@W=J zt#tTUlgC>}H*2rw!6ol>M1@Unax-%!Y!BBXH%`8#tZ`_}oC(HZj8U?*W7kDng`RWU z3Y!jvX9-?7)EhASKqMV>efYR-Rd_@qW6CPiXB$-SR-)8FlY-x&J|!_#bsxB*pzHnq~-&T2i(RK4c|B0F~NIW)kz>*|51<8~o^~6d(;(8Ddkwwdsy}8Ix3S)8N<)hlIBRcw&o3$$**Ld`iqx+6 z9?h7u?~)H(Xe2d6MA}$SV3AO}SM(#FAh%Mby_mt1fh1xZX#|?&Lq9t_Q+4JVWbMI> z(N_8N7|+IUJL7cc#~M1$I@F~NT>AHY)l|5G%P=vSkNR=cPvZt9NJ4c=UJK`9Oey#d z>+&gLGQC8DqhAXZ-yo#N3CXl0zj#R_+Ep6yHy(YtD-^fz`RvkxCC#PlKF?z{xKqiY z%l>Y&COFjDdY7|rkMl3yx;+}G5#{?!VobePYaCICyIDSbeq4C8`>}sECGU}TttJxE z`XEm5bGp~naQi59(&Ol>!HUO^%PlF`p4ylS{Au-!2ICj$icnOgs5^_upC>^Ck?J-C zQCqc{-20ZAkaF0xC8Bj4O%qcP3tc~8BQ|W&VSB^+>MKPhX90#&ywuWeTAf*BcC)A9 zQ@?h{MqkCFT}8bC8t;{?x*}p9m5QRmhGAZdYknDF%e~)WG|ML>m9B*8`LdT)h}L1~ zYx(*`Sj|Ol8mz122F93=>Zb^&ZKA@w$4?rJi`AL-LHJl$w@LbB{<@zL7~?fMwclPO z*09f$Jzj(?C7SB24>CKJE=v?NZG1YL3s#h!JLx>xRo;OX!!Ha{$&AY((H{W1&SO)J z_#bah`%LHG=X29|f_G+@e6uJ`^(k{eDc99wUSJefs_4L3?VMp}&GqQ$rn~vQTeYd# zLztYd%)w^$fK{WVt^uE7EOR`qK5BX0 zmLBv;D}ift@oQV$fQ#<$PiLR|!7nedK(ExCZ*n0cJ0~Drg!N=6kpyZjMNI`CLDQy5 zxLW-HMqt$5Cyxnz=@kT_l)A0LjGQaCFw`G%J*P&k7fG8WA(Zf>Z)^o`!%7#-_q1cz@p zQTcSyJ;+SyeJE{qddcQ^aT0St1o>AZa`pEH{V z6}>X>Hbvq(>3sl4LXfNFS4RAF{llXI*6t?4?uxIHj4#ol8H7f0p@!k>PGzHU@;*;NcE61tD)#KxKc6rSFq_f7HKK#Ze z`Y8HnOv!mFx*FP~+!Q2$FeXV|$zqg{E@=xSe~aRyK@r}@2*qFbcm4w@hIPI3!f8zf ztZI9a1J}dDO5Va9-eDxIy|KhCSTHOi>tsBK3ayTq!%qteW9KLZP!%S|>~O1tXs%NI zRJVb#1Oj2{iS@+*I`9*5cVD_ze+Y#R5+Q~YUWpR|eOBxtQyFO9)QO6(hBTJ*VIawhN`J=bcxU^wuz?tDV1a!^qi zW7wxB{B?CqXfyre)l-LFi$-|mvf78U-dtUm;&7@?oDHQYIzCnTLGAyNKvTClH*hH+ zDNr>~^*PV8-p}5wFwam=srVyI+h3fsDzh5v&A}D#E@Fa~6PBZvPsNLrG#-iQ%wxo> zMhNa{b|5{T*9%t-_R~qw?VX6}esssoM>bgR#u_(7W|I;r|HTzzt*XNvx=)Dz$I0+% zN`qVOvv|Rcx2yUx8($tbOd?uniSj>V{84G8r{$|34UA~-_!LHqJ zkB&*TcK4m%ic+etcb=27A+%(RZ@90Csbi&enQ3b{j!$G%p3yEybeY&j%4+NZvWE4^ zhr7?(yGt4)LlE5|-NG9;GN5^Zl;U5b_W^$4JUCZpmN zjv+pOqL9^3O&waBR_uIeDYA5K13mR%DqI?&NO;J58JbpdP-?S2P|2Vtr7=sKf|!Mc z%fGxyQ_lPeFJV@$637yETI1f{oe#(_ORfx{H)E6^7`|NNkb+Gv>c8dh-Cb({<5aK6 zTC`mF*2?FafPN$t_dr7X+eqqMDjy9(z5HZqzLVPY2#tpHVo=Yzm)_w0)Z#}zb?zsk~1Ag#E+3$MP95;2r^=aF(aegsj zH)>}|Np+tc#u4!kuqf9@yNoxC^@_O2+S|A~Kqr|f zk%ceYuCw{bpPdfb7CzKkgRqF@+P=Ae>=Q7049v6?w&KmB)>EZMotWroFZ(7rKnD*> zH)V*BmbEuVKDqUuvH!47uJo&=`cGhTJjfRJGMzx-1Xd5TglU^)&{_|*-vgro=i#bO ztGOkh5^4Inq%2>U9Pc*1du&vh>qVH)xx>4lra*m z~ltOknCH&|)zgqG0K!{nq+=rasWzcTU`H<+(Q$X#48*DM(8R z`g$-4N}1GGV)<9S=E+{y?>w4z9O&o7uvp2PyxpK)#hkB;n zWGgk^SWKEzzg`hqKNOvPa7azSkT+SH0vfp|v!<8{3>ds`?$hMt{0;}D8{u{8lBbH6 z+Xnq<{%$pw5T)6%^aj+Wo`;|Pd3NFoR*&so&xcYtXpNsV2Ifd7D5J0V=;){iBSH4h z3;L5vlv(14c*Zw{?nnN0i!tnwa%ERCirzgAqiN(qDOnw4)H0QAdql_~;*q5Uq(cAk zTkTe=TExgq!!BMv9BUtFE$u@;poi&s zPaOqW<9O{-B7`w=v0Y{~IKYN?=;JFM+QnIzlvX}VDL{AqVqB*0 z;d4&vWQT^w61}bAwr}$_DASbuVm5C})s^+Dao4t<=lf^Mzd!|go{A4#d?7?{JauZ( z0snC!En%-N$Lt=Pvx!Z~r72<9Bj<|fKA*{qRVU9`7^!xjT|aj(T(W_B`CPAEFMXAA zcWv*YlIX0gv*yMJGoDvsGNqGS&2nxwlRC|hl*C@a@~sAu2{buTf`~lxJ=can{ms(W zOlR%1KbPp`kD^+%$R`@<;3MePD;+)9Xpus%68}%~AM#VQ35*h1(Y!^cnnAAW`sN88 zw-&rwzM_`v%saSY4wCgnhti01 zX<(gyybJfP+)0B_%uDWF5Iy2?pb^HaYLa0rRjYmmQmKf0wH9^G{M-5BD@5E`8nlbE z)DvfG3p>oGH}3SAOe21Bp`nw?_DBhkl5n+&dc~JCFP37-W^a)u?SrBBTYhQV-m3M} zAt0PG7_c%06C1VKo0g{ad>%I6Bx`Sy3eWal@0D~KhDjNagVztq+uf9JIf^nO!gEoU zrtSpe5^1t`MAnmb!DKx+k*mMhyQML~ct1gNy>0w0m!Xduhitt2UbG zUy`LA;wR)t3h`QO6|)N>9$wRpbS4KP%gEDD;KfdB&raqiE*}2-pf9O*eWb<1l~yFy z#4J2pVj}L7rPsuKNsfQQPqC4gsujAdOP#`US_gK~18n})L|wd@7@9qq{ctin+kp{f zD5W72mkBKM>6CJfT#$Jt!#IU?XKOHpsT^d6jrdc2+h+^!NL~bem*wBtDJ8%nHw?8@ zA%S&_A5y=M*$Q?Pv1e`uiyC9K#~LN&)vJ5vCuxtcSfn1SX)!Hwi`ljj_kkV^L0^R4 z&y_Ev7sbVOkgXCa=*`&)^>6K)vy-$)%vw$5Sf}Q^GsGCOl~WF_G=fw6CHVy{-R!8Y zW$dZx2H=i)lMlx3c57NHP$j*ZuHCz+y)~BA;<4e(IJ0Veh2~jzqzg=~#y6)W_}+v% zJzS5r5$|P_Gy-g!w!H60NUry}1h#%wDUN%Egkb?5&7A_^uSD;ft zZS}8{P^B|dB3D=?NpC8zhTKsB zSA8T8Hn)VSeV^Q+eD(F~vl;~@rI0{dE@$5ADY6>&8pKH7hdzE%^v9}^*T_(4`-Mvn z+$xl>+kbGYx%YxYiLp;7JEH##$CKUnf%Fm!pS9y_iKX^YlU~x zNhlrj&@GdPu16zWlPNh>6|uk!`3id84HwLa)icMP;PO@XKU2PRTwJr9sA_40o3v3MJBlhxY*e-F?8c!WsSLpe<%i~*r?Q4Cl z!#2QXUE%nHLuVx$(95=WFyD@z)FK@8B*-uATn9;pZn(DiVF#D?x3kA_C>-hGu_ZZD znsw3)J^KJDTq9INiCpZ7Ls`+<*F+K-qeKY>o{r~TYp9w zsc#LPmv&Ty)>NAg{!@JEOV;=|o1R9g-Zb@dGAH=TfYcQw1dz7r750yysx9Cz?Nf<$@ zzBRT6Eg)aiD%Hi7^4uT!+sxqVMiY#a3YiLX^=t{Y=dDB;St+`Q3UBy&;2cY=IL2Nh z1zFYi>%9YLZ|&tb-2#z2Y++6%pAL?N73KMz2(~{_FM}Mpd}29)mj+!YexS{W@Hx}T zj90Q#@LIjmfN4G0l&{lhM3H3K#E}f2kN89~b}g<561E)7WR_>Fe3%e}+8AwpX5Udcxsd3yj&Ntcru_w~*pmmdS_1z&1P z>t~%}ON486sK;SED-Mofg;%11sp*YJO}M*9&sy{se^bsbVplPlvEWs!10ffW~UEi`C_hlk;F2)-rf%2r$t-sD*6+1?=_DQLe5BM+< zEb4nJVo-)q{G~+mS|Dn)ykk(rs;^j=v{v^^)tn4GqX#X;d`4eOhJVA+wEG0Zu%xsl zy@-y*a@c6P+wzO(58=99anH7Zg8cz6rtr0pPramMX>L8qF*lpYH~Q|*s?W!J>&vrE zz(dy0-h?&II3~mx&%wG{rd3Y^*>l}W(BW8^#SkIvg=_=V(`Q#~ql6^!b z&P{PWkZ7P$-SGCl4R2cy+A#38nuX%|Qhh?MWL}&+kAJx-PGXJlID=yFac%V-0CItaPAi{J4IvqWM9?@2f@+xgaGf zjmbMjZ)@Z!B$y2FotEP~6@NvMgG8*h?-PDpE%l~OMD*S=zr7kO-TB+ud*_3~43*Jg zTjrx?yibMtT@3?PcL@nR{PvWg9;J(A!1)C3Mv8!^)ViOjfpxCel!=wrj65MJ?LL&c zPGk;Elqi3)I}}?fV;4aS4uEia;C9zb$tSxS!rG=_nQeBh{JOGZ(b-M@acW+Aac)hQ zPS^zB`}`t+OMpLFfa2^vC80HSiE9yArcYO@gCC?pJ>zML9|y-_;dP^~CzU@2g!!sV z5zy)xRK)QICtthi;Lx34l$(EX2U-x$P}CYP?V&XeMcvua&kYz6Q^gTSk$!$iTfB_N z0SA;polL};>M%-$QhJ&T)j~mt7Bl*8`VbaPHxeFG&TYS$Y=gmuKjo&Wt-H=aq~`W_ zy<}jYk+e?3mz?zB@5H)}Pvxm2B>A9-srl8S;3)HsNL$TM$d}#yS-s21Z?Sy^4JOl8 zs$SnT45NLs8Lf7X4Z3}vBE{~%7mJ+c%v|ujsf!(O?Dx)MCl8I{pK?}SC30~MA$_{k zTx3@n8T@X3B6_0DvS`lRO4T5TGH#~w8yn4TetdVNEknOIzzivdRipVJNdbNFG&w?Q z(o^4(=EoV6cia4`yP_XwJT`I=beNtHIe>>>V0#>L-b*`bL#Q+?z620x+Y964W!}b` zl&U(w!a$@vHYYHkqM~NpOcu7~Skt%6T$Oi}Jmr9=7Tu%fRCGUH|0QVq&l>NdpkXg9 zP3w`*`Ss8+cv{>jc6unD*1~<8Ml41s_=dP1M>?$)S(5VCyHmdygeT?atJfEaQg-4z6PWkvkcNKo-@Dm!b*0}+n+E&{yQh&tvZv5>?y?8{V+MMsV zW67Aq_#Ja@IFjO-q!#W}n8G`8d5J5~l&xAVN#FW^9Ig(-oeKvYT*Uv}zcs$Kgjr_sB82-o- z|2Qq(h$y#Y`s=aagT`Yt;icVOvSl|?X{pCP`Rp7&P*@Nt9N!!oO|6mYJw-RN(l*@q zPDX`gu#A8Aw4=!?q1pA_nsb5_F6yF?axpc_rhRNiV95y-fLD6boOZ_qf0_#~e;vA; z+oLyfe?6=#(x*B4K|J~T6w_wJQL10ldNc&OKbm5JPj9lKfP1<0Q8Xd7!yZL)TW?nt z)UKE7iY+=`<|nX$z-i9ycUHCQTq?KfwF0Lg!g7D=jL;mgo`y*c?ZLA~WSZVA>R{nn zY^J|NJ;*|4hMKgW<2h5xHhvy7TEn!jD=z23M~WMx{C8G1WFX)98KVvn-@(>ZB`<;P zTCyu6FbU>-T5HL*9P2ZU(VO{upOVVlcWSS?`a#DU8TIm4)_;TToSHBhx-psvKNc%M zHxkqu?h*EI4()S646p~6-@Tl@$I{fir!pzgU?07&=3!*1ZCm~BmSxefr09x8b93M3 zccf@Q@DR|@FW$=pY9R2d}KJpTEYjV1Rles`O`hX83R6zx;O~!o|hF8k0&s(pGqU%9I*|+|l zVJV{6F*WOoQ@%Pd3B%k+vKECiKWt!qksz4}*!-hijw0tGaaF^xt`D^C3D}5F66c#h zF788y1pw!8LgTM24%!}v8#O)T3X~u51kZhWu!NDjzpdL_J{3;t55?w?%ByjV_!4@) zDp{&xihPgj%368FQ>!(C`H!|Iacv)#VvgCra{iUIcFx|E&|=34D|Wa3c(wLS!_*G0 z%}5olw#c|O=T?ou@^1v}> z6gX6>(d5Nff|K{9#Q11(G-x0%*V5=gk*>EzC*oIcrUKT4 zGb=CcB;BGxJ@%1~BP_L@{CV6=l#p0ZGjkl|b3R88%<(U@w1w|%)+7VsoyL>=$}?pAs^_wgiY~GJ2S0F%IiRPAMu1)-J{3rSyrgd-YDT!$1+}c+Mgt&(z;0d!u)oV zt$Jm94cB^@HB`in3FS80aTlN6r+W`ec1X3M$qf(HArhEe)7*^F5N2;YZ8X7;=c)Q> z>XzSR7wH<3{+cSWX1^^(sDt-{v)<$fw-~mndO>YGfwCwQWqbOx!1T9EqcP4aSq8>; zO?%R*UsGptUB9{xP6W479*)Gq_pO9%3V8Zf-f0UPOf*Pg#5+!{^}sSCZ(92#THf-U zY@i?F+?avWpmuMRs`p7pa@2Cf>xc$t3z1PHNXS~2hxxu#ZDC)8m)KK+Z&yk~r&lOp z`)`yl0 z&9@8;-A-{*!dytuv#jU0=?!^$rtH_j$vGgcp`*+$@Oh`c>Z8HZ2EN<;GfWKmPyxFe z8cy{=zZ`MAI}A&+8iDfApoR{$UoE^yIn^=Q8JnHc{WMtc8S!?8dZt6T_n8F8gEp3< zDZ9xM*bFkN)^15LSe;_ao~U)njcak4CPMdP1wR!IF7m9b;vd&eM%v8`t9v=@nxf9U zVf6P5fg8$>u-SK+pvVXqj_k~RMS<3K$7Nqb|FqVmw0z+&%;Xr?@RoD!RFc=nF@-D} zo|YK)T{I7eLagaY&cisj!N*?b!Vt{7F^d%pxx=t_PERPuKYY+o?*`N1-nbAWE3NZ#K4ZBW3_$u=VW3e zKxiyy)8CF`$ZF05mMQsFuY;6VT4lDn)$Li~h)vD74VO5i4XbW5SOtG*>#@831`I}i zV)Bw2I*uug&NL$`lOL*6w#xA{Hh%Q!s#r?9n)+0=8`u<6%@8%CK4Z>9`HI?^@ih}N zwk60duA)KfYbarfe>jq1q9)I~gV49XV4x$uElgtI$8b*VO>$bBd5Z7md}cTCSSg>+ zdtCHk$ttvAO~cfxfT30~1kL`^BsFiYC5GO`7KGA*t2RH8mc7D2jPF9VH^=uMiB@wHi`%lX#qmeK}DshD2NbH z5s(rfgchnDktTxF1QY~RAV}|^BA_4%9qA>8P=gR!0wmu_z?pg8ciwM3e|~FyYu0i? za_>Fo?0fca@3Z&s>{GSxI(eV%Q^WWXd9x%QsSJ@*N*$?oc0YI~h>_NAjLOKaoWh?S%z=r0_M};xD2klZ7eqH)_Qh7eI;@gV|By#lNo< z@jWs)$va@*t>fq3A36&o)dXh|F;Yivjs!r!`dTsmlh%!!+rrN6n~*ZmA6F9C=Usc^ zylIkdRs8Dv;hoSwT(iFcc`caw?T(~N2`t6=o?NT*^sCEh#2Si!_Ww|~aG8qj6K)9A zAlVn6RkO4lO52P(>{E4j*{YlwAiX>I)X2(TiNRv7 zoL%3CUn+`%?v8mC+*NG@xrP=gFiUu6VZxTj{t!TqOP;Z^cHy;wL&k7?T?s5Xe}M)? z)nhcJ3!0?6J)$-s9Y!AJNMDdc|`1Zw1> z^%~&TpS{&943p_ksN47s^v)+1R%~WQ4>|g#D~{)e)1SB^QV|{g!{VKw zhBc(_)Ta#j;CY#R2o#}2+$Z0&o!}Vy@FQ4STE@qMEH*sjMyfLippXxbCf*iWcKa1e z7Cjb2Nf=Z5y_s4e2y6sZy$8%5-cab%+~1OOMORt~u9Evg&iKuVleKW)AqRVYeeeHk zJ<6{LKPwB}`f1@6Y7{ZX*jyJqTp<3Wj5V5MOl=i#(|+wMbuBPT(j#f+NJ$V8c&N9+ zRjam~2)u81z|x}Cy(IQ%_;6CBz#4B|*F^zx8v|I9Eg|aPu6{nH`M&TCb<-CyXozCE zo%2J-t4mQ^Zu=?TEqdtozLIjyzT%YjL|&sRCAms>)j)r~ss2q3o{m#i9=LS^Pd&;{ zUiJ;6<4#Z=8)LG=Qujz73GXHws^_C)Mm~d6uYD6jPH|F`SHJkx$a_6crUiPJj?HBz z4#F-mj8F*Ea2@gb0$vDX3H~E?uJL|^14~%k4h;(e$V7B-wfwbYyhtQ0e`P7ws%=$Q zjp$+txtiz1AR4_C+N)rBjmTbm-b5XU0Fi0RJw3v0Rx$iNyD&A1W>7qcM2O=L3MiXk zFw<9SX&-NCkQjjr!j?YSXq*~5Ct#G(bYxy585~Q6rHlCEs3f9aDOUYDV8=9~yux_x zs}Wh~6nZtaFF7L#gb&P8JR24O+-a`#1uU&KVO5d`Bq{fW_b2(Y(mn=eL}03q-->hI zkEd$ls6##R+Z!W{0!yd7+UZeGB*mBhWOPsGdr1V5jqCtLWixMN1@fG3^j!fS7%%dZ zjoaPu-=c&R&CX=R_~k)C+HOZ>Kx2GCu#y>I#6a9{#qA3=R1YtTEBOPeU$-lE^A4L7 z8caG}AlKoc?*|XNr$Wgdk{QPQ=?z%J{+-Evp}|P0oic+9--USsOwuk^?c-Ov@!^of z!gc`z-p=TAkPQBYnj4}oB!1; z;w|W}E~zn65l@8ez6JR`3*>FocQ_^94#Eo+^gJGW#hrG0eq{isz{5&kIa1J40%X%{ z?!sn23PhcseYT~ZlPc1P+8SMdHUfqoc-7drv*P`xxVZSxBB{u;hXQ_mA-GGf*o zntSDlpUiE{ZE|`K)=*;Hu}!MkSy$wjGohRo)L@MChzH&pwj_UHHzca$IJxltv(i< z#8dqU4LgUs(59yG4K7R#fcQg+~ zcvvPB)~BQlQJj5tWa&@Sq%I?4-uU`*;NGdQkI&Qng$v8bFY3ui`(n<@0HsP^u zH$9GHZ4z^(J@%qyBBD8ubX%6jXy6-k7BNAo7~(Z2(&T_*>G{^a6(`d66X}JM z2wAm$M5@_`@cY;Wha*WlC=HG-PLK}235aebFDr2MZjb~DQV9;Uy%@hTtgESwMTh{3 z%n&#VU575q^hjO(ieI`&K4xpfP!~pau9`B6*-T{FYO;og+!!`~XrwVmAr`q%kH7Ar z#U2hAwpk5aakK%azcm%Ebju{OKRu!K&M*{&uEgn68H-R?r|LexXLz;>yT4ReC3CSl znKYSDLH?So$xcnrZ<_546TKe~t-7Wlg0I;eB-#5tsAE|;B7k1Kba~nduT1z$fahuQYavr)TnG)$#;b_TZ=FRRIHIm_x0QawSid}JK zH3>%Aj7Q@f^@)lvQfkh_-=7k}TS89E#%Rl3jq@IoH`J)sCn$PbYfCANxgI-l-BknN z3qc<5AhkT!?gp1K+2@X)IhPo&!J|!#ux|U(;usm{_vw8&Q4w#pwn~P@9ighffGBm} z;Zc#|yE@(2uzO(9&SsO45bUdn@x$>zE1ePv3${qwDs>N9K9y(toImOp*d4R| zd+$3oC}>cXKVc2mYL#l0YA`27s3Jr0H;6Ph}lzAr>CU220hh zXyyZ9(VJI55eId~P*vS4jU|4(&%?V2kDKkNXfY`Rb?u4RFYZYgf!Q~w7i-KuQk!Jd z^kHb1*IuO(`l4awKKnTs^ZAA3heKlizcY^r#h36Dq?Q9WS=^{P#xtnu*J*wfkKDVO z(k&^2?|-Wq>uM&v>v{SGVREg-wvE+7Y#}^MEp5UYuGH%}0V4OFF{5U$1Bf+eCLJt> zNXmNvx#FdKM2AB`O@;o60q^+G)u&MA>b%HEekE$(2)ur8;Ov$$O*C?h?^LxSFx6?tz}zi~}UZBtMr=?t8UK=WHg4Pn1CQp1M5;e%uinIH{Pyxt3m>rSWP6J9hv zYY*ZZLe1oJY9Dk?*Zh-rtfMLWuo#^;RI*g?`(fIyr9?$ZkAn-_pl6OH)ez>B zAJ38c$b`a9*$^Ci-H}4f!A;bP^qh4vcRcV!0)k zRL_eqAD3K;gl}mcRMklc+no}D!XGjk7 z02592Tlvp9FK95*^#rCLiIMo|kHRnTrvutg|L}KUSsSP6+t5gE-KsPz{7G&(fOPBz{Si=xn(cM*Y~RJF6{wN_+$yojUTN^CA;6S*SLk&^73$t3E$As7ejy z8SJO?i!44o){mpYmlQWW5pe2{oNncWy4uuJjL+Tonclv=k?pQQq*uqRg~!2Hi|Y}l z0RhzTGrj6071F!5XlrvkmFCy`&HkA9H7YzOqVX2QFmFue7gvzC-C`)GQNRz`cm3GY z$Lz!DH6lC$8ejQgW4B>od3S5~ypLTa z`lpVnhYbDKK;iTTL$PN;RWMX9YN{XszjvkOZkSuu|f zO+4;W2C~p-t@kGq?rKqjj|&Uf^P=68QjgAt9^087TKnoa&#{KvSv&0llXkU^v{~i; z`5uhK#mP8g$osGW`imH8%YC8-f^^90DsX|UpIIdtNqxa~euGLGct5JiZG7MZNJ=}B zxsR{-8R6S9u6pp%6DPm^ zp~>40e4`RVdtlZls)TOkUU2c6^#Nod@Luy4N$7`@5_rH(J(K*L0s7R>Xt2Ml?RXFGJmz^Q45lw&>oX8)4hqVy|z3 zTK>SGqw#?F;)R~9ctcoBu+EED{ln5B)!@bzr>vMKCHl11pmZxki=OR0_KRUviqld; ztOBLk14$a3#705(LWZ6_^|tW1`OxZ1gV5Mt#>GCxCIQ*7cYP@Zd4O&sJXQaeHCdyL zoiTHAML(e+XdVND&V|trv$U_|+^Ez7fkaKCvX-DH0h?b$U_gl{zZpzEt*Wz^NZ^c& z6Rz_-@fJx2nJ|x}au{-|;41d>Rb|TqX?~YFOT6v`T$pbo+vv`HWM%EXLRI&-u{;_U zTb?JnW76U|S2Qai+`2C01UDeV zy_rol<0B*OQ_2nd7Sb?Dj(r;98O0Q(@S3Yqbp{s~UtoBRaiOr>>@sWc4w}SsLVAl<3d&|$RC6*`Uf0q4M&Vs0t|u0H2-kiTfWYwu_V{o; zdlasnZWgS+OvDDbs!y)!HBb|)X5jU2!rE5)LZB`t@nm;S0-`(5!!aH`m@K9{ZzgQf znsua$-8HIR1FfE5&OrgSX>~Y3tl_Q7q^E`tHEaYBX|6Eo&|Wi>Jt!^C$g?PGp1*VD9GSyhB|bAs91@x1zS29TLtm_ zdd`Ne6QM9u$v%m}i%}Wdp(@E4+6!5pVEfO$b~luy_m9X?%q$;kba`&Ee8#Z=z-=%G zZgc1QRIU+*o!UTjJ507@DXf+fs5$d|P7p>C476?ip^a0{$1VP-z}F8i^Y@8leR6oO zk*rYtwo4Uce{n_gJ%k-!Upx?y5|Gt+tj~Hn=p#uGH#{WuaqQ~u2;!fsq~lS$`P;|% zm0oY+Kfc_j*_Cwn6#wsW*RAhGY~CJ@YQ6aoil2blt_J?WvwP&&Nt52Ivmfq+D97JQ zt$Bbq6mnsV;6|1sg?Z((6l%|%!9HFVsV*!=n6 zOMQuLdq~!cArZTq&zJ{^_wAy$T_WsX$+P!GMVY)oeBB&5W6=2D`E~} zDU+UE2;yk#MbCsD9AUQHsgUO61FuJ}c%d9u6%uKgJr@!sGEv(7d!vn?#H4v0zD55smZsw7nERi}?n%S_9zZMDn`n>?GS@6)3FD{sMU zs8z{ci$R~MSM|-)*BtXbrq$Z_R+l@eI3{1~ws$ZF>1vUL@Z(FtcH5;7k}opPweW?q zm?p-HF2}$maYBrtI18(uo&D8B9`ZNWudS>LYPU77X_7>A$UCW`XT&})NBDl& zr)6@mUS-JAIPh5D1(NJLsFmZ(s`sT?)K-tAw|qSlF2xaUPUX+3Xa({sVRnPHXD888iIkhH1x#!lPCk#(rsG2?S%e3M+CSnBYqzpwdZV~Uzeg4gR zV%0}Lnz;w$rnfbC^j0|IZGrM%LA^#jCiEg#V2OIVW-127o$b5@Q)S8>mX6VzC**)p zUbA4*yb{CgVl*^IugeI2TP&gD_&oc|j6`*5Mbo{FU4v80(Q1a51IVr{uq=*cFGH&p zbn)Ik^lpzoL55>om@ihyNIL`l{asJo+SsS;!5{WmP==~;SA9~SA8oBv*wQ(?mO5nR zM=5ms>Jr!mvt!SV3E8YMjl48I#Ao?WHx(zccT%cg>rzsh;)54%rL?WKg1_DZv7%P|aj=BAiD}kzN>(I2y4%?eoisEB-8x z)Kfc4hh_}cr4z^HWU+f$qrEe0D3vu+3MdKbQl;t&PI6IM3J!Wiu9?z}AI@K~AZb*_ zLo3VZQFA%f3x+d|d?Ls7>p3O|B}a0+T{Al6{=>d~EPD2gl=me03PoD2(wwzoog}{^ z&TM}*Gzek2GpZH*3gIl8SN+TV8EyJO`hzkaW16NMA1l0k4x|So^-_X0y?R=E%Cp(;CNk;yxc#rHPC`l%KmB zcJhz#15!Umvd4Oe>Oc0ntO_2hKcqFcizlUMQt^Ajj^Na>e4{gv!iC!Le8F?p$H_A7 z1OQ=k6^`uv=uI|{&?x|~tEZM!ofKbrbr?z7TJ>*G1y($Tcvd0X}WRJ#2={>4PJ&xkZ*O(rYqq?h}#4|d@FRdN=v7YAL#L_Xiv+VTzB zIxMP-zhWgvJG`VxaQvqfh8>&$*vGbs_yGP2Ey*;XOSg z^WEYz;jkwNUn@wV!4*hH2hB-m1Qsf!`*Nks+aYlR=D(=gdujv*IQWAxhOtOg0bjxW z0=|}rboTJbmbugH`&aJHYG=-WvG2W|I-we~cDpwfKjL>_n)wZ2d`OkRw46V0&w-u` ziy^|X9|$eW}w=5WlR{ncjZaG7vR$JJB&?2l}%xBLt!CzojTu-nm@DL)0}? z_p>?d!)BD51$iuWYqPpE;j|yJPCOM6lkdn{bjKllV|To!TTJV;UNk?pcBnyQ35WQ@ ze?Rlv9h9<@2m86dY*6%qjCH8E^;Cm@sBh*{iGq-a#J7lf??+=HE%l?{=6i3ZIdw5$ zQx5-_7zHmI2cAFbVUB#-=Q!4YiG7r>X&$PXQgM-8(6Rr!-01avffoigSqn@g%QP+^ z+G`YJ9?zclcbZVag|rVw-$*hI{}6Ww5RdwOAeXx$7rgIl#I2FxH~Aljk?Z<)6~*NT z(?iw8iJ64zmI5tI^W<+ZUg}N~rq^TZ>@&4+Ox4swr9{Wn`qyAfVx#v!DIwCO?xu67 z$YZQ3K?MbcP9KwgpQ{|$kPxfdyA!;$f*vd)9uw9VJI3=AVRN2bk2_@ZsOq<_;4-bz zJstvS2mwl(LTY5yXzhNdS05NrXTT=q@lH}=`b>Cg(U-?3b;}GR!WU!D8 z;Qk_497P=*Lv%#KN)Ii&1LF@AL;6I)uV57Ir8g! z)fE==opnQS1|CUc(+^U>rYgA^Qj&3!&mzOEu2fhpvk`rZlt^OZ1D7PbVXf%8{0Y$v zmT{o}%Zg(S^blzEfZy>-Il$oBW?t-!LU;{il}J#gjRG-!WLT0Kn>?!BgqEUfPX%J9 z-T+TIKcS;*rcyhjV!mOr3F~*vg+*K?Ad9PTexnQn%LpyT*SU;pc&DEPtD|esp@;-y zFNGQ4sd6US3c?mXHrASR%f^j0sn(U6j)qwBv{#~|--pAds$Jn%ribB!k!%zNWUyuG zK|sb06j+Ou%c*<=XOA=jhOVSC8_5Q{7}Vrilf9YUc}}$rqZLV5Pg{=JNhq*uitnm8 z@uSl>zX3?L1Gb2M!P)C~E}8CQcgwmEF2Iz_DbHP9U2YoaS_Kos7Su{UAV<(3Yuk@LgrVwL=W7sPxrVW8h zi5U$6eA>Bs%RxHHN7?jorN+_-qP(2R42-uvMd$0#<1Z2(mkoXa>}RbV#7Kc(n(soc zzf%$Ey7B7)Xy5&QkFTHFKdO_;HOk9>YdB`3+At5JydH@Cu#eKLB~;D>i6ut~&lSiq zt`8FI{Jah%p$RjvCEJw8j<4&V_*Q0idCE-W6-qZ=IRJv6Rf1qMJzH z`y@fb-p0XNG&p-cKaBEBjv&Ve32>q(kN!BRn{+{d*)$tQPGnai<{lU)l7jLzw1Dq` z{vk1+NKPQ-lgDhd5lu~jLFD&fU&y4Gv_s{VglTn*_EFAWMq3rNa0kc;B*@5@9yq$1 zp(b$E1j(KxYI$W<1q{8^ZRD`GS&Y~qWy4#G>XY~M8sBy}X2u0dGI7+ViRmkj4JvAV zJ*!5V;VuY8;``7m3prH`Px>HrC4#UEdx*5ErcU+$F_s{X1-n6OO$TqAbP0Wy3%9o9 zeUJVc_(Evr=`Qcwp_IlPXUp9nMb@=An~^LZB#F6%0Vi}?h?$V-R^(U03znd5NgX07!6DncWcshF=a7(bjBcfFlGkP`={Y~QNl2k~PpDSKQuRht$7R&68 zZ(@MV0`!q2AoPR_^6Vc_uzBu~^_x|}O;jo4;Z_Q}axrXf#?Yr|wJS)=Y*F(^?l25& zt#lFaO<9eo*EVLY9nOgkkw`Wjm1ijU&lyqyodGU_Csg)hlg{cN-uHgWkc#EZtlhz% zE=)}m>8EbK%`L}KgFoH6sR1JimGeUK`UW*;D)JdewUWK4)mdg9L~2>?IO0VUwhDUF z1)U_loBRi!x>a}4w22)@p8q5guCs*O4wa)?x?@YHCbVP9X=`8)X;^V%U08D9jB)Qm zU=q36hB9kygfMn4L!BZ2UjKpl&6pIkb-QrvbZ<4=%(W;!8d>3OkL6Gn#azlW(L+|N zL5>MmCIk_4r}DQXJRhRb2VnM$Y7B<1A9;!Eft`Lky$XXyq_=j{<&*+k#`j=o=b zZ@|)6$IB$LKC!){+29z$Su=OY7%0T({Yo3j@@|`^ESXyLT0jxc13=I=T3VNy=eE!@ zyRUV*&E5oZZ;t`}Yx+EDkJyXhl=?2gze6hOB!jGYny5go+ zO1&GD&@r6s($kJ>Tiv=nku+%)*oddJwrKr9voS?l(e_bhp96%oB)p#44CQ^d8{aWm zOe(2nKz$+0GgaDl?gT64WLzLfM>Xz9onM~UT8d_81DR+)S36U`HbtL_O$8y3v}1T+ ztHP-YVngyqymRi4(XZ+!=tc-LtC;L-rj~K}Sr%(uf{n6)Nyq@-!4-+6nV~3F-R8y8 z?_aQ#T(8*Zt#X~p_p`+916$pj8DGu)QgD8r6cr~V`Vov2^r}rbgM@I#DY(~;Yo1B= zcL`}r7Yu8QWtE_Njv0zZ?aA0tSlNUfcwPs?0>8?C5lwb~DQ7sS$ei*$VkpL(@<-Qw z(XqlIR-P(LDH)2DO_5(|cc7V5Xgij*Ug5`3C$w-x2LY>Gn689?2pSRTzK8@!a zoyI8?)QU*2_3r{5ITWT~F0PBGhHiTVNhMVk*V?xQ)s5=~<^cyT;{fs0F+--Ycm`=e z*bYK1?X^OoKuf;nI931KxT^X3u(n6&*0?MfYAi1ziuDxswHL%nS6avOBg8VcBhk$8 z7P~=XbWgM0Oh_vBx}odn*LX^tb+|obbhV2SxGcyX_GN#_w%>#p6enU>!bm*T2NLup zyzP;~%F}s3pZj0ZGOD%LN%lWJ!v6&vcyLyf{Asv58?ZMXTb6sGW>`8vI3J7Z@zf?A zO;PV3)m>q!){KTc*V$*gVpNj_sm6V@-g+Ey*X9%>EtnJ_rICa?i&Rr=OXqb%8@LSyI!^_!TQvnYP4F8!!0 zJjIP*AqiP-H2o2X(2ls7DpFP{#pWr>6oDOF;L4M2G(@ZapYJReaSM;bsiqw8mYT+g^amlNX$If5&b z`I?*9!7#HFRj$YFh6AS&PMU%H zk7|b8#+{``n8l3@glsXYMzjmGc!g&+@M|Gw^Q^zcHHVp->s=0O1U=Z}BFqySaS85U;* zy+!2(=(*;-d_U6O_h1FeOBN7@Ga7~)35~iN;1igJ@c8X zo#rkmR=M}=;NG&^N!O=hk)CGs!LEIwO;mGmwfhU(X;|re&lXj8qh)o1;~ZUqRjNH9tp3cuDg73SH7zBfF1AJ zEBEvg*}T$E&s@L#Izp2e@BmFVGFfI7ih5T0pqBXqz<$b)1j|IGrRTJ^VV_1ptM_yf^P^?6J@cHQ^{+aAZPx3 z^Jaxb)2i#U&jj-`Vb#HH1k6G1L_`_>m5oUro%$(r=iWyu1a2k*B=W)=6DR)9**4lb z0p=m-)X!NC>3v&l-;kS4#0Y1E;8g`SCVaO&c%>Vv=Rj-2ft(uo`;7wtrn%`yRbm4j zB!0Za?QnjEww!6o6;@x`8yZJrx7%8VtDhVddY$QgBZWnp8}jjONjcp#c1B6&_cSZk#fS0WgXJ}d?l0npZayuLyZ#U!=2?-uO4OX0nzC?2cmi^5(%IDu zU{g`gHbdZtfi1G%gh0`Uu&A?BjjkMz%tco#qNdNVat@w5b9ZIOWfz@Kl6J}Czap`O zMeysXt_FJl!|_S@U|t#{XD)dv@T&UKNMmDqlezNwM@e_+AnazOpso4iv0K};Qaf(H z&LjK10b_t)X(#Cier|?+)KwUfx8E1EniX91h6nD^u z6@v)v5elx{H9IE+otbStvf#D4Hc9&k5>qac$SkU*cbbL&nmpx$ZqjWjGo?`BBB$+5s13Ta)qEb&-WS#Yf9sgeqjAafd^%k8V~=cqntC}`|G3%N@}KxC>s5B*vws1p;r*2mu>^8H-pdmy~6-aw}i7!V>$=Whd#D6ezm zV3+~0rJfu1PUW9(m$elOV*LC5O415*GnaZM?(%#aKIpR39>4JA2^ zauG7J2)uyxJANWxo%*A?@qDfUU7r!;^cl{;IToF3vNB8mavPK>>vl{<6NOuw1To3V zkoBj5tAUeYpu@G!*@&uj>k6L2-FmbB9OH0QN5?=d2foQul4u;lS@bQ}8oMSV$%S0M z1aJ?)>|VDuy+uJ&<s?$I>;&6z#zT9{ep`woRug*z`y>tc&HVd1Q3gVPWLkMQ~607G(Woo`I=5-uOSins#=lq&03_ z>y;`8MHam|(KRdOkocQB*#67;Vc-;s^F)ts&O;JkbDya2kET4jJ=MzQT5S@K6o}^= z%;X#F=?U*nH##ik_BD7aKl8SxZ6ZRUtaUukU~5@;mksB^E-Bnr%ACr@kl3F;^!AT6 z=*6P`?gSXg!y@aVaevj*rhN*%mtRbJ3l;v$LpZ(Lk()+`19^xnMm`0`;UA*L+8<9V05Yg7+~ zJAg0NjW6eyYpBfOsrAVNKrNO7Wj&mW$m^3wnSyBQcIp^tu0-_E|Y`3n;#&9`3{=ddZhLv;1%@x+lh&t#< zLRJ2b?m(Fhg#rSpECLx4js$=D^BjzF`1`(fUEo}_?ydg?Dm!uwlns+BIj$Sf_8lCR zk^lQ$ah5LIe#)EUZ=rgTJuqzC6wFXE;nBK={;w4Y=*EVFwC)!~fOi?iK;4D@+_W`e;>n}}e5Xjtl@ZYeRfG_@+W6gmG?(;eD3>qPo(ccgR`(yj|!92Cds3S?|#o8^h^5X9WNKih}cYz&a5u(lkg) zw?03;A-Sy$)Gyd!{LZ0}HicZ#;0_+Fi`GXArb128hF1()<-C{;MTvM>#t|TAJ^-Vy zgEHQu>rv>%#buygqx1V$XhiO;<_{)=XMNyKa1W4v*tqK-6IylsF+jRIXq>FwZ3x#w z{q*iR>$;yh`BP;#<{;=&ga3w^)N$ejLpF>V$NK;3ID0Rpidly?E9PE z29_3FRkO($S6qKjpkGyWta7oZy*DAWxFGHBkA~-Jm3A( zY}WrncmNgvp5Q?zbz1lBt_Svq>kY*TQ2?SfqCL z73KAsq7=l}vdi-AHBe<_g8P-OX@Wn_fLt)nDS9xT*HM+e&(_U6z@luTqck3$pbID^x)@wSa%vrD-z+9P2Y}g=&#S)ujoiSd81Q)SP zkSfL5Q;XvDqv7B_!KUsjj=f!gHpC&cOj>8`IA`j?B@!`99IZM$${mw~doG*$#-j4( z##`?OO=XrjPtK?&P7pg|h)@03``VP2+Gi_y6Ei;P(t+pxVTvp+UqplE^4gTA5r6&B zTMeijPYZxPwVTvm{71)GzjaJ5Ba4dM!B4l?gz2JX6n8)@z@GtFXRi)845f~N9lb>= zRr#fB^J;xIV$^*|pIFpw#7ZmR;S_f}RyK|nmA7KReZ4u2wASXqx0)R0L*f4Mw|~(f zUY6$eya-UHvODoj;yR^{|Iq_NLY=X%`8C7d4);fH z8}EVK^HV&MfCW^tRyL)uW2w9;y)3^lP3aa_Zk!)Lw}QS|Fz@~~k)cU-VtZ#%f1P0)`E|MlIJRZ6 z$Xf@s4(YC8XeX|9(lkj%u7Brqv}0n;;Wj9x!Fi^tvJu#Ss%)Uk@d}G;C-b=89=LgF zrWMaobVK8fhPV&(;n#}!a|sZcz`c(1-R%SE;QENv1h+8o_PGtYffl)OpK)Mz35tX! zPTU9B6`HoQhC^2gP-(DvJkiaR6I^HlRqr?}54SE*@Iys2d%y_Tnzn)o<)9v}t)u}* zEqk&gynODvGDXxOG0VMY{wHykBa3w`w6K1ijVcF5vi_n$Y28k9b9e_ApxxOo$6>&^ zCV#RI>9r9H;&>nlj-l3>Z6+MWF9$vs2J4_n=mF5)=5D{#4XDuoOwohf0E<_sYJWEu zA}OLM)sCMFFU&s%ib(9~1%SpKXp0SI5glRdI1VR01^%c1A$?P7sjAz5IHx}~-*?O3 zSLsq2Pxi=Fb;-z^<$ZGYI!X2bid*ia(`Mgm@;cj58~EkZmv0q`$dfZmNN5_D0(C7s|GG z#)Q{7_UZH#p>P_kT1|4k|4g%AZA!SX z5LG@1d}Bjha^ZApwX}or>uh=Pu&$yA7A;Srt; z`7Z?c@1w*2!wihUNzbYO_6isG{s+bbYG~#D&i{JV50qykJ`6tn|G*pj8HfNHUZlQ3 zcfZ!e^&n6())x-KO>Y}h2{7|UiSZE?GHS%F0?o&ybHVu4qO7?kmF%- z7af?N02y7b+`)C){xcxX_4Q;kW7Ze`-*i8^19~nu8q0|qt~pK%6&cOb%=`E$i#kHGy`ZZrs4 zMTyss>$JIY?80Bso__|{*JD%wfeB;{nqK3EpgHTp^%u|eUVo+}{_)2D5m^4;ZrUiE zJ=X|Lth)_PZvQtm-R%qc8)?-V)1|6wg~$xdJyMK1o_+LJim(D z=i)U3PyXA&`ICh1-|l=9mem33!F42~pmJ(XdjBIX!ENpv8N?*OQk)aTC{rg@vx=gP ziUQ0#oj2mR|LIMBqyKd^2xb09KYw;!=BIFlU&i7b5p3qnT-1NgYVKb*eQxBqCPB^0DscY8 zHk1rLh$T4tE-1PtT)h+*u=tSDp098eaEh-nzT^nXX=@HWK0Z?u;n5|#!?Dx<5}|aq zXbzf6P7WZR4&%X=X_98jmW-MfR-x6f63(HU6i$`gWpHxE6)p>i>H~OSDw$uk9U)_y zi{^Pw*wXJ$CL3}D)Zg}@mv|^q3u4L)j#du;(D6#j>3dhm$YM@)V;b^o8XG{$vC|?V z+q{_esR-3uwN&0{!X^}>Uq`9d$0^x418BHm#`-e4`oQ;Jl&cK_*nO-8wZUjdm$J%L zdX9j6HEc|cGQcNsMAE8{BVr@&uTuFW zE{4Ns(Q{uh(3qTRmK0}=sUP)2$1{ms!n!^CWR3-1BuH?DVah!pxb#${IejZB&0;v| znXPr(ZZE8zDLj}ps9Z)Ec}(303E49yDQ{+Gf?A!MnS1_k?~-{vLX*^$g)|3+h_mJ* zw6$JXxQ<`3>6f0I4s>?ut*S(ScX*$>0!Z62O^LWI0qScbt|1511(xEC4(78vQN0anwrSce)r`^J2@NM_ zT1)dJrx1>4z-Jw| zY)w!C3B0UY^{t&bgBNV8<{iLkG?wHF)S~#b7imo}e9l$3bat`4F)Gl%>Zn!e#0*4l ziWK;y8k{aesnXWAaITmp)a?i5NMjpD(i#&S=SJjmo@43SJy@-8c5P!Y%qS5Wh@Jn^ zf>26eTUW2#4QimL`W+P)XUoyuPKpD5-Y>Y3S4|oHMu4^lN`o|5Qq`%&K22q#eiA@8 z+`u^p368Fy_DIYMYu+^#Q?(#HqaN#}yEO8!1v{sHi$Kj%$23|=_~!M&%RIrhSyL?F z113%q*FxTrHJpy*BjU^@eCC7Q+GKLRFAeYku2*$VEGV(m8+-Klt>yg>B=hKgEe%Fz zJU;U^o&OOgrbcs53M_H3R%g7SPy$OwQ|DZu3FfTfic*_ujP|}zw!;JV=xQlt&=Q`r znv<$*=;MZsFP*HXtn+Q z^02#qd^mjK-SrP?g!a)TEB2aMEJj^@>CE+y(E?OB$>*9|gcH(tW`O|yODsOGOLuBq z{BT}rK;d;z0j>FpXt6*$+b~hZ>ZH|yU)17!dp-Fho&9;gul?`rhL&!p^vI;y9(=|o-KMRne$}ZCVO>r<$ zkB^PNY`ReVrEjp%F3^6UC@LW1zCB-3s=+4OmkydGZ9&E|;<$If=*KwO zx|f$1a;jrD4|g4EjBA@)x?4}lWwGN|QSZ@T;LL*gezz-xz=it(Ja$Q$ra29>Wa;mH zwUci2AZ@>f(I;|*5;xLZO*N%Qt-2u%8t^Q0aCPniY_xBgC0IJwfsz+u&#Y?ge<^U$ zE~nw`j^107v>LbJ(WQV%kEj_x|K88gU>5O8n~7p^K(pOyCH=erI&0+twSIS1O;-h$m)<&%ST_${|FpEjGlbIOLO`^&bVzBi=Gf2~=o zCr*@=ul$h3M-Y8?@9qd0Ep4l=R`AANN^noz7v$Pd^H`F*yHKqt0cx{Gd(ClLd{ztG)Q8l9BV5w)uVzv3gJN z?(N@RG8Zd64`(8BeT|)7fv?sX4_N-JINqKeaqr`% zkxlxGAug0P1 zCEBas$oek$G~H@nmK8nMl&_m&cy{J)O&a#Z(@;EINA(e^>$6vqxzj|x$BF`@r9!hx zZvPKlS)^$I4jY*LNIuP!{gOHJ;5~gNaP?wmoJN~)qe)K3>imsfPlpG7lE${oB9D5) z`8WJmw1lyp=vyzsdg2Bjg+g+sHczUdRQ&fU2xT-qtIOWfl%M^TO)?P;NYX+dx%Q{e zj%eG>cmr+H(-Miyl5Q!|LT*n_&^pMe+jd;6J> zMuD=r6XCzZO;M-unLFO>3I3)%drx zAdLitZT*L+E`C%@pvA}GeKj{r1&^FhT*TiR%vH)2sDwQ564%&I&LGApXfe-UC?04H z9Fs9CypehDJaGx_Js^(lDd;%5O?9bw266!DyQsAC-7L@ADmtc@@ikd?lVGN*vyz9X zJYL7=&A~D6Hj<{msl#urj6U^oxU2E`A5V3^Lm3_2`F*O;;($k47$kmMg0?WK;#F~e>s@((Rgd*{QC56gHByGZvL&9n6`O9&pDx>cLqSmv{c z{qWZ4AQmR3L&5lKId1TrLvX1L@woYcdHBbYNl2?PZv;ecvUqx7^81HsBV6@<%-)=5 zNuw3N?`t=p)^1X=jLqpD@v=NtEC|ttM%jLib*|)_SgT|TMyiLHG(?UyB2aYG-!*Qjzm=!k6tbEAqHnQB@{ONcNuiW@*r6!*d z%|P;Qrpn0BHu3WOy@Vfkme@Sxg^N#>OshLRUg|Rh(?~CNxix$hr)r_MId;}f``RqW zo!fnDRB>{z{oeSTO;3okU#bs=DI_9X-ngr25b$PEIz9_gU(QHE;Je6l)~AL-mp~QD zH@gPK7CXyso3Yp1@|v?Evx10TtHm}mEw}9l%J7$2uO1P4g{zlas1l~M->jvXbFq1D zHL4=YH?s&l;_f+&(xVoomWFkGb0RLD!N*o)obDUjL7%1C@((~?+#_gukKFsJI(d8& zOLeYvcllCvz4@_XI<$+~-}s~QPB`+AQPt{kx7Ew)Ueo4lE0Id3w_>ne$Lhb?WDg-4 z$`$+~aMiSOJJ;GhK!uZvb1J0y7oCc{e|$7xMrsB z9Yx#Z(xtNnBrVTi4PS$$XswkUs^rq3Za%^2b4tRWKSU`aH1-Zd3Z>_wd(OgU^3TkR z9@8I%A{`f1i~VR`wa;HgYm39FW$ui(p@3<1w>4qMwX==yuZtXWD)?%Z_ve8Ke6tLO zy7k5OHKysHUuyekHH(=!I`wD^oH?Z~;vg_|N`ZtcKYMo|r;2q(${_im%A{KC4O?sf z;gPhfGa=vdtx{vfikzV3U)fRGnuICY)@u}8%z@;eMUuBiq`O|P*i*adqH%P|$ zHstaaHDxq#J--OWq_u@C#hb8qJO2u!mY}r}O1RIy)uW|ih@Phv`5YH$+*GdEeI|VS z%rS5&)mtR+yct$=n12{#viuknB5~zW({=k_~ zS2DRAVS_-tr~`QBnzZ|;1e#VwwtpT@6EedbN|!ACG@7cyXE#KjHnv7uWn44WG463s z{_Yt~e(!QbtZFMR`Hq*~d}U1sS50Tu*lbJgWt+6+6!9%krd7FabBP&JV^;&icz#J0 zpB|}Uk|j0FR<2w6k6jct$Tad!_)bunQE}Y###nX|&9#%$b$j@SOvHoFt=&ccp>CH7 z4s}<}C%+FKok=Eq9Z(~W-C@qEC9mYey974EkCNzRCyGbyQZ~QTrZ$0s87@1 zg+sqRwLO)!o;<+zz9Zd?&n8F##%^THJYoJ8ULCwIxN>Ge_00I0r@-M?rJwLXIT*W+ zF^m^H!XE+iGw?sBE?zIaz}r1o;v^F5v;cEvlb2>PwPh-0)_+P{@jLtB9<)%9arI(xxzTqI-e^SZ78mmd(eZ3JYhE8pgx(W>JBiDP&gFmZlT-m(=5aHW{9K z{NHZgKcQ%~hoN!wffChUS+Gf>6PaW7%iBMW5>^_lPke&hT?a9Y!wT~33Cm} zVj9L~^*^)xJ~Z3AJJ?*3Ks3j1TQ;$O-vD06is5A=#Nw()KAyXO&#=&WFfI6KJ{P2G0|GCv| zMhQfV9m}fCsmNQrs#p9ZJaO_F7Kkyoft(z=^O*+qrG;-7049KTz&zpb2(`pVIRV|w zK_?kq8CcS;HZ^z`dd7?K1$w@33fO_{;$>kB{wc2)^rvu(8a%gk7H5|f%df@|LPw07 zHK=z9TOyQO0(UANB~gWT>M)8vv5yVwDfn;+4K2ND9X?@H`yLd9zAG1I z-loI8x8#(+O9#(Hl~%69Si|6drswPp_vd?GCNHp0L5n?SlfNjzOF+V0Wp5}eTHE=w zR3Y`{ku>9lMwd74UQr9;COe=dBfyV~R3b6%l^;y{{%0|nhdRyvtIXol&2?+ypL@x= zjcw~KsJSTlWoXam<0*b!|S`bk6koT=yu9T2Zs8DQIWtM?a;^#ve!!1P-nGNmg{pY4hAqx5ZdP z&{x0Wwp`Kwaj{>ydlU7UDqm8yY=6*`2FMPJS@-ajua#v|0HuB`yd58u818Zy2r55S z!Sg%1f$#hqpv}0PLY7q=RP{^k=&df4q3jg_lE)dsjH zpdGir8hJ5K2-x)Jy;mE+d6mnpCd&`;rY)~V`;3+;*5oY$QABsKW{%>pL0NHskPSAE zUB}Pw8hcG#l6R`-CEVcQYXd@Lv(3Vrn_@+0jInF(htAK3uLTkMRpx5C-_@(|WlVf% zPm!?-?80wl(0F8*xy|a0(j#5w5l-D=P_AgPhR(2}-fkjmPnsR}>+dL_^5=X`5rf^b z5Abq4O*~JuUQ^$Wz)qAlmFN3Nu%U`>iK_FtLH-1p;x4yI5mEfggyB;A83f5i_11CD zp^PYcWWo7^J{I)*&(16j|4V8P7Hb#(F1;bdVDRIa1@cca*$FhLV60y~t015FRxCcV zP2_}dYOj4QZ(W4XTuu@&y#&dvowb{H9*=YN#KZGA5l(cmEYN#Lzpj-ie{bVDm)KDOc!c9_`&VW& zF^84HDtU`bL_i`5+|LZY*tIFo7w}JYl|M04K<~@92Tn}`k$K|;HB`x%J=KQ5O(9b9 ziEFJF5mX{%tVLsQs;pwYb*sFs6`lTHMYf-_H-zjoUt{LY;QB#= zF1lRs3k&52bF8}GM*VGiGdl1hRQc5C`=K8`7IrrX&tWO{mg)Au&r98n z`^tZon^fWobwzYZ`Is|<@g~3M)NrC#g>49vCT$&$v}5hx^#4^`k32NOP@?O;m#hNB z8H23!cKQ7gip0ay*srA1M>Ff+rWK!|DlLRT@)ln|>{H+H+^2~odrllh6?+{ett_sI zuCMCDU_+GU~P(w04UJM7BCspSoisYySH%PAYr@&)(; z?)>AkMlQhv_quP;pLrfF>A^VBz9wnpUc;qJ%u1{4P@o@I#<0-6AhPScFVSP}+=8QJ z;(kd)hf2&{*0^cEOoo=|0Mgrmg?|PEJ>E9)%|k5!e^pc27}WGh(k_bTQ=p!ZVvIG8 zPso#8ioz;{^|+10^!GFr>>l}dwh%ZHSh_OK$CL(ewDt`=YJnHaxd~&)l9wva-cU}r zYXqodvi&OU;0}N|LrqV2xyy9E$rJN;B7~9HGQa%}%S&_hy)j?|YXENkPor;du-jR) zy=f-xzOBi>(8z0(KPBa0M4hBhezo6AvJ-t;`vSnO`6f<_3j3Y#@vOV1IB@~)7{82Z z`ht0eyPh5UdbeS?>MiZlb<}0>d0Ne{BMG@Lp$TUGs@OA^J8bWDB|~J%x-*~48S-Ez zlKpr&s~;>|f-{Iks8gfEPy6K@i#3okGF)9zdh2?NKhlfpX%pl9WL_MY-m`qIB3^s; z=^%%D2F|IY1ff4pOZ{al(`$)KzCkfAt@^(M%O)pGtD zQ$hCAtl%0kse~4^7QfpN(o3D-b0DfhV7=!hlMYGf!KMf@uCG>PH=cM`^60T^P+U#^ zBi6um^5I2-+ZA>$i2IIt{*{c2p@(HgOv_CeQoCQ$`imr~rHGU1@cZ)Ope@^<`zNcX z+N5xr*t<2(+uV6PtUqeD7Jt4mRZS|K1wo5$GbJV0Y@`nrHCs1dUdC9Lj_O9eqvD-+ zUMvZC3zfEdZdAv$a`42C3_@dh-(Ph`UEO?|&)AO;fPPZzv=~2__`wRPx*Ltl+yyO8IfgR3(lTn%ai?Vn7&yYRm{?lEsPtO4gMk3V`Zp-=MtREyV9y1&c}7=KpY{M0;qD6Zo3urr4I+VXnGpx0k+#OCU>o zRaRO9Cep2eYd$b;{sBv}5E4`Cy_>p2gKMJAA?z|`EQ`lpBP64-jNhZ$6idXR1w$i# zIjx2--}ikrUX7gFHH4n2ABY zUHIUNK}@wx-l@dMo^Dm-4UgWarOi*j$W7@Yb2NlvifD_UWrJjP-99Vmo#3u9{0uxD zy{W_9IVZ9NRkh-!4O5F9#rl*9-o%$N0X`SX0~e!rSy)(!yvS4z*EE!)8yJLHq5^T- zX6yZ{#3+oMR(pGLS6s zaAE#K$jKGE2v1z5gj`4r+nkfim>6o`>j-`>t$2V!5$dkrQp0hjW#I!gWDrFm8;D&= zg+)3Da0Ih(S1U3$;zldT(dJERMvY1Qk&*i&2BJ5M#HBD2-#)90L5t!5KKoNUM#h{q z?{dBSM4kxPxmgPwQ6tBBiYi>Uf{|p_2~=Hfa*HpgOSHH(j2r$f zvH;eF9p_b-tDh#5unCRUqVTEKspVewUo|+yyYTJ+5lSo~Ybf|-z~cedF8?{bSE`z# zSPqYw$Bh+1@VmT4*F=}sG_e^>-Vuy{q^nr*_zXEpcZbzFu*XBC4x{sxn$Y@rwI=Hj zuXp#zgrho5(}3JM>E!e`zO6cU(&550!2 zr$-b?4YVaO4IMhLaw1(2F5_b&U@ECsMH~Qm1T&_Dw8Q-9{P(IDWaV0MOR8^I9MTb0 zL@5?Q+YO%hd(P+YAYxF%yBBSr`K`Nbmhn|vv}ld3rA}uRW`uJXYxXGT-3mJZbfjDs zrB1aM_o(9T)azYdM53w7<1TWsiMpDTHlOAGhlZhl)b6xg`r(|P*#45GlKk=t_;`+# zexD1z6s_k*Qsb;x_#|*p7RM&szajF2dqeyBCR&Z>#WP(`@|M%$i{+KUb>8UTT&~u9t z7q3!?i*$D!D-sWFYH_w+<*4S7P`mO%-J~xE6MuZ`I106aJvMfO=J{l_N#o+JK#xFeG66)|jf>A?KFRjhd4+#mMXQY&`NX+~%X837A z-*xKelj&{PY>Gj-8KNK$#mFJYzlguAeWvbA<=X}56FH<1VVwwvEJ|+XVM@O_p~|{@!_bcm08ZTD(Ug0UNm^AryE+ISD1YOC zq8*0Mc2L`Q@V)FGt;pycbAOJZ*y^IS)sYUGK2$%DTyOc%56k9KJ$;PYsdK?fexH0H z&e5|ku`cTIaO?FS6Pp&L4Z!DNBzc2jh?0}FRPx>wZ4YHzNag|~EPOXGFct{%TFWpk z5TQU-z%A5|U$jz|KvkYDb;HC*sUjpnx%VDpE&0=s#rs|?0XRTPOZu#`-QW~#MBBMM zV7dDeFk23Kt@iWN(1n>i^HMNeMs*gi{{ST>?ymv90pR(*QS_YP{y1y|(Z-9n7YCpT zwY{IT_Oqp{-#U-~l4DR5NDLUtahcl}#dI?LBn+cVz=G&B%&;5PJ9gyQU=m{B7J~~s zc4Tnf(8o^26~3_ciO{j$1E2`N7$^8gPv~hMQ-(Qm({awpX?fJ5 zQ$Muziudp~l)xANBsSzV1hik>dy9wrSsOCV;!Yk)LJWC9uf_hpf&+yR(^zfe&=N#? zKu}cxx_cY~=A+F~;qVNdKhkAaA-0ja-SSN&Ng)neSSzc{^XM0~U(h4!U5S=b2c7$0 z%HTR{G2&a(5a!tP#I45_{h7^ts~neeoaG}D;>P%SL6+4V3C>|B)*L_>3zT?N%{)WeFguPHYQ& zByTLIIJKl1B4X1f%W&^j60~^SfMO&qJ86L=5m94HPSUDxn^9E`a7i;91_ncV^u?4G< z$pLh?qp5{=szLb2O3;UZjs7b>$4yTR|I8=I{(TOgPEDw52PZe9+ILMcc`Z1nwzj!? zrtSh}Ve6!*e#hKmZ=78a);Eq1yXYA4dvZ_%)lLg@J!3&yYUy|n>*hugg__?wrF8?W}kue!_7mv#8x(y?@iD&B>^0SO2RX!rcZq)y5w0?pu zET)6tI@puVX1y&2V`7SOl5FeqWU+z?27q0^d^Z%X@)oepxr~VA>b0oNwU#xjyk|JL7l3dn-xYLYYXPMzoy1w&)lV*HL4^ZWN&xNn&P|+1 zli!(6yZ1Xwn8giNW{pqd#?vVChF-O2#o1(@%`mtuL3K2v{bucbmbFvoi3Jqsb}kFo z!@QJP9DR_-uoDRA4e1>yBO!i(%rGWs&BlsGY%v5t+i!3XPw?TC35*GnHw`6rT32m} z)$;vhlHm!~v>&lvksHh#ij#x(JG()hQ8V_&{EvU{5_3g*UCE&nA`5hnV}ETqu8`7y z#4||HCWxk&eLb)Fk+}Q!t7%aNWatRO@%INRqCU04ThJJ5`bseb@X`cJ3%X!88StM# z;vWrq=I~+??Am4W9(9dHOS%d_dpohQ|C^xn{b7yUZ%6xcLhI)bf6NE{5>XKMtyLVK z5a+{~SezLX*9o)(oP#M6DqSsTH_}7=Kn(Aslf^eu^9s3^ms#G zf?2>v;HpLDU1;kXw{u(Y$x6^e;2mj67t8b?kgHbA-+*pE6rH@l;NG%+V|H@icHEf) z%l^pd*ClsqIQYnz_6ba&GP!hii`Tgm3P+7zZrE7WC|k1fmsD0k;s(Q*=`4l>bebs! zP)4P0K4sWqw^h*`otV#%BkQ@hRmr1qim zbp1;TbaE|PzEE2pN{&LgDI`Ah_?yJJqL#TmfZmWKAFH!0y>3feK(YA0e~*ss$Y`PmH7uNYTM*d_NPiu zTQMvjL*M}GZS zU?&9YHzlE^Z+oH}_BKbj=a%VoE1fP`AN&l?c)~&yYVlm7OIW+nRs5m#)eQISdFQIO zh)v77a~AFZVr*8DJ`Cs|d_99tMnYVXZPrYxT@N0p#{MEVf>$dO-a;agDB-12L|32L zb*6E;wz$^UR4&~A@>Kj-lA_9azS+-7C0(LLv5CvHMH4fgtVmSd{S)BiS!z>8CT}DO z+Bp6;WC42F(2?0DPx_iw-*oYr;y0<1!yGRU>;&$(p?lZYr<5O>&(__&WMqRm%zx>4 zVRzfw=_vthWE6%0G73-O$D$mxf#-F|=Q1Cid&PAY_V_!CxB74Jc6}!v{?TOs2Tm8!R<$G3w;~k;3HAX$MK>YoX^l|@&t%kd z@@8c>T+Hoc)OYE?iLmWs1RxFu**MFH$P#TfY`vh#KOgUX$aoOa5vH#=y52EHObGceYn$VSkcRk`S|3|p^SU1~wtwtDpq$zqGS6>bp0e3{_MB8FddMPhDsFD50zJAi@*Qpd zcaSo-S#PD!nBac5(;~>;JYd8gjA{u)4Gz=AbF(D!ZQ{XpfnPBJ9U`{Vo?`=Q3HF6+ zR*i0G5~Fz~KZN&I!p(h`Vb-nsDPJOP@VeDoUmf+5M=4uVAptqxv#yzBPqFp#8J%XXr6*Ug9W{+eU}bu_$1e)vpr{zE z0WRq482_1{)p-1%1IoF?14@9>qSK~tI6ny}!mt0%3fsNs?2gp)bJ+JPPwwTRxU1&w zojVNFd7Vk%mb$EkC^Zi;dmM?O-(QpZT=5>z<+Fn~-x3i{f7Qr*+VJyu0(OPO8j+M{ z$Ec`^(~AdO5P_x|ZC3p{S3T`HBCaNFHMZk)WBU-w)Zo=hzgc_?#eY?-%v2(3)4<+4vE7yS606hRb4>9`dW2r!0 zS({TaIbN4i+SSghvQQ)-2I4~2GRTd!BZ|55FmR+>LGRRbFprpN43U+h%a;bWv!G$r za_H?E%#z_XU%UT8o+{=B>b9CTe_H!?S>{C<~v436H(i4N^Ck~cVC>pkR} zDX?R*zO``(=%?c8kjgJwF*bz30pkF7`Rkx-_Yaa(By*RV8-MFg#-S9(4km4KU&hb!k4tKff(3qmS5_Be^cR~C3EeF>cn`4 zC9w@&K9-jpz{s6>l5Ztkj0FtyM&59DOJYg@h;TlG=120@?SV|l|6DLB>CUg965dA@ z7AI72%twv!KI01tHpJ(6&Q720b0&wW)(#4x{QJn^R_f4JR>Op($y+=^^4eq}jN-=bnd@+SkS@XcK?q5dWTF7wYs;4p z=U2h^a^EM+4&p_gX`|i$q0_A1|Nb3JA>dFwH3m>N^S?aPC3+VC`?UbDVJtIj$-2y* z!@Bp;DrWId3PN)WQT;X;5T`1F{nZxIYIGqVt-rfGnIcDiRGA^Y*bL6|+=D4ypn;bC zNgr_6`iAhL7U_mIr8q zO{tvCnUG;uFFk(w(cnKt z8XIMIie6S>Py|Y-3Eh^l1M({|d5U?P<~7>8tTqvny$N@F5g_K0;TYD|K-` zGrYO+3Hy76tR%doLtE>tq`D13OT>_oZ+TmJ|6C)Y#_ zzfYVbBVF&zpR)hE(sZ{K`On@WE|4wNrh!E^3|rAabV-ar>{P-{YpBX$C?i>BnGc^#J^4DC%}| z{Xz)J0@vz^3?G$({Wl)mm!6h}!KI%9P4E?IX5Ko=xUj#YtUccyGa>G@Zfd|jpwF9k zC~Qe}S$k9{`qA5{pqwIN%q>NxuA3q4y^(3y`% z7RqpPfmYe6a(91T|Cl`pdZhH)O!dUxJ0R8gQEtZjhCqBAVkBc zfAEoE!1spGOny9;_SVNU2V>zalmSXyq>q3(yo$KAk~-z;Kw3a}G~d9tFG=WPwszGT z5D|cXTyts!I=y9{9)~quRM(jJ6+52zx?cj+_3dBPKR5e}9&aDke*Ve^dxSbMc9i1~ ziK4JSd$ZghK}@$(Hxi>?aODPAY&}AayNjb_IQ=SSeM1tEaG%r5M)ZFBzU1MQ)K~5M zsCjV<6Yo3pz?2_pD-)0wR5#ipi{ae_D5K>s#fy+#0-FJ;2kU-KQs}9j+Sg-L|tzS*zR(MhgR!Sw}NU)qzDPeZkW1_rb*ADeILr4wmFO5P0*7P4U z(E)#WZ7WaW6yDIbCD8cW4R;$dX7yP&T>9Rk`oR379A^27)o7-beuw=OdhMQE~75QQK6lHDL z^I(~&x$lxgnX2JFBann!O0TI-&)GRS>xU>b8fS6ODN$UA!ni$drrnsK&FHM_$~%Uy zz#q&L#$^-6bZUv6VV;#+`KK;PA60qBn|=3ld~9a_IQY>2mQ3JiCw0SR@asZxi1z2N zFbv(0WCDSey8iDh@FQ4S=Bnz7LCXzq{GUO*mSII%TxiU+Wn*4xLDU;DarUmzL)0R0mVgG4$qXx&Y-4NGRDdnFwSAe}R#ke`Cy3=xW z==QjY%yJ&Df`yy|-ny3N9*g?&y00bqYDdGNGdfs?W{J|47)sQd{cqzlJ0r@kb{xSJ z+VD(7&uG0ICs&#bsO+%JYgp6=^sBnR6oL1ANfvW9Mp;1OqeC^W72TR($Yxn%RCPWRf8H$&VQ?ZI#jdoH z|B`Yy>ws&Qus#`DWM~CiyFKa1`hvM7;zTkba;!v9Piob*aW545SfSI_>8GS=6X*WD zhOPr^pG6aZlTjn+zgdCQQKQ$gmzNL+_m#G$RS))noRg z0m$w+&UxXHSzoexyFqVLsJJ9VtYS=VV%bPMw^sFO?&=;RcJfK@baSbl$#JSID1756*hYGv>0q}TS zAT87Y2Y&;;-eN^b!~Fo%338?Ur&K3YIRr(~XF0W}ZstQbiZmL%oA=bR3q?Lww;!p1 z`RJ<5-YadJ!(quN1$KUyk2bIjjuwDYnY!O)Fxa_h5S6Uo?8TmLr)X7d-86#hax951 zx$8-*wS|5lcMovG!p>+6Ox>{qG^luU0<6T*Q_UVdPoRU_KshB9?Zn4cR&su;3iT{0 zlm8N7VWT~GbeV4t0HEfCc643zS?Cw3A1@>2-FyxuWq`=O5d+rR<*ErwhuMm-Sy!5M z^jr}5YW3=;+Yq+H3Y&J1iAO%|7YXLBr$?pAJDO*{Q*)Y zDz?_N&j{d6w<&)d$r3hd6PHf8{ka;8Y3D=9zN0&PK}s^d^` zTJC7CE|i3AU=eJMtdmCyITXFTnk+tEs<&XcX;L~!YRqYMIJyDarif3Z#{L9Lx%_lk ze#C}6!PjrNPtOjQZ>xa4iu>2=;%fSpLRXx+3_u&}QNq>Z{|Sn5`O;8I9RJ#YG9I@W z(CoLft1xBy*-aw(|99+YK{55X4f>2_4?blDqTRCbc&NBGGk)iSE`AykTb^p^RH z{m*}QzJhEF?v1Ff#=3j|`OAtODLP_F3h~yAjfa>obOKEBSxiIf|bX;Qb@4miU&%D8Eq4M-jq!iYqW>JZ-! zqe8`d3XN#@uYAe%*`m=zld$7df>!6U&Zo<(E_M;*%^j`B>2Hlpso-(}VNm&S>915? z2=9b2z#3pW^aCF;Ai<>D@uBT6SoHeL9eL2>BT(Arhvm07g)SdP6d#AKY@-*oFy@Q# zm9Yd}k*fRc5{cNl4*ZO4`dC?jh< zYWiFG8B_Bs(eU~qK7JQ>-SU4%;CMc6SLD#0d$Oe_0Dpt=w{X7Xk>zcq>8ifu6b_l^ z4K+IO7R9sCV&qKXq*r&>g$#6}kVpZIQZ zl^Cf$LL<%SE8LTPA5pH=swNz@RmHJcqNuW$o;*6twUO6FZE&+V27oL*$Z@~NL#dV@c{aQTc|ho9K# zF%rQ|Wdzyi;CPak<;3|LN`J$6U4DLhU>?P(Jbn=ZuYJo4zZS0Kt5r`@bleSnMLgS) zC`P@xBI?xRk;m71dNO=nUY}NuW@ilLsJxgarg}Hk0okbQK9jgPuY7iW6v%Q%Sh#5Q zJ!BO%m9=6j;pl?bKVhli2H5*(c<)$!Ma2y{+Xz_ONV;m`Sx%vzE)LVM7zt^O?Hdqs z{iW-+pYgPXpa;W*nMx`l+2k)JX|L+W%ipLt9nueGQQMz~pNq~m0qrt0_IW;~{BJ4V z*snx$Yks=fD<79qQZ_*z6+5!tzukl939?bIQxa0pwWE^1V>OcCOSDWOrx3JL{#}mu zj-Gc4+qXTvN^>KG6sLQ;xs<|^-OUeZFO3^Mmo(rX&~LF!Ob)u@NJvxpsTaWt)TM(? z-&ew$qg-`e>~}IaEI3iFg!(HV2yFNbNrI+Xb#G9h)tntAwiZvPYfsDp1sLTUW0 zj--^`lEA**`iiC`uQ)d~(^c4!-2QP@wxgGTN{YV8b%5GyT%jeS^7&kJ0K_REleiU7 zPe3cm9HtbP#9Srm+~GKeX@k~1WO;-9rK+@+uK>{HnwQz&}jEj!`+~vHzKJ-k_$ON2JqYV7GJJ zM*zm80($=E=W9PA32^It|DiJiQ;8#59eaiR+T3(lwQIjtYM)S?FenxU0uzT!R0Rs^ zalMURw;BOc-z66htY zNj+qELt1L#%rba%?3QAHuhbF-gn|mHUeCJAjyQ_+qUE;;m+ZQ=S=wE$)hUJ zx+?MldZ(Vg)P_Y_lg+MhJV9k)#{Mg$`N%*$M(-A#vh2uRy4k} z8>kLk2Qa}5cwSiJ0+4!Il0`D~ViCPUM7w}P!11NBW6%|3JhMc;{3?wtr)x$CQigaj zJm}k}ZT>?O-Af4n2xc-sIkgyullO*Rp3%L*5>;FJ?OG8}ZaWsH!mcsDo5|7Jl`6;MOmFk^vNjZz1*&RPwOpN2e zR7gu$f*iLv(8g1Aobu^@p-bQb)H=-Trav zN=;*V20ps)r)IFfM$TS5(~YTJr}|;YF34Iv(Dv4QW!nfT5b}?(C_&-MB?K;Ce=`yv zYtAATbun^7NsN-J)_?~KPAjG(p-Rjco867_4V%EmsCKdxXj9N@W?`>1EZK`+-vk`>4Ha^R@=Yk zuvz$OkS zcgK3Xw$x@^DR$!q|EanOL{#RMiyO$7LB?W)`h_j_(f;dKOq8H;_PgsD~s(=7cGJ^O~wpZF}8zTC4nSy(R5TLg7m)QeIva7cL2U9fiycm;A~Czj?LVD=-ux5N7(5^Zoi))m%%Q6e=W zexH_C<{Y%OnyR)w?SmJD*8+byE4CbCoSdd!89eBpFP>4XYAoJ}pY{?+BiDT752oJU3>N92984O= z{25#Uw|$OSfFsH#^*f7pjXO-OPcBQP{ZofGxuiVH{MU4#9RKw824y>r8z4^Bbb($%4rZ z!L<8?tN3zQW?SruK;e8^(2BzO+fei0G@vI9pKj}49ML9I6}4? z9d~@XPKHwK44*-SYBHf6=j(qtGskWgwq=J&8}&_$r@zrad7(?I4fsAw!&1z=jAywX){KN6J*lB1Tl6tskbq;uykZ*?qW2yuOUujDdw z8I-CG*GeDq8zNtSt5aa)oO=Iyj_0^<5|2~V8sD0JixTNHta=-Qaz^;)$i3xOl(wTZ z4rO?L!E}pTbe4tcRK|LdsJ#~zI@#wh;RGVe=Cz27z(X@lM9+Da`;4(Qj9i zW°aET|(`+jAW0Dhqu7$zVXSzan|#i{5T;8#A-Sv0j2M5gG@Q}@(wC2=~-RPwUe z6o4GHDeK)cA!Cl_DWQ^AC%K8A!L=k|C7I;q6%OXH2B$ZMP)o2Rlj~yyNEWDKZZlG} zS4! zP+gSO=UqTiy)A5dS3a4h1$eH7>M4?5;kVZVeYBkOawR?*K(bAH?&jJL@N;>h@X_ydsPIS9`&xH}zX z(+ha2zTMruLVEtbUJBT5K%1|`s!f#1Nnl9rliG-vphZfbQho6OjLo}mM<)_UC>h8G z#OO%dRLoNRa>~0aW>QXLjz@m9cN|H*)23F1uDR{NFeD?aDYp_H6U2fwa>CtyJ5c^L zD(a~UK>7Vq$q6(?=&{v(k_(WIu+_;)YyH_o=2f;2Of=Sc));)9}r`N`sM{z4$j#w+1V2V)twL$}7KJ z5u4M4B!Hil9bKsB5|pV9w`wJpL~fBlYBrUM%Qv@nGAiyDD(B1QmXNago%${iO1znj zV(0hRFXg$rpbBUD1muD+r9>03>xHl~``U!cU5R?<4Xx|!!dn$_x^^Q!{K_8Su1Olem^6o?@h-zcD z*aKKEcgF)v-TPP%%;b9MXL}ZxaC${z%rD@x|4=>R5(j{Z^rY`il777Q35D=Z?592oZr4RJpFLYUvGYh_Z@WD9C$-fWY`UwERZ7wiQ=|*;L5=yBn z?~e&i15gW5g46!jd5Yn9WeieGj*=g=4p~llpPcD~mV1WR>2-np@PEcr6gjWxkt<#% z%e``K9>N2eJf+o7Edy!0trd{1)@tm1;hogXZZq6VZ6R=d->tKMgIZ$3whGGb9=~m0 zbwd;R#r(Nc@vRXZ4qcVrwfOs0`A;P%QEV?K$E)WD1s$7LmZ$D`>I;UkpTDfIZPEOR z=4^!XEeDx4Q}~v9Kmc1AVcgbniD}h+s!k8nBom6cg z)FdMNb}gl&b_qXtXH)?B{{Pr}�!lu3Z>JL_tACq{EGf2ndMuj?xqaLAjZ_ zf(5WpL@81uUFp3;C<0O=MFIo}JwSj^6G%vMwof_F{k&s*-#GuypR)%eKQ^0{#ae66 zHP@Wibxn_zXAOubtnXCOJXK#x=X?=#2qpDOY=TDkskWLDsN6Uk36M=W9#bPi@R5A{ z(oA^gBaV|wW~!;c^PA;Z{;MHGvIG5oBIaYY{g(!Y$7iSnBQMvQ+CGt`wWEHYbEc>2 z^Bj^){tGASWPa44fYQan?F&vnGL19SN26|uC;t0%QwDzbu(9?yq5Q(J1*(yo=qoP2S8D5UdwyGxG(bZ^yl5LMunS4f}zxjSCPIh?vNiL&kx zH(7i8AQbmAZIFW{VeiKoco`})ux`?={eFJK+D_2nC3tLxf_7(CLP>aWjBBitx|aSE zK>Q?8GMM-ow)`VfBBiw)(4_ewEk5e7F1?n>t%R!dOyDTlUm31Y z_$BZBwp0VF21(G*S1lSb%a4>DP0;`{6(FLV=C}<&nruqIOJw`vIv#R?V+39|V-*4( z2w|yp%+uxlkMqUH-v~sWc+$L{n5N<0hTEO5ybFjmS(D!7J&3o=R%(&EBzI|?r-Mkh zZ8Y-+#m@QbX}8mYVm#$ZH1p>DmW8Aa<#m^fuVP743Ro1qST)b(?~DSq(lVmEnE`w$ z-R}j4h@Y^jzpv3~)F2@>HvwSx~=DcwvjTyjGpw zTQu~nBIRKHb{}*w8+>}moQl{-c5n_*xm?Cr=GHPYx6(Q+alx)MXj@v%EbQT z4GF1+aM>Q;0^I_zth3Wt$MzbP8temf6&?TYGnnpOiO=@zm^8TVFgR*n=tk3HC;RqDyq%Oo^fS)1*`oN~= z|2jfncsJ8Gna@GXD$n<(KY3@MTS(59HT=NSvFWeW8u0Aha{x8GT*!K9BveIcGJK6P zDp}gJMR5|h{0<=HKiv*w`(=w^li|oZAW<^Kd{!&O0VjUFV<8)yEWjXm)0tjAbKAUr z`!)bykKM8QVsw^P%9G_Up{E6t@~abhQ<839{BsKx*c5xRSj%tfD52LH@m>0%-#QMo z7tgE+jHQ1IzR>_V-EclRK_hi+ASVOHY0}<_6W)_13IW??qrU;O2>iTjv)nxu3yD9Z zR~GGW+4|Tw6D9clrX+T&^)w%^_%Hko5D1TDWox3kBNwp8c?DW_M&b7P^K5pqVN|%p zB32)cU*f^@XE@9j&MpWJ1L55BcYAl^Y-X>ol$@ zHffy5I@S&hbXJ0&ywy58=|1&1Zb|ontghXGRqRS6PoyAQiV)4}&3l*WZMdlJd{qFT z;?nItw|g$r-0qf@EA^W5f1cNWcm6)>2r9LL!Kirtt5;jny^Lxf@m{T5o?R}wYip=h z=ug|+f*fWp*}(cUZiEPb4Njg(xEe%C61zm{lARS=GuJ;F|WC2ZBe zb)piAe-?-ix_!EK;LCH8L){G95b}8BH`{p6-@C_%s>r3^+~Zu)syowe^uC`43O^XfE3R zEE0Qi@To7oIpHo{#LVD{a?9~xX#C>K%(RJ19PTr}3Fp9pr(J=rsHPHji~a(A(ygnS z*|?Bi`|zu#BmTWS%P`=4t;7FL{jB5e;bH?lkL|@bXEKypPr-dJ9q2pkDtV;snprh8 zLS&W}d<#hDzVXO9#?vXug7t3RI}xDPa|wRqd(U~#jaJYoP5#Urk-Usv%JM(|m^Po| zulLZVAyBCVK_4bwO^>-F_=1g5`J1;F_%sE-F_Zl50_PKE74Ex-N1koLEYK z42X^Rtqc`}DtyoIS26c=Om<)Y_AVU-`n3y$54`J?hiD_&UB?QHer13*bUU!~G|>Zq zzK&~>bmC@h91vr|UF6EY2m7S?{<^q;$m2Ot?%~s~2L1Ai;fFVAHNFL}eE;xtxh71< z%s|E2J;K7Dt0)$TI1U2fPnNstU&jP*a9TZX2_4wY*i4buQu=AX6)A_@XvB0>^YevJo*JIfqFBBXOS$dKRqx7wPsl#ig-=Z>&|APnM|Imr>Xvm?|PwLe&EOZ?3p%w3y(T)NBB#xABKJH2DVux zo&ib+)r;Jl96p1NFP^;U1KraE$#nhQGaoL-SLq6=Gli44KBWV zo;eB(R$jdq^aTKd!~~f74(qJJ05jlNMqQ$@+8U+6+z2VX!=uXLuH;7`QWMkHZ295r z3o(}=VLX)_Yk$q73e_tWqNb7WgHFOvpE*`E`Sdki5fonEdUWv8dYL;r@Q`)bZp2%z zYcZrt$%~mSpD}zGS!=q+5hT;agGa7Q(<=ukBBB?&uZ$Q3wM zGi5|P{T(GiOHy&DEjBZi@jBT0e7X^+YcT4RebjZpx&4ZU%^{X_7i6K^AP+-do9}Nw zTbMB6I{t{c24d@(16Nv}mlTs%y{qLUZqDa80QvS^4NzsP06ENG;SFH~U1BsjY-rGT zimyZ~2I0~RmtIPo>E$S*$roX&s1#codZ#WD-;~LFr>(E4!mn;~VCn4TowM-i?keYK z*FO7lJEf0|=jw~AdBa~!?my(Vuoz0$7Z%LCYb&T8BZsETjZcX<%mEPo5AC+h9X^r- zK~`K@zA3mO7`ozliIA|*FDF=}fBFb#9^Ni;@l@UR)#}g(#%xF2W`9-Umzt|Y8~21x z%LU7GKGrjttCJZuZ)?K;D61^JLQ_^5zJ?!6ZeJY(eSDm$VDU#K^H(6JNPUj>* z`I2HFP({79oZ2tXLdN=P}q1LFKfzn>44&d9rYL375{i)Du)91ppZ?`YPs6F1xU?*h=*r!E_~fsc zpOfr`^O4pf2=Cwf4sv%a7Ww0pOe^PFnREgNXt(90erdd*ZdHRbCW|MsnkbjfZfSg` zPTEHTG<2y~u{i!*1|Ob^06*@7ON0xX#xv>U4i~)%p~D&g_At7B@!|xa2G{DZMln{1 zI(&ASr;^@n>tyf&_{CoF5xtcC*OdkR!$3g0Nh-G%^%0)T5-EcLk~{uI4}d}%2{qoC z53X_df^zqLo=^o?!8{YRhR2yAk=WFVg<+Nt5*Lg8s6_V1$5o#M?AvtE#M(tZ4Z$eJ zt?Td6WglEj#8VO2aQz^dZI3zWoV;bu*%&eI?!#GuoSBFb{4JBEGw_YR;X-Ihgt<{1 zOe50h7O=P;hrlNk4ajG7-nqRLjSSmX2pX+=n51}cWz+U{Ir__q(MYV#4VD}ax+l+` zgpLOle2(tojdb<@aX)L&Xzk7?KnkHmofam6yPke>4cy}}5j5F-wYT@{6^GyJua?h$ z6<-xfu!iT1+@U(T9>p1ZhS#{b_^ZMQJs=2HQgxnR7*%Rn7fqflwh}B5Tp>~c zOdLsGJ2t8g>o(tv$)=EvYL2?IZ{w@7$!9a2;yWIytw{OnqwCobk-NN{b6p`_yBg2r zNcMCES!bTSc}<^~Lsfss;CRcgHE)!TDp%fbO=f$yIF>_IFRoRN>b}tXahmNJ$ofx3 z|67)4hUJQesoK*@oMYRmp4@Pj9FgpP+_si}vX=SzXFRFzno^(k5Z}~!Sg4L`>pfkUZLX+kn)U7jB8fE@VNcyA>&tG zf2g-@D?dufA+t&g_6xU;1oW$hEO&A|6Zp?YvTy674tO#%d@kRxQ3`yCYjaO%ce@>P zM7_pDNX33d{p^0s`xNyDDedI*5btbnqVdpa9#?ZOI;Ot^5h&NXBoD~2cE92@4=-0 z*kdMWvsxv*Ax(Sr+C>M_nP-)$8TV=$(jAZ%e1I-)QuMGzNsY1WxYh`P%o2rrLRI}h zGcRwvWEmNIq6-dC=xXj)x!yKHj%hb@!6}@c15snT2Gb-nLdR|Jmrt{#`6X zPYtFkpKE^Suk*c{cRgewli6WSiBWxv<Skb1GM&Jdq;GV6 z?_aLi_g~r+@7UfAcn;}{h#Y5a{uDYslbJa7MVjR|l?UnUBk%GRx+6Mx`pvXVEg$`1IZ*aBz4mEh;vZH}cMref z$h8e6tx=CvYAU);t4wjFM-L20W27Mse?+~S5N(zK=H>fIA0QR!p1f%1ET9hO-iJ_I ztW?!M;-@&@1mn5KAlGT|)`uKP8xSBOcIpY=2FMjl}>680tLR`Ag`c>XY=L;Q5 zS7IV(ls+tSY)*axWM--uf6-{`mXvB}*t^iw@-2Nl&3{tM3aA9;m7>LMM4-RjzxBH2 zzHkzIUY?jjHT%o{%<~5n(@k(k%L95_U)1I>vQ2Gv+TBLv;k%c|#s`YgUms|Fbr<-= zvRplPx1XXePkJQEw5!dTTK2?HCzYtyskWXXapSl5*aLpyvLJcTw#9HVJ=J5Yd7-C| zsTEx^nqbO|n$G|7NT(!G-h2Dw{)4%o*iFX!BP?KMffPeJWZu z(29Z}LfJNh#h4dLMMV{_)^CvQg28Ig$3tbbksLIf%Tc~>a^fOxyOE(>enn!TGhHY1 zKj@>~)#zp})4U_eX3|snQD3r_^n_;$StiY*fpvvQ`lt$gY| z2#jl+lWmD%6-7)HLX~GOCygKSE$|zz_vj`kcZ|$QL8WYEVUk)@_RRP4FWc$YDtY$G zrWw?7*8XU92LL-&gmVkbkuec@l-rWMyaDtMe4zN*F#ydO7+#gl$MLROLiSO{#|E|g zB*sz}wn^e8vL!#dw7$u9O{hccFP)(A2r5xI3e{skr^5vO98kQF9@_y+HvB=z_ClQJ z?ku*g)Hpr1T@fe}Pvna=zGw_k7E=Ljjy8Y0j!FrIRO)B_ZLtn7r3=4MW@B>L?!IhJ zq^Z{r;Xqe@-e>NNldy({f(dX$IJj(t}*@->xeS|?eo zh>-28p*UW&4g+Z9B{Y$Ns@rPNZnsXZ7hZP}U$N0IBkRRYTJz)e4uu^Ruh58SdV9D0 z5|>gWNrq&i6HUf?5vO1qH=#irKRZeyTEZ`fiwBmvoJr zc0Sx9@aA&p*br?qv{IGktwFh4uaf^IXNiu7WIg?WT)}O8P4s<>)B5smd1E|tlYZM5 zcm7_z(YBW)cAR)5dYyXhnpTMOv}orygB8w$Pb9sKYL+C4*9ukF^JpG&zRF&Rn7OEY zu}(=pd0pU;3v^`-L0pTVGMS{BqvHZ7uCApvNwB2p*Y{a$3Qx74R`@df<~$sXJsC7E z!pIKI=G*u#(cGl*30A1B63@#j0x<6wY^`i58@r_AC&tdX@J*uWwaj&&>>q|gj>XTk zDvAz9(J9{~dxV(=uj)gSRloC9d?8(ib+*2?cGy|?=&an`>{`I5^Kgn%6};TGrEVs& zK{t7Uka}pa<@_}Wt#rlrIA6A!J4^SP1jwYffR?b}QxH#G`1q|UD@sW6uPZ2fQ63Lm zk=X7D;&bI&^|7yJy4)gw^2SqAm;A@0t9@o7j=Ec!=#$lCQ=4(Ha4i4UqvCLLBlt%- z?81}I=|rM0n|(WMU&kb_*Ktb@9W6lp^8SDtP_X^r4~_SYzn$%qx4{>45k6e6b3{C6 zS_oX+G5qO{l99d_qS*t4V~wMR&Cd7T4e;D1#0_Hw?Wgs9?scoYWELDD=7a5EyXHX& z@zH!A{&i4z2AF9`S0FLfW@V^}p*j%s=8jF9{&i6cAzhO<1HF|#@}Y#-)5CH2i1zGY z8}M_rmB(@SbRzA5KAYwRP`1OA-mOAD{LyxDAokFBdX7x#-`$A*&~!D_zE=6A;JR(+OQ8aPp8{42I8 zq7eH|rGfA7?$Q)*XiZivV?kuL@PFXpCMZ=Bs6F4oeG;n@8y7bxtjs^9H61x zf#m2h*wtY+djhLA^Nfsyp);H1A6Ptix+^K|o74Jp>WgK!U0bMJ?qhWF74A^|YLx+$ z3-dmWvLub3Ty*$SkNFMxqE#Si{fBKzC9ynaZ9wKFNji@TZ%ovovzq3Nr|*7cEz&=f z9!-yw9YnklSP#FFHxZ3F0&C#wE^OU=t-@O}ksDj!35x&ur;3Ds!+bh@XYTWSh!59| z^n`or)205s>q73SdHXQ&v}tlyp}X;mp@Wk-0Iu*y)!}7xP56I{2E;y$_{2 zhgQsbpx8-ozhOPYTzkdY5*yn4hW6|Ys-S$lPkn;bgIH@`38zn!{bc1LE?M_WVW~eA)Ve`t2+K#AcVhq< zy>f6DTTnc;E}LR3b}>&-M~x1iO_MbS`FY!{!7=zwu^aB%>7|bx-A8Su{zKlm;55Nl zNv47Yach4Hm- zDgpD(tWz>qbz&S?dzsy76StJ$JfmZ)R-mKQ3v1Xz(Py0_VV8~P{W~tt6`2twacAUe zjzt9Bg+bt6CWjxWh?(RVo1_aKC`07~+B6a~$Sm`v9pXX43wWAER-kA!1A|iEqv*ED zr#16623nB2BXX=%cg(SU@eDnpX5c+THl?3c{ukwV znTZxk)&5V@xPuQ%>O7ESE&rLAr@gw=^I^Y-)#i3&1bIO8f?0&+y+GtGh39`~j_ z)lYrHiuc}ld|q(uTg?oW1-Pmb5)yuQHXAeB9^Op;hq3Iw0T|1lf{a#dTGU24{%aj* ztWOZ^)gC|$|LdX-{D($HpFq-w@YEId-bfEk1$k`E4(&}o?2Tw&VsL15rfqH1K`1%S zj$0=*^2UyTJ2FCi;a6ITO{m}JTU|f*`$+=kP$n3~hOiP>>E}-%+)=GG0Jt1x$ z-y^BD*gntKTLwi-Fg<|~#!^+HeX|c%ZXaph}zS879Q@HFHZgaHnab1c}0L1M4LR{82Gy4E;-^b{!$Tn8AT3=8%Yx{lRv0q zS~!Hhch;Yu#7lbdOnm9-lsISQ+8Z~bsu~zbZz}~36kba~ToF_;ob#N#-YXF%&Bxt~ zZ;Aog{ccRH{~hWSaHpAKo$bF0QD;+(gv-sWb7a3Evg_WN8CLcU?3Q}b{afzr!(y1uNJG& z;~vQ!Wso&4cuV%`*lM{_Sl4V{zyI%EA9B`k*iFrAaKood-(u zwb!;j8zb1bS2vkVc?9fEYi`v+LBZ49=#;`x zveBXWLAx`7IY8Rfz}MKoG+0m1(Q(a=S5i{)_0XT|)6i`*+x}0YPL+U!Eymz zc0&)L4)PY@7Q`QBW9aGr0)NQ>WA7OU_Aa)75KkdPD%YFd@DCSvnSYutd+txEn3 z+&cj7k;h?E@BB?IoX_Pb3BLI#E}n#@`gk*9c(q9@93>>&5Iba%lA7wAoSa;i*WI>K zURsrzOW$Jl+)H*RDQex>V_?gvvjpl}u_4){nQUTyg?*GqbmWMeV}mEHy6v>hh==(> z#EHv{W0~?Xin3HeqJ-{A7{xIIUHnHsu^&>{XR3)G>#$>U+FvQZQRLU8Rn%!4X^wJi zGg*eGQuMho_uf6j0#&gpPF}Tv)l=}Yu$tF`a9EWx(1_=M`uo%xgZhpZlW5Dx8qy_=fzpDL=5Jhwsxj=9$nLsExz;fM!S>$X+7{KwQWzbY1BR) z*QKI`2i^;FZkc$s&kg;oMK*X_`1)3{&Y5Re8t!ou4?H0sNbJNGt#o$_O-;>q&p1C* zGaJj_F2+fI{}_jVfBa zELEX&Mip6EM0J;pq_-n-dzV|l7%wz(;h&@W&(=QPys#=yb@Rf&!9%GtZKdr!^n-49 zDk`U{y;+|-nUp)u4iK0>Z6klSgZsQFm4A%L!>j*Tu4h+IFr$Rt z)k_Iw`D-~WrZO{a?+HL3vfFA4g*<9{ig{YwBUt|a<(D6{pZ;TPz5>hY&x-utACHv& zvBYk}|Esf?A6#YnXK(%g{xIL0k3ZBy8L1w<+Ryy2R`_o2{r@)K-nxIvWOA_EWUL&b z^S5sY`msO#j~177Mc`0nh!@N6Iump2|GfbI`7M8cMit^!S663ZWu^E3*-jk{o~>L{ zWB|ST-<=P~{d0Yc0h9W#E3*F09|Ld}_-FrhsV&`xQ~&h{c=F$OZpQ!Ld<4>ytrY6^ zeS_7Ll#&&6YrD0sw)O{%wuALg$NzQxkmMhzl7cE4Sa z37k9w`Fcj}zh-gfS?GKl>avKKD5#7JvVZtJORvR+8+5S}1x8KZDj2Gx(_?Z_Au)q@ zR&`ek&|m@3N?Rfxyxz9o=cPzKnu2XWP`K>pqX-o4Ix6)G34#jBJ&ZqwpK`5+p0?i% z%7%n_pAvahBkYUDM=apf-JXELv7ZAE->SE0 zI7!6zReauI_}c>fQG2p)`OLiHdCh2%H&*dQlH-fjR&rPfWm(Xmw7R+13nf&!m8oQt z&{9}Gf>?=1%^$FeZ*{)DB_b?Vw>29+*p|f`Aznqq^G+S0D^k@6TL)-(S_g)a0V zFT-1K#(htDN18_RaFn8h@IL$_VjgkJ>IO8Ft-VGm69cl|GDS?7>&l5&;lD;8&I4fe z=Ib*-){WlMmq^Qk-t!fs{zm!W3;66@|6)wR@@YPK_g=VNBg*>Ca`=_zXGs z{>7@fV(?;hs^X}p3YakDcfjB=UOy+}*sEltYc<|&kR=ai2xBP?GdKhp``EMxb(@r- zJeZBvrgnXex^YE>q(wzMS8a8*pEy=izKE`3Q8(u_NwT927Tkte^8KFhRVZThgXYGOVaE$1X z_q@oYv;)s!f~d7v37cy0;`_3>%HLWBkE>7(7&qXB-p9y5`gK%c-z#<)ScX%}EgODw z)=wSEfA9!s6pW6>B2*q{s~}{&QJ%r{^cEF0-(H}&b}nT3>(N4133|xJz?gy|GZ+3o zgBrd(L4q@$N9iiAt>DN%m#D#{VR;eNbZZM$>*Kx7g=u6Jv+(RQC^ENFe+@J3D;_&t zNeh5@T82R5P4U~OK3*gFS$I0eTzm-++Y`fb;v;-!IsT_9BD{YfFo-4Ox6hQqQ)htxFyt2;?ZE&;U7|a@0ON$*`}_M1mRLOjL;_otu+p}T zR_$SPl=2(Zt<__?-eocK(i#Q1^y4{sg64gI*mkAz=dgKgKi$Q-8-L8S0S(H1;k=Cn zcz-BV1}H8;!P3MeAoH{nuM7$2=x0B=g#6->6?SQN%nx^3AmdBO+(prB_mzV^dS>EG zEKJc@)0Y9qQ{PPmC{K{B4i%Spt+``^8}73AJZ4HyCcmTAIHl}cd5h@R=3i@>6TQHmgjr3M zSMugBTx(<{Yg(nq&ER0nOmHP@rvrc=j!@UOfblt$`d8eC<|{!?kE^&9Lwmz;yw^vT zeZ)IsV$(0j`#v~sPqX8_HO*v2n8HS^17?3{-*BBnQ`~Wt#X1F(dB`Evb!+$wrmq0Z zeK))mP+9uH;LL)|3S&HvE@*)+1FL8Wp@uNiY}WD38m!(Q@F_mpZN)uXB0L^-T`I**Cf zW;21U`DgLG@@t5dmU!hd^(knCx)!oycMjYQZ(>MAW_5wc*2;IPLi>22LG8NG&Ir+` zU4hSDmcx&@m1g3hI!7_IhY?mpO_9lC62iaZE+=oigH9t|?;H3LkK7pV<;s zt>-p1$&?-s66%YUg$M5N_rxhjly#q5&A|i6N#X0%L5?yd?>%~C@b0&)N6Kja8Vep( z+~~4!g)zZv4iM71jv4!nv0f>jQVH5GO}K@)cEDjb_uUzi0ShRYyVzG`xK*&&I3{jg zZ^&%adTpM2q_0~$AzX2IIKgvvh28pjNvpmpWV%QcAwM!a*Z{0ug8Er#vl6A<$iODU z8f`8R>g(FH%6g5ke$1d;?FB|LyC_YbcZPA7F^>Q^*11yq2XN@8Ic z0xp8^3fUtOeIBP%FRW}l8)QW2@y|)qyewB=CeZ~+N+QK<+7?kI@Z>k(CojuJ*VFgg zD3uaHi~G{NY2u^%mc$$=GI$$*g*Zc(1>(*a`R(0R_xm-qAxB#Td5MMdu16`VTXIdT z{)1_G6t)?IDmuvI-Qi4*3yWM1Y0Peh1L(_DCPNowv&~Was=LgwvV`vpmq@GW7Fru$ z%M<^LR0fDI?+H)Tc|;cG!3QZDcJB;wWsqA>3~CA>PW8vAm=6cZW_$e)eamMR9qLIL zy9qc71B_Q1#N&@`zgVH_%_z9~a=J!4k>Ul`Gipp}-umeu$EbQO!nCP0u~XQrE~UP{4DJ?&^;X2a z^=Qej4QvUkc#p((sQ{@+B*RmdKId^bXtCUI7tvj{p7ZK<+|&6c?AJQX7;;U_+$*_4 z=Y^gXS&z6Aak0-J3%q^09;lpjH*oEhfps<$k96l4+-Co(cvysV9XMjqig&KUy$OWM znXH2wM`JQw>#554d3B+a((lF;RupkuhlZQ!y_SxCO5mcxdkNx)hxpPhOG>sEIA&^; zxyZtq{cQ?5OidmCmi~AO``a8Hw(x*S%Vi=Y0HoloYg*P&(M|p~gcv$kh2jJmeE$K6 z-ld@+=~_0rXEZjHO}$?n%RC!%|4ub$6ceAlOD84eXU%lx2d~kVO2pN&)f32Fi+yLO zL2}LXbk*XI)Yc*;Cew|6nwnxxCo|6B@?i$SfHt$hW300RZ1JdA#3?u-E!pNg#QlA^ zscxB>tJ{mSpz*e!#OxT1O)5)0SSg&GK0OW{xuO#LU+0HnB?@2w!={Ozec2Ub-MUYc z%Y>~OqVbyA^d;vJ9OK|~3DrELn9!Wkd-u}mn9{ejdPSjbR@g^+um#65ofpZ8;1~@p zvRhJ(*4e()kIBaA3gD)Mr1{FgBG$1gBtmK_qkjAFd#iEwCeeP-U(YHVq)4gr)N(z0 zTHr1l{;ai-F!^(J9q<0EQA^yN;H{@l-d*H(_h`kU9|BImDyj*CVx416Hr3Y8{Bs~U z`R2kxnwkG(jA^|%dE0&4TZv_qBboAj(7#L`OA8rFU&_5g=7_7nGy!IAa^KP&hT%KA z4VyfJm{^;!K9x>P5Q67Do5HA!9vRoFYHAj=w>LU5%lWLl@is6o@?;|BJXH>Z_M|Jg z&zD@;e@%kbK{$+6fb0*k+{ggg=vX8zt37ZW5_bNtWs`aXepDU&A=mGp zx*Ey7lMO-FqHVD1rDnSyYQ(sif8>YW;X$2FUMy`VZfz(5{h=` zchf|K68+k%I<{6ScjxRlU6*eZD64S%kN3JZ_Cj^;&PSL z)#ERPwpit#x(H*84Ds4}m07i4Rz+gDP@OcXZs4JwMZ0ak4s_|X57_SQ2P_o>w(?h5 zt`TN_k6wM4_oh!jZ9|}b($MrJPs_I6B24fN%v~p-tck}a8_y%x<%OOL^@Qf-=6wX} z_Z>b6CzgX%#;l^gy$?t(Fc-1l@6$Bu!6?oIZY;Yux=)0iR*j@LOhv5E4&H))aqbat zY%c>|u+Zs}dqhu1jJQ>{d-I;1gCHoh+>n&ct+t8b4A*GmN%{2_l!eW+vq;BT()xQy z;3Za@;*HS~@R0%opl4M zIHhv0%Mxq0hjwzO^xg^ff1!idl^il5BYxh-x1lr&!z)@n>JNe!iIn*}{@na<%ESNK z`;CA=;2Y-`Ysa@YW||dvvr1s!J(R&Rb4AJtV~1(C$xhg@AoZx>IzK(vk7dPhtvLLn8v>Gq?52nb?9$W zaoUBY3M(mx?xTT$rfulBa*2PtFVf2fVNKzAR*rct<1xKxw1|lyy|5oVa6cHXmIdAw z?V3xJ?k$2}O=gx85*9B6&lnzwm$5iO4dLn~Mf7ArPq_5gi+7tSuJsD1B)cXv-K^;p z$UO@jbVzLIyzJv+Y#z;!2`O}?2GfP(+~?>2jUXpa$D=Zm^S@t;N2#|0x}N{`($tpz z6@Cz}$iuuS`C|O8)UN@%U}zFj(Rd`y8yfM{sFDfy;OTgkv(sv;G)HIYCnq|jqrW$DoRGD~8|2s^?d(&h^04}lO1A;Puxbi(Xl+k-$0T~drty|x`(du;4V27JX6^)q{# z^|b~2mWOIThKKGTo86&$8FjxZ+*``qJdX&XxNq{+*9kEL{@lc-52>WgVwda&7tO|7 zf^Cf|Sz#lS-qTlhYH|R-9&t443%K;`p%HSHHTXCsn9oVN0c^er@fUD5Lvtd;(lckn zL6>WInq7)X?TS~*2g zsD{(EqW#lHucte$opdP<$4J**>VvW|ZG zwwH4~+PwyhN-v{F!p(Tae%z`_hIj?pzcuS_p^z4~fzaN+r|2r8aL~UgIm?~&%q;@Z zUu1`v*v!4so?32&-xxLyw`mSrP7u3%n&5MJC3QR^d+HZQraDQX7t*fS_-2O4?$(`lwApygW`=y!hA20$pcwVnruR- zi6gMlBrsBHYtN)~dc{2(&kk_vxa zt}!{b#+5g`-|OV>BYMz)n7H*^el*4xw6>1nxE0p=rgA{4(0NMF2PPB~uRyyJ;^+RS z8YfW1+fv8&ZJXiA(<6h`rotDhza;M+5u#xXuwWU*F=A0B$5pR*g3p<*s_Q%#WLF{5 zNK)DKNtIIuC3U*sT8+E^a0&(SQ)f#6^^&p58G06M+zh zQu)v8@=(UJUCH!Oy91V*SO^Xl=avq8UXb@*m!*l>$cgxNub^b|Bmi3#m0zEi*uhc9 zNqRiE+?PGIZw7-ZWb8x*w^WM8MR>WrPpT!106PZqH->$3c5Hv9{rlap^$btQRyoKl+(^o@NZ}B$!OTFib+VfKovRsayg*x?lXhl&uh&$api_w6tx3Y z^#z^G=s`D0z^VH5`LixYef#zU zI~-(1j0W8%jVGsiC!s|XU!Bg_17_}WT98#(%_^>mcOUmLVxt0Vex<>w2xOteetAtzCiMZJg4=3 zpMnky3=G0diVpt#dA!mD!h0RVST6l%Z~ANFA7RY*3DA4v+J=}SErAd3F>^#FP}=uV zG)Be~|H(qLB=^BH+AZMt4CWM;_I~jqbo$#>W2|9QdZ*(D7o=*y#&TY1-P%z>3BpFC zx9yIoSe@pe3COOvt`(5kS>sPTIzgV$s?eUk8~+{=p73lx*O&B7gT_9AjNevw5^ctra}w}@tFRP4+0)+Fpdvk|5sod%VM9a9I)kORPklm7)P(mff+Z-YE)`9@CK6q-_hb#6CEo zxXYQMaxV-BPx4}%C%@nc5sP^mg9n?NZo0aoqcC2Xx> z{M`zN>)5<+vR>00dC5ok>V0|}<`#71qV*RX=e+@~*bKKnQ98gmF@xpgdG5WJwpLLM z`Sqd&$G!h4U}3k)`aua7?>)ENz#Vx&f|S86m1+kgt9_0SAPWr7*B60RVqAWI+~3)QHALREl>21e2) z{}S8vrd#Ij)_L91dToVOsv(COn0l!JOMz;wM1JKGK5hc zC(*z}PuNAYl(%=j96}}@M^E{w!=HyRz{Mx+mrw?__A{aM!Ka4*wO8I6I2Z)X*uFF; zkcQeeLG9w|dZ0?7_CR!bZ?u`n$bFOu1PisIRuY+u+VY<@eXD*(i%()d2-n%3@Mi^= z1;kw)85*gR=4=RiSG;72HG0=-yCKY*Hv4O8ia;WKA4MX(U(C$JA;Upol=TZjWaacQ z(BO!&_3aM>;hvDued&5%j2N%-DWC91>2&Zx{an4@_AMKjd4a3a_J@(qZAIF8FhV6m zBDl)HLPdCMuyI#XfD1mOhrFXGDBg-K8Z*yJl}@+duMHeBr?Z64)q18Ts3F!};+j>% zPHUCJ4i~$s%iBBSvlKAsBM(9-G|c)?*-X9x7~w2(DHTr7Tf7fAH9*2x({uFzz(iuQV*B93uMP2Uv2l^2L|zZ zXaljzt%16X`tT#I3?*xS?;a8TOpX&E>^vIsh^a@PqI60N8dRm8mkTX4qn?#!!A^qMSX!nX6KXWK*I z#6#J*8Y zp`l~w32n;f4DYDm52ZPaWKQD#y;E{IIGh!HE6jsXjtA>keE(KnR(7zlYTSwvYD69@rpcNpY!Vt*8dc|Kn+O46&)PLR}rxqI#o zY;p`TAw&gb;XBt^rb&fAr9%e`PbI^4)q1jYLX#J&kP~RB*J`e2O{%kB@!u{d%)yUI z2|Gw6&dMGOYj~H5E?P97AQINn?|=QikysJT1Jm==mE=r%Lu$z~UBD;fj({v&8Iiq3 zFL_19v86n?aYrFLc)JgJD;&#f-74HuNBTm-eV{1qr$-b&NdW7)XULkSNv~zh!Q65A ze5cz-({$EItu0(#S%W7l9_j0J-}uv$qixX(J*0)`>reaG1vGNbsieWcZIu51fECLdi=# zy1;KYEcS;Okg3p|Z^jplElj;l%blm{IMXHp+i%WR^3dBY%mt5C-StzR-e}}ia;XN= z-Kx9=j>%-T&X{jc#)$!yTolob zs`m4;@O4f5Eh`mLMbI^T2|5akj^k4O4ODHnYpC|$J~=>>&8@(B+w+jA`t2EKAi=Su za&iE=g3}}J>n0Bg_wHPquQ|~*A)LqsEyb!T?eeiC8~@1*l%rZ5hlfS?#9@Wy<((Bx zBmT>;>Gs4aThA0WV8viy%hh}eIe9tTwz2T%_Am@KWqC6f4&x50KomalA(fSt>3~0bi_LiT{`=!pn>;#!0Fv{N zg-25Z_6O{=#@#j4k*?ekWLU%eSyn_TBZ1vKs#6kf4|n#JwdO@-$0w(oC0QiD!6Y zd3rhT@zwtc4&fLE%aC5pi3soAJMc2BO4wKNS!rZN+tgv~Uq>g@7&(uD!CTOA;=nvkZCa5!C435#nlZ^4-w#kfOxRl*K?-v*I_fqb7q6q(4|JOyN3N^fnI zv}>bpH{Sx?NyenFk@Od>#4Ag>{}+4j8P)XGb&uMPVxuSsf)D`(6%myV0glpafQU#3 z0Rb^oLrH+ZQR&qRN=XEyN$))s8M0AY=3z$6PrZe+8Q3T2Pw%BZU^hftyZ$#hgKpb66{2H$hZGR}mTZcwJVS2nbN%;Jai^t}4=kpV+OF8D6U<=kAL&-qp=-)GzDY&>lijhphs=9$ zOElCl&wm7)kZV~tAS7E}f=w6kAffjY9J+3iyHe|be{dA!s0qqWA%YG*%X#lMVC;3t#3~tWoQPRIx7` z>j4XcEmiBMpA30Zif2C3GML@B7@+$KT3joNRykz6<7T-P#GD@p_=pi*1l={7v7g=Z zIIG|#E;TyCtS%X=X2CdC2)JW*qGOJrg5NL%9qON+UUtQAN0qGNR-v{hnXKf|agMC! zloQZWWJs1ItqTV1Nio`RO==(W8|tQygBEf-@zSMHvH-tdAS3>`8Iv&-1Xn;z9T1SS zf8IaJw^L&YsI0N7v2UdR$hK1HP+p#o{GK^kL!`^*_J)SCEFgc*>+ZYcZOH99P}q`Z zi7$xEIiu{g5*btBLQIR1w5%i?rL{VkE`TOQ;QF(do<}eV)lZTAT3;=Ih^R-(eC2^* z(5jJ|mv1pNKWk0D?NT!bgpma$`sa)lHK?VoAjgY-dM&peh6L?arWiNek4{?39CfnA zi)nA)PaMHgz~RhmC{=H2c&>5vkGpH^Kz@gPrHwLO20c9lS?jTOP%6govmU0l%WF~j zDVTTA^yhxc%K9%YjY^bVJDI`opM_>|a>$=YuOL0TMtHk1YH!SL8A@cdCF~W-ZMKV59zQjq_9F*sZlf+kHQ(vjV1dA znAP~@@-Ff=V1bf^5Th4BQAWJSK$6UK(sso_0$h1kF|3A~ek@`$ex##=o#Eb~o1|0T zZXNKof&Of9OLV*n9IqU?v&6uaS(A0uGTPQ6&9Ke2xS+hOB^8~xe8%`puG1KY#gN4K zuogsXk1Fp!6B|l@y7JCclLTRKSE*uPk1J1k`#>t9WU1tc6;E9}p@z!RYsw>>Hd4&h zZ|6t)iAuMt8R(nX?Q+2DBaBUMO8f-WG@87Yo~~U+>AFVlPun4q z_n%j`iP1Fps!cpk95XC~HI}OiWCQv|Pnrd2*)pF70-@gOdhv7T#tp~yUC9SvYy1QX zAKEyNl;Q`2;N1bUPpgd{4p*U=u3&7WcYMd9MMyQ?EG6|u8w?`-;@j%=cVOT}38k^wW07w^HMti*5BxZdd#ZQB{#T3*xBi zH0rXmHc^(Pw69L)KOU$1^k$KvKn)ZFwao+SlQsQbpi)7}l!jaIOvhuDPmWeq(;C* zT0X&!sI&m&-n}7GX0_jsj%RXwAnE9ESViF|6XYCXh;tJvB$W2DcLdG{3 z$);tA3Gy&EqwJN|Gbt+b5f}8DU721x){&e?C~Zh(^{tqbGm`?e7lcSaq1SsU1S9s^g>V9Gt;R2;^HG&Y(&nKYZmzahL zK*c}5{RN5v(_lUQyc3Tw0T7hcuGF%`FZZ~B^nmv5m?UUJjB8&YH@Oe~_~tYcAUdqd zVo3U9j2xT=Bi&5xX}Nc=kZUp>Q7LF4ih9zY9<4p?vpqNx@C=zpPQzy(3owet=%P}g zM8&}(030w*qmEkV~*N>rbDABudUh`rDjW6RIU?-+i-=!%|LBo7@4g=k$+O6_hV$` zuofD2uKbQOPqUs)_y9ubR8%~s`e|ic!K4_zV6mD$^rJV+OXP`z!a<&2Ved)w@HyfRK0Bl{^}dnTD>L6%8Ld^&xApOp9%-wPU55nB;rBgc%ZJVqk8a zcnb-_`wf`^&WKlb7(N&6^cj z(#$o0WX#}jngB@{jN%a<>=Kv0IuW+*EFVZ&l;Xa^U{*sjq)T|uTHU1E#eYgNrg`(t zV;JuTfckrT{BRNtNG*dI6Fso}x??SL87FPn_w37KzQ01(j?ncm-%}Uv4F(ZtK=Ocm zUq(Ocl>f`+jq_0LSls^E(`Cp-d$IE(suKT9x}{kw=AtiMNxTTI)Dsu@KM=%mUpC2B z<&-7SG8mbEyV!?m2^y=jM#;7I4Fvi0$1&j}Gqn>Z5)=apV~njToJv2u|4{d6Fxyx) z!JnGi4tJ&v2T8EJ7bnXs7^9{a=1^OV0XXmW4X~-Q_*Uy&KUe#uo>E=LP#X<&^%#N$ zIh3N})+f}YSIF89k~FCl^DC>4Ym6UcIj1I@;WalbGz+w_*$RaHznStvS>{ishs)dz z>{%((mFwfU5#Moz;GuHCYTv^Akn=W%f$(Agoz+P3H&whme74sa%+)-7w$c4AouNc`cmQN%9DY}-v@iaH8SYI7 zNi~N@XC#t^Sb2Fs)?#y7A7rzh>lUpXw{Ettw*L zG|&3ytRLEEh=7p*3Q=q`!omVIB|Tos97mbnH4_~rx(wQbVMDHPO*xpBXFSKqpz=^B z%Qk^xWwto}<%o7m2(v$6wn9&K%xwqzvyb5O!rc79Od5GQo46Qqr8XD9)iT&v^0H47 z#X$dd)=s&WR$VUWi~##P>~^K9A84%??d@ZM9^AwPnHFFvnk#$4DyUcsgSRT`f!wOt zIs5#|s*tmuD5X@jDs%Z~=G+U>=1tYz)_8Zc;kkC)KL;EiT=Lspqs`U@t0E}xham}q z%WBA8nR-T7;XABe)eQegoh^RvPPG=*6{w!H`(SN;N!laomxz=QTmOyOH#^V-ukVnO zS?Q-58y3B5+FEFgt_p>|EAL(4I5B(_L*yqb8O6_7ad6CHvtdt{X_;Azp8h~ITWV>I zsiq4~^fuHX+H($nliGDV{htr8xO{~g4P>u)d@Cs2&ERA;kCGkl7NPw)W|;4Cwo5kc z9c@&{V%w~WO5glea6ab~kgbZZpbieSQVpDefLYFYeV+grejgWnV3R(7XAn&;JjAR**X#~@C=r=cpqFrN2U{X|uWVnz|l;xsk}%w{`>D2TM$&S8Xi*yRq~g>YO8#NG=Ju?|uQ-i=&5d9O z3c_#c92H8qeGy)B)vDY*p-x*gXG~VI?2c{jjatW{D*_NzWkllvP<(U#>^7s(PYmok z=%r1vrV8uQmqxr7Mk!5muk}wl&t~4Xl!2`lYKa2;HhN0Z%69X&pxocyq#GJ>DFY_G z8W1Stbq!HWG2CV7t7k&(WurML=(nU&NKz)&arq;wSwDdgnc6+Z->u@;@nlT9HlVxc zXROas%AyLQZG2QP z{lPJ`kiB5rK1$F^4-eVz3+lbrJtHIH08kn*D^h|g*KfBxmt)>k5N9|6^{>}by`ArB zX60HjB_(Ul0efjdHhu?n-DP@17E|}+!}Rn{iPZ$LY#E&k+A(l=%o5S#mz!YwfjVuk zC6!r{h9+<+d32f<_^lmOlkbLowj~DSkCcpPPm)cW{6mCZchSiz0On8`w_sn$=Fe)^ z27q8cbeY~-Cbq#$p%uH6ac&SEgR>LGV7&j=>bYf}Z~-s-o0p+sY20LCcPK1s#pP*@ zYYap?o#+qYQMjZ;ImN>$ndH4(5sj8OpHIV zz14ZIy|a39sE_`v>Nzvd=8|b>i;GoDmhFQqD@1`7NF3$uyRen%MbgIn@i%P{V;JMKD0=%fTn?=Y;dJX(`vWsnB*LDq_U4Q_jPKb$lC zS?Vu1w$`Qoe%+9ClgMy}5y%Yw2GnQU-C_-i$t%Pau1l;}&7Ya|as0Zy?KbM!L24+3 zef6^l*>cmFB~j?1RaMKTWU58v&RW22ct)acU zeb-Ls!da?=SllaIOK%2Vwc?8RlzGiE9(C_F>uwKgmJis;o29IR$Uohl9i&-ra*+#_(%%w~^}n!i(LXQ!HPe0==Ejeev5bf&)c z{>#`~ZD48Xa{m1JzYV(oW7VY`1I^E_dDvTAO>L~GsOUeCrwQmTUw%JWn^lo z=>O|4Kas+xHX&VH^3pOg7})#hzuo75-tv?|&`Qnm2YU~)(f6_VepsS`(>q&14Yein zU-#uW_yn1EbGkd#K0D`Y*~HuH)n*TE&ZkX3vH&A@$(VO#_x<{YoxG1wz+5yBfGHGV z(vP(b%&(*6gH>Dd55~G4gBH1nboiA0#?=pRG@H(Y8G^GN&xiGe0oq3S9@kqlD^RRL z#=k7ntr@xrm+)5D?N^{Y4+5!KnZuZ{S)o^Ir$tG_RlH=?jV!uJXDT)3=5bZ9+QV*pLbN<8WGA!F-g|s z?&TWgjTGDvZ5aS_#A45ca>ndZ=1lCVYA;&sVn4iV<>--s^?kL=E^ngHA8i26I!f|? zy>{m6aBu3))jBzg`ZYcon?y;g+V3k%++yZvrm3UM<~U%C`Uc>Z?^`4!vkL&I5A;pW z28FH z5i}IVPn*-z3qk-T(E=G#J<#}mzzgVQ@c=LRX;j(oyZ>f1*-Xt0$l*VHFluz&M{!@% z3XY2ZI8l6Q_+o{?PLY(Eo6?&3hI~Qqt^~|rYd_>|&NNa-Cu*d^{nfu$iW&^?p?UQV zi|M(R@jLyc`MCoHP2lMmt2dymCe01ZVBuu#PRU1OT^oa;3r|w^eVzN${@|*IuCa`U zom&BYX>B!Y5EV`}(tl%S`p6}_!SflI5PQz4^grm8I%ocw z$)BX7<~qVSUK(~0ep#NPyWRfggv>C-)Z4O`POf{g8!=Fyd(_{Q{wrqecCYqgy)Os9 zb)75mr#+w#0R?K=T$3yHP?6j0+&qd7kukw7`YFHj8f8DDxN8k90m#7&sre!Qq!5Ee z!dn>wabrNpoegn4wDE1n$~4eDhQuTlfeDP|L$%|Wh>S%jYO62&&Ygw&X_zHAGTUz$ zm>TKb4S(`wQ+|N7(R}KKb5;(ZhFes4{9I{ph!;|IVn(i~eXSB((EFsq)^e`96juyi zahmVg1mx{fnImIq8AZ>j_;UG-t(`)&eRjU<@=Y_*zBN|4J~emp%3+GZUZ58FfN7&y zf*CMwBMyb~5d>R>*e;R_^vt}MO4HLra`RzB@urpx@1-6dWj@?u_d5qkX$hi`j4aae z)ZOZoNqlGBr1bHp%`Zyeyt%*faO8yPLe+fR3IhaQHv?S`IL56tDK`4sXz6vP5~S4A z6KrYF?KA9iHZ|p|n0dQfUV}ebs0TYgoLv&VgT2tne*esxq4eLW@D%7dGm<=cL= z)^1!rvtyzB!3W14%wPc~7>HPM)22I_#I+UM%lRchh`c>G3+0J>Zwg#|xpaSh_bj!K zT#U_G@42z2N;k?pmbh@}lZhhhv-WmrBPkX3A#%DhW%BFSuSH~HzQMn{F&HvKRWx&Yut!zez_8=yA)P3+?CN;E{I}!^9%vBzxJkGUb@Kd&f2wi51g74 zHGybuNxR0FAr# zSO%a>LSQFx#BbyN+u#J%640RA-erBAHwDwhs5JqlK|CyfujqFCF?Q0^abV3<(9RnE z$;xFXnUU@gZ;}j%e*GVRl-V(aFj}fU?3XTOJ6rVES=)*YfDImE-6e)%b1hvPnsv(# zj|)}$tyKZSqik+`cy17s>JMb!HbCY7VWmtJGuS6j~b##Luky(VZ=$IN?&A{II*j9U9kdghp-KUY>Z)dfsf>B#vF zKiRp$Gpa9TSyNJ9FQR6(Upq2ln3kWyaJx|v;-S|xx_=OT1Z5GUGvu||AQJBq5Q?eO zCoKM~RkNOVQcm(3g|vqDs6#-mV8bK{;b^zi@LL0F+X+Cuo^BOEAGy0ReGZPygZgcB zU6PPM9q{q6`uD~pD_JW8?eGuc1`fDl;2&(dJ6#n1HRj5M%ri4`3I{~FDLv5nvU%BH zZ2rJgIPaBmAnD%ZCSBTgc`{t&fENj>3GW*?8jBpyE^^=K|Dl>%&tI3g6mW zkDka%ITY?@!u#vueiYMoTqlh_)`nYq(>hpg!C>E1KLBockn_d3R`C)l0PZ!gjRjUG4B9V_Q%;}SZ_aA_Jgi%^k6g}h2*&FN_b#Ny49j6ZZ4gqj#yf3Pi*l@}R|K)vUE-)xo0g??xvz4z?+wF~llC|Qp zJs>LAn(T2_^DUVnf?|3Y7$EKoqJ^{Noz_nOw))4nSV3Rsz70ph`GxAdw7j?_S&Mcc z8QIA66@Q$JZ8sRGA1lMumE5%lfx*=n&I^KZc>sag*du8-DDL@%-p)5^0Y|B zJk?dnqdX6F(04%12N>fcmrzWB*;6c*C&NWfcGw`E=LXTZC9{<1={o1R`0fr9%wZXI zsruFeN{1i}*Hdcw0EP1K%<}N?SbiqW%slvaUVAqURN(w}*&>wkG4B-&|9MuS%ySb1 zEH4lS&BvQ!MomRMB(h-(0MMw08guUKWo7zv6U~9x z9LxLeWIAr;<*lIH_pxz|t-kmwvnD67FN*3VI`F9_Bwm+4qwW9+6O=q*_!v{?@pFs) zv@J-dbQ?p>)OAC<%p-=*U=x>C{v|LwFH6lTCD;4PiUc6~%Y?4?SHUrQ<-jg$#vhiu z;ys$eWBEkN5=)>l`s~A*C=Q`OIJ`ffygR`9>Y-zj@3Io+bGU})eF~|z9^J! zh4v9Yh^O9cjW#7rR-kO_+MM;#PSekTL3Q*&MD-C%&&XbD(tbfUo%ipm{cMXeLgvBq z2aY_ff&~ZlX3en_Q5iVH`O2fJHappn>P>eo2!B{Csj9qcmH!F3e4!-KZDBAk+KN%_ z-|Gb%%f;X)KBIb=*&*7Ga)+FNzhIH$#w?3??Skjp+8f6uW-{o)%6<{=qa`kN?F6ik zGOWIej5H*v&5vSS6x4=4!Nn4POe)x(x^ey%x5i#gPR*v!3n^#MUO0Q?mTvuLwj*!P zlM>-t=`LzPr4EI&OC#0vgL1VP5`1)SB>T~{=~2A^G74rjj;isdjJg#q3*j<&rJLY7 zy)ys#GhaM(1uZ%}9u zfd2z`Bp|dh0&Brv+jBTq#er2fUxVff4G+`kQ}f#X;$}r&*GHt)oXd%3O%)>qugM?I zH0h~*&5JM287jCK`@%uSL09yn!+7NS>EfBD*GqcP&3pY=xY02TdkJNT5*ri(p>V5= z$vX`{$nXEu2p{aBH{GyqTOH`;{74mP8QScQ*3*)*$0_SgT)DZ?S~%Gtx>y}Np4!|T zbu3HJZFaePd8y!ye&dc`TZuwh$6?_Pu8G607|5hdof?no9Pw^GNA#M%qOBMAI|HlM zmYERje#@xw@x$Hzi;-J9p`=%%K2ko0@MtBt4*DSrr{a5)ghr*jRvV9ynrsegWW5Tp zx#dyQbUNgrIMU?RCUt7tYd#>S3%69u2;#{m_N8@m%+MhmDyLLhxxUcepTGy0*KLwI z#bUHv86|lqOHNy(b+~hfJfleO)d{pSuFTvaFH3pc7kINDl=ay+e|9gPuQ2Sz>=CDZ zXFHSybc#@&c~+Io5^m}l_1{}&T~%{GUPQy<(;ffg=MhHsDD{lgboG+!NxYD2C<6Wa z)%v@q@K%jx2bxBF*ERLVA1zh|c4Pccsk{>oBeDHRZH|l?twJhD3B2I?zWff)60H&E zRMSI41=&xn^!U$5Tein|h6~M<9MWgoyOVaqRa?p$*QDyTy+!My$!I|jK|fQ|>oN?h zuX=y4n-BCkAcou4aubxBn4G<41-T8?zRYq+*2S6WHCOyf*2#R13y#^hiQl)no>Xmf z!L5gC+)|w2eoHIV2Y0nadTUZoycv0;VtQd|w$f^Lr{ojdRuNI?kgS$X$T=z)pz10_ z^=rY6c-qnAlcxp(Z)|wuUSurfAasp)&S{X6x4qTQoMyC`%!b4W<(>>CNo^}Zhs)sB z{>mW@A+E=eJH0;xcnM1b6=>rS&_^7(8t}ZOOlMT=IkbEqH(vHXmzs=DHr}Ngq1*3;JFR_e!DOT3=+EijFHHF3 zHpITjknX{wHzZt-9hTxf#$l?l5*$tn#e~MQ)j*>81=JtzU*qyZZOpn`{~}Ykmyn|2 zldkNiPob9>Hr%!E5YZn4_jIm-m2*Z+7cHcj`&{M`v?ao~v1QVxb(7c1w;BhJcbggU zrN@l^isTV)C8y$OHTNMTQ@uo}q>pVKjkP?x#{8BlcyvzELEw-)#wr5me3;6lKQj!Xh5CW$9!VLs~8K~y6YwC8CO)G(@fBPO4Hc` z%C@bhpRPJSt9gV$M2E0ec(NzE_n1%(Q#qDda6LSid2~}zqV~i* z4(d80?6*p-xX|h@M32o?uCInfhoktf&MxxpQ&F}GOm{F}^8JG1#)TXc> zH($1KYoV7s)7>eo{S6=}ZDyZA$Mh1FcvP8A&!NQ(wTaE21xRvqSM^x?wM1p~kF4Wq zPje9LC5K}WH$Mlc+G#1k*PV_TM!P}nw|VuynLRPDBMW>3eESm;_tcsM~o&LuiQx&JD4I#Q8pYTF3^ zEQD4>OWvfwS;z$T5gU(#xC)1cuwj30{N03!%74|BbjLN5VorRy^AH!_y_u2dHo1BK zSJUr}8Wq#Wmac#&zYbAASSShSjv?z-KamipVlIoIMOB4uyyrHZxE(X_XkHvznHRfb zZHP$m&37X*a~tY4#|X%K;$nIUACzg^W7a#YSKZR|(MO9%Fpyy8`FbbuHc+5qne&aH zCu}u8Hs9PXNUfq7plp&Bt5pS6r~$^6VgbIl?}U;5c-mpVM^4$pugKeYg&AH&%lD>? z@Qw;8L1zTaTD`Xsp4PQolk7rDC$?(0wN`BcC=G9@{OCm$j3B3a3m#p*91QP{=;oxJ zVdo~Wk$k1?6FqTCHoXVd(q7Y79b0fJ?&|v!*qTwQ=g~;?XA7t6pCjK+Ad@?Ik1%1O z2<()vT;yul1hUYRt#}5-BmcZ$)3&o}4Ls*2+U3=xl4}^85Ssp*l$I=NBxUOC`+0Xce2*ElPm#X`EGsmzUSe_3LqLWv^%C#yD`j_UwnpRG;)~ z7ao(-tWopxKz#1XBOc_4vYsp+U3F(kSwxdX54VFH+dDuFOs6nbhVIF=-Pcl7{aZ&&a;alH^l+9|C!YEo$F6QHw?g5Ul~pOzptmf7SSUWq$_A6WSQ zX5VVi@`f*6Fmvc>J!w2uJN2{P)ZQ5>%?p{2pyCT^m`F!2_S5j2t>ss8yyc{w#wsNY z;)ky+$xsvh#E;-N3vFBeg`mM-U8;A{7iZIx+3$ z;&;5P$QO=@%1mT+nhaV=S@IvOQxoxVhFi)`oVEUHOq z{J`2Dm#D%^9xZUB$&QyBR`ZQA+fNrag#Uy!RMl_im=<3t6UYy!nd9;!<_vLaMI~{v z3@flci!s!#_9E@F%Z2OrWZRm>P+dryQa~|KDCb(sU?N&$* z7$N6ydz=*Uzv7{Kk>J!wn>F}%M5#S;pQ4i63qBg2q8uY76#kci6R1nMe zH{AOAFZ?c|AX+4HOC*ynu;DZN)AjLFVw%GSVz^PDk>+b!8Z|lqfZNgBL9b|%-cId( zh?dX+@4$wspAm8N5)*jpp6IjTZbN187{n5sLOP`7eoBfiCy1`Wsq^Bk0*y6rAxG!l z>(z0NJMJG{Y!5i~h)WDq#5>#yT&uF^~^6B*ZZIR@fyQn75y<>eA|kGI_KeA z<+^&`g)O1n>8PG6&5ygeMR!94b4Jl^)(zp%uiZ_HFVx~|f%6NWN?!XQKN?oA;v7Z# zUQXdB@wj@*3HX)!X(ru|x}_T+ba}YtjEK1?mPXgMZPASJdhDc^sBl`HDB1z%fVZ4E zp@pWD92);rTxU&OsvR#a4{1JrOjIkt^i7xVa#!-z<+bmHxvMX9P9R@!Qebj7ej#-R z5Unj`yo3X`s>^HMHV+RF_)r}$5h^tWUW$*-=FwgXYn)`qz{ym9$j0EW3gq~!dX#oF zE29&!qW4Kv*~2&8$_e6GyxxkJuCEeP(Ub{A;;h(CX55Q#_H+4`-PA;@S9&6(>`4&} z@>}85f9!UwVN6)-{%6;iQ!`7fdOo#PQ0gI> zWCFEJNP3AjdjpFiEg7>X_fX&K2-gFguf0{kWn2H262MkWk_tuR5gRA`1>S7+7f#A2 zR!7H8y`{uzQ{3n4C%pzc6s_CK1ofYsOH{U`AlX`KVS2)-ae3>G4JjLysV3j;>S>7v zu=FQT2wnqu5oWCO5IilAZx9TD*Sg;&_+J-7pdqTySA)w=T*rDc^7QWsD%q?apIiH0 zIRo0FGxppwFW}7)FYasidqCfPdL>xxMLb-#ie%s{s7mmbPn1Yy^dM*#D&?A6^qQ|L zNF-7c1$33He^mzbiJz=bsW%sib&x-I?uFY(jDSrjg1}~1r#SVhf}Jz^bjG=glX*hd zAQZs`JROa_wsef9vC1r-eBr>(%|%2G^{&8-HB8z6ajkdJE8tt17GTEaAA`_3iq?ooWeg|gITV2;LAU)=UUQi&DEihRgB)v zt$g#^D?a%S)|PUv1N`X#w4CB?j;FStz<*RjDEMUKrE+?cs@EH#8i@V^By6Lo02dO& zLJEZ(i(o!US{f{-E4{IFk@Aj5ToL)-Pfy}guLHle+V4benJ(gV?j@LP+i|8egR zRKiKJB4~^v{EE|9p@dU^$$39H$7fLABwE2?GpmkvGmU2)|NMGK$K@f!DE}SR0D}+u zI*QMI9o;83U%U!wP^QGN{Rx$itNCp_WyLDLjY=pF#!6vNEjjJVPNp)^FhXh4SInL^!N(fb$$cM+DD|MIWzPr=g++cG|%B0D$i4GgSq8_VNW*Dp{u zsFV8Jc~f-PrSVZ4=j4SugCdkH4u_d2!e@^NJea^p2M{`!0&jLSP}IQnNd zM++)@B?+5~Q>))U?ev5;RmL!AcK2OQ)U`7Y>hjyD&|@!UV9RdNZc@!PzG&Tt^8Wn- z-c`n~4Sk)EgR>x)AXWkE$`ScHt!?zKHT@Xz;`Ww#-CM#@EFPA5T5GZ}h#4D4qg zQ5WMnP;}mDq)OJuFK&y;FPyOI;reJpci;Bb8hO$|k(%h^*xno^&;Z7lwxczBtgOtw zAd^8LJrKQsj`;g6-jaOBgWs&>@AetPqNp|Q_34v)aci9k@^L&lASdp@Gp8oYT^n87 zQio+>gol^W`DW#Ld6I}v29MI5-^>Joh`1H?`JeGwdhZ0DD~l_KtAgv>;izciq`Yzc z=+Bn(bXX(2hpWse-%>DuFKWX7yLl2|0v!qus7-IoFD1T^ZJT7C5ObZ-rFSAC2>fKyp5t=Iwf?p&nO(c z{bMVS22VL!?%r@8+dl)U=fp3%TjNuI%oY1D?s|&z68p3BkNraXVS~#w<2iPJQ}*RQ zgFW=t+JAqEh{(^~mHD5S85;U9=D#0`$nK*3fBU8FNsYPFqQ|}Tp)U-a#BRf%PKxUW8U&HT)=VL6pLTH~SX+@^OT(37JTge#9%%zi_vkZBcEVxX5?}gK zfyZ2rf;NDWF}6t|Wf_G}*gqcK(qvtZl%%040T(AnZD`*SX`AV~^H`HfZDwCfRmv5@ z1@af}lF9x;TkD44eOXpP6|fZ+lBj&@WZBSFBBBvGabtt zlQALef=YqkUmXg`H7lQSeRt4JJc1M&mXno%@mGSb_uGz-c#j=d_Fi&&&DQEOS_(5V z$4|DzaHML-L}QNfUWp{vd~E$OsEIHycd;;!!T7D*=2P}^+E{6qO8tzSYJF3LS5GUr zc2-4x?ewDN7Z;bS*xn314vQ*}FnAw0R^Y-haX#_ULlQ=Lk@RcPj_e4d0^xn%6+16d z2s{#xhP!rbv{*m0jauZ+AwrgMAJqc(zn zAd^o-TnL3g5e2hK)P@=p@e*=fw z8O>{f!Wt^qI zY;WnKzOEb7QURyYzDtz_j=kyOnBM}ar7&VJ(b88Ex;2|R4*un^_ZZ51gQT`|da%Ut zq@~YX+U&(7$r#UwZwj9CeO+J-wcMr75frdJ3F#I>{CaT#5iKBp<;YkRk1(&KDYug6 zJU1p`E0V#fksyHhBpSLirxy?oU;XupquE>ZD$D8q@AvJ8dyKCi>d+(;a={@=I)|OQ ztf6#(dq;i9xdK~O9j!Z~z7);xe9Hwh7EMaXsW;m5GYQ}%d%3e;;&l^-6`8>n6YI0ungmI6y zoS5O(6}yq5cW}hyEf0`&m>!E za|v)|zL!@-e`9+*9P;bcuc%p*?jdcKkJ0P`@@BAS_PCM&n7nK$Bv7O|CQ6jS#yIwup&tr)4;;a{2#=ITp zQFa#k5V+j)sa7R-2qM33rA^3**tiz={dDUYn5(d-42sd7+J?$ zbS+egdE!M+I%zcE~V3|e|NWNkl;YNYGz&3bVG$-+DKOZ&mxk-fMaZyBZ znL_Ym5R>|xdMPq*PWG70gqBOyEZ^K1q4@JIW@VtmMdXEQU9p??+@oB=CJE4t=|K$P zQI@aQLO<8gb@z!zfnzUD#vl*a4~ES~d@~I%S-SJnFkFV?xUiYtMQY;Oy^n4+bguB* z5Cyl#tE=K2iuKaAcljr10?^_#M0HQ{*n-z!{q2vv?zpFi zXexGX@A;Ped%oOGHY<0z#zDE5Vte1{VvljG0I!_+FLzy33Fi3Tj`I`p69`l9k`GAT zNloR-IGMrC^&!`m;{uVQ`w#%g;2+T1yASjw5S)p57Lz^>qHL2|e26Xl;515?ad$0$ zu=L@W&);C|N6Mcy`RuX2A_P0aPNeE=#o39od@Cwa;i6k-p<>Bw`1XawTY7s=QH5`T*;Xb1=2^#)cNCkBN)`a zDf2aur)0z3oH#$nuxYbd_NiAN7L6iG38zIIR*&RGazW?DKHqp{qu_`$x$+@oYffNM z^F3;FI?h(PzvP>-^vZ_R7P)Ln%`dgh1{)?se5#l;NSQk-)l{Pa722YoJfk|=-6vH zR3mXw>hPMzp|ep8KC`w7YGT+jaK_?% zkQvp|R~LTEfep5xO2A}V_S?smoF3QchQ}l0%%nydE=jos`RnwriijFSbZb!bX_MJ;aj1%KGlnhati+;y3g+W5SJ^STkQ1s$HE)t zgH@ieRT<4aH%rEaG=7IV9!~8rnNMUU@Em9lBb9_ywMJ9~X2$aba1uGQInz0REFaMs zO$&_8kOWbs&%2lRO&DiOiA2!*=4xIp4&iX0r}EA-mVwTV^_L?L-{cxkIeZjX`#$7j z=tg+lMLT;EhyF*b^uejy&Y!O|ss(D{Cbj(9j(o~F+<13oycC=~z53RnTRi?IR4+xr z_S=T=DUpxU#B&1dY&@MRKWOsE)QKAr746$s2?J|?aOz+<*-qn2gg;-KP$a)_ep4L; z-snJQ$tmm(iy1?w2XR~VfFIhf zo+y1~Q|@|m4soGa6QTx+Z9n^NR3Q@OZWOo_Pbi`^?&}J;Avg+5&Xf+yxO98(!$@5*Ak|^{KXo{gvM4)1P(aM!VGJCoV7NAaJJN)dgdmdiaiRwe}aB zVhh4G^VN6lhrRDSINYN2d!K#r&3K3$zZ(Zi!#rKjPy4+?UaPwD)vfQcgtG}(iY63k zF&m@R-jt2VFIVYbdLOZJZ;_>j0$9w?xdTxWHLe$4JrCo4%Y%1>z#Bd-ud3BHJlH{& zPs{7+i%~?IgGhmlBD~DB+v=fBMBZP>SYzdXN`d4J=%JcYu2il>E_2RzT=^#=_lwA+ zaO^vmayW(OCeJ>>`r9W)x$bh=bGC94-^yQoIXLy>9r?iL=M##yFCPbHT(vuaJ^kj= z4NIs=R>ZnUp!oPnuK23n$_2Ivwt!Frw$Qn4>#aZdLR-T!br`ez7jKK$Khn5)wy~X8 zKbm#+tguI6*dGNyZZD)j19Q)v6P>!|vickKbzu|i9W5d26?v+GCS7g)!d=I@;GzJR z3?(Wt<6^=NUm1~79~MX_ebtWJ(gTKcKu+^c!?bKv%~&_izL+Bn+$2o+r|z#7mwN&tIJ(K zFd>)l+Xd-dsmMy({Rb$=<-R;=f}?o}ocP};sw`dfT3-@6WFh@^pSS9qGUd`1-k7F) zTbsS>(9&Is`vJF}Sq|IpGkfIDrqOzkygk^5dg1$C6V5u@85}3a9>4xxQveyfT!F8X z_&c=zoErqjil7Z3*7011oZ6iFoHyRye|0tR>H8WFMFYNTM+*-oAGsDp4uvKD7P;L1 zQF#4}bO6^g&LvK}w`bwfjqpIF`Fj(Jbk1hJy@#dD2GJT;2Q&k(Ydl{O<`nKv^9zj$ z)&Jux8t1IFeADe_$@wHzVbyNS=1_Quqv$@d#e4fsrwFeVn7mMIF^lt;c413Kl!)obz7tqu~1R?R5@MYox+7|j3*W`?eDqi51n2Ld`eEzSbRmrXJAxN z@dk>2E^arf50!E&*1kPewqk$FuiTo6x69I@oC;QB!z|rudK#Ug6!Mh>x@UkVn!s>FMf;^!=zgD4TjOT znhld#DiM)MvG;pLdAfoxrz?%NDzQdUGJuar=87Ljd6C4C-fe&F#pD0nzkJ$TIuD=Y z*dBMtSPH;$%$B#`-O`+uwU8GZu{{w(jP2A zLT7yr>$Lum#?0uKuTNjJT(Yu^Qhpj-{J1@$T7?xMFLn8C`%}~$gsoipA4B@p-Lm1I zm0_oF4HKFr1$a<}@z3Af4dzQ!(N$4xWg7skzpIs+eK_o!7QnvA@tGTQ?bQvjRRdtM z{A}6Y6U>R{%j@Aj<_zI57Gsp48NijCs~NOExT3XxlvLlms9GF2%;U&_MH zfCS5wFF&NwP)LkVmF&IoQ+B-%#lmsDz=on&mq>ZMvoh`wEvx!m8$<>Z=F2()nUH3i ztZ4ZnD2K1@a=~WlGj?SQrbtk>gJjc9%f-&^p{|AGis{vzgh7~UWette7~>b?vgKe( zR$KpaQ||znqm9DJ3j~dC{HCsd@zjvWcvg;9?hfz(8w%5-Np~Xf`P%$Da%fshrEfcr z(n&uCLxEzgIu5mfjeYw~xq&9zC=%Yz_??|5rZCcVSU@>nyd#@x~f|(P*6{@`^no9%byF6jsF@vTEnc-Um zrk3L)`V_l%I$e);!O-Ug)W{EprG@f6B6cJ$8L~#tkoxaNAan73B?V4ZS?*Fin;Qi; zb7`+aN^TAFojfnG1T*)j2Q>a}Al~m*`^#K3W@VIwU-M^sgEZbf;j|3;_zpRM57I)s zuab{QB9UI6TvgfH19lzXXV2wW;n14CjnC*`TV^pG4=#AIm7qRpNwq|7TV)3{am6Sv zl7U#nVh=|}?>%F>tR%o?Gq;>hk$saRYHwEYjk=`L%i(9q;e>Hy}K0VgN1>f)SvdK+{Woyssfh zo2U5l7RQdY@|*2cQ! zFi%=U7DVqVvu_2qMEM#zBII84px7lp-ajwml80EO>T>txJSt>f+u6#4ZoQJZ6#43=}W`zp3A_cFXt1M0`0xI>5yuQtBU>fMUu9i=!&uamWKC`QN9E)nki5!;Afkzw>{7E5{^&teJTB? z?nos7?<7TLKT)$@-?U}CiI_8Kk%yg#JC6CnAcpa7*uLm=c^D6@v`m;UJ(_kN0~x4G z8jlYQ&wIO`8<&NDK{-(19__`s&HYb1gXZr_gBx*KiC#&A42=LwgHSXEpJO_#u7#73 zXN24Y-&od7;=aHc;z~{6C&zUHxHpoYMtGL&MCU#YQ6T`Y$7t1($(52PI=VyZA`&;A zK>y|)Eaq*8$2gE^Lj;ilkxbL8opbbY6k@a-Z(P0{Ns&4vsIq~!L4Ot&h)LwZsO)3z zGgHD2aV)rp8p~fmPy|D#5^2$?%8OCe?x<0KomWX#EHd+Uwb%^90eFm2 zEWBT%UwubsRa)`L8^?{$IAYX4%rJB6WaJ$dnESe|3_-*_0IIuniN5(;TK?DD!zr(H zb}K|+k<^>7)r)yVP!gBK?XsXj*f(P8L$_|aq(@7QVc$hxR$;gg5(ZiVhXRY)i16|p zhfcQ)w?6$LqQl^_n9*LIsGUjB&75oa46jE%3}R-4-fQ-#=ML<{`ciJGF%9R)jmo%C zkM0I;hljU$5XxJ{x+hTKM?@DPV-jDQEs+zzDux3S0efqv&WKxXi)82^-1_tC{~th`P#hG^ACevN;kVE1~jKs?Wg z8lL1m`p1_^9_kS~gJxienAUPRSq42`Y*d*e4Nl_ZVz1{M;&~OPix3-FBIfuCvMr3a zBzp(bNLelZ5_Rue-H2^8=DQZ`C6W8^6*o0+*rD~d9-ET$wV23P2Wc^P-m`EDy!gPy zO=InC`pBeMj%fYZN3;C|{;{}nEsVbXwg_yX$&VRiGzhvp-?l1CZdGZ-&Q_D@0}AWx zK;CJN(wAa`hKsk)f1o0oUK8&KIMJdun?>4NFwR#~Z*C7sfTO`8}lG z&Eum)j_}3rDz{Rg`YOF|`BKTAn4b>HC3_jm-VVl!^@_Cn!lSha9wB;OvH|Q4EVG$i zpBJTlMAds-3U~h*n*Y@mtCGO8ko<_+=T-uts8G+gEVMXA;(vy)$A{Q#u!&iyWR_9E z`9nB$SZ86xZaSh$20;5wiiRhBHJ2;Es_kwF7mAnn)v-0uJGZr`xX;OhV#eAJDzsm+ z>?_-mCr4dF2`tZNgNNy7**&;jC`zZ>d;Ikyu~E#^V^LRf_>@>@jmT*G2ot1-s)6+$ zs|L~m!B;7}686|Y)g}#|c-9aTBK@r-TJ)fghLZOkB4~Tsak0_(H^bSIbfW(GI>a#F zzRTJL2Sqp=4Afv1BKD5wW5-|iC7SIMzA4S5CUalq#e*B_LL_qNUZ4%3r!dn2qgGaY3L8Tc*VbHNuUnLrw20{SkP#??pEm+qOpN-*>%v-|rhR>R>Ruf(5XL%oK}o?I z%k@VA$(1Pbg+~FLq`PDH$%-G_z<~-eUpLEi|5OXtSEU#15lwv8?cSL;Z+!S(87aHK zLFHFYhrahyRast=+v$DrywRunW}ZR|Gv+;@9AD4Di@2VIzR5c}_G{1AH+!I2#Q#hS z|H{iWB!H{OkOX%)T+;#-(1dL+snYe3nNCK&k4L?wjY&P_ronFt@8TqwbN6=ezvK-qO+FIyb`@Hp2KQEk7!EJ zPO%(+kg|jfis)GS`89Y|2~t_k-s>-mOOwT)s(tD5S-Jk95CDZsEDD6GsPAZDd$<;F zQBM%fg&SrZ28(7crUz7Byo8bQcH`R2+i?Y2;t!=bI^y?5I2~lW5rnz>7eAf1zvJ)0 zzi58;Y*iVLA{NU5+fkJ63|$`gW&#l_sL6k@#VT`R*w44 z4nw?P-p30hRh>nto^y4X?bkGc7L_`MZu*ONqM4|l1yHG8x$TxC303IxO#Iq*RIc~! z+Ud@2w9!^Ng2P5 zEq}3*T159fSc1rD*!Oj`5PWssoR(sLP^X=v#B2JB%7gddVJ2nE8n07 zr`swhd0gWKQGz()`+?~P3ytoe_n57tQDkRJ`d*o1#db7JM6J^}T+wv#qL@SpTkDf? zL-+}p}ceG5_$YtM;+EHb*>`4N7QDX*D{B`%kh7b!mSV_YNhD35qeNm?j8u091r zZEH;Eo4}E_C6xh&W*U4}#Id3YELFzvDVnq@^xw%Y<$BMNZg@+6EYy2@`J2I?-c$kc}&o)MM zycPMqiQAyyw48|;mdv#xZavvD{&n=w)PyZ11Oev^ne$*AqwDGqq z54_()@EQB}#-rXz-7XK48F(f6PD>Hi^5dlx9Cod$|9U4lSxP{;f z`CdR*bK?VBNRaSVC9&04l$5aLe6a^}DrPieqsDQ&-*jeWa=7=!tSLcgkh9DOX1uJb z;Qg?rj(qagE^rS$wvg)&YixM4kr_O{xP;F0=0|e0XRGao#;kcwEX;E*8?ldA^fvd; zhmz}>Nbx1St{?s5m7$i&m{fi}jWPbX58>bYxqK`X%GbR22+~f5!>-lOnW(2%pl_}w zKtm}hRo>m3&!a$Q>BBs`o@})Ba{2qNsGOfTk&_L>zs1(e8cKwaB=KX`?1NIP3o`RY( zgRTAfIwqiF9|VtvYdstIoJhN6L>(o*uKx2y)RIV!7y=Dn{;a z!!w^xYCzWvi-we2L)Is1J2rQh3Fe;$+=vs}ynpUcUzBPj2B>x|_B7`D_fJpMAgFzD z-Gpv&bY&KOPUKf>mM$$1Os2L!{H*7q9mkBq@CC$tvE(!xxFPD6!;5IH8mKhk1xA)r z0hc6^6h+Cb$e8Zu>+O|`MFeoX;00`sRT1G%r*e4i*&KC&VlHtczmuyh9!1ImlxdihKS&l4ruXdwI|_Y_ztLPsTq@AO62r zfVUA($mc?XUVl@%i*-MBwY;~K&C>Nk_zh`l?Ty0_(Kjh-4K!u~sp4RJ&=5_hp*}oc zdn_xR10j92o%AakwcBB&O)3MU)773Uc43m=ctReur_|h0sG_&L`6Mk?o1rjPTfV@k z!XX#v!rYomb+A-MaJ9#XaxiEmy- z2;1K=u&(m27123PDyJ^u-*AYWzd>A+8s~$Y-^YD7<-Z|;$#SIsj@+|#{A=Fx1`D9G z_P(U{uXk%>FfOYP#-T;jVU@|o|1?cFIsb*u`Ug-1IO3n`{{=P#feKmw24@-nc2(ee zuFroX2SITwzc1zg>zk$l&Fdw%t)XS-&3=sY&!5@OI|q1Yfc|#uz{&aO(!XCV@M*SE zvof>Y$SY>y`w$lTY41Rl1N*HZ=b!!A|MTU9D>)rwZdlY<<&QC7dD;RXjJ6k~Qo55% zLpX4Obf@_KIZWw3iRZUL zXmq9m+d8l_%J+$z*>GG(OcqHv6&s~`>`iJwGzUI{=}Hv9^o&(vmcGx4og75yV+QiN zM-o$inu3>Ov+gocvB^CBe0Kk?>yF`Fw?%u7k}6;b;|}b>qJ*#%n<+%iR8h(WVo=q4 zruDscbY47FDqu3USU;UmGpfx{#5DcF7N51?BgLQ$hPjkE&&nX_=PeD zrd<5EK^OxtM9y#AVe7VLs5^49`EW$qe2lsXHGM*iN?Mmgy)I=4$7cmHsl_m{wQFav zG@b4>l%0L~%4c0_dSsa*xnH!|86txESWb&-O0+{24i>C-sWK|KFWon+wl+x- zKQl8&F_?vY&(b1ci)LbIk2&M)$=H5u*ley(H@tmGa`rCbXwbTs{DVR!qr*y+jQM&j zGYTNep#UyUYxEVnz)416B@w7REi_TnajL~v75GPXpV*uNwYW8!+HS;&qyU{!*pU3P zv)O7jMzmD)>Q4M=jL@Wl_hIqo)K;S2+Szge^A!MzEc0LY;v81aYO~$y@%q4Q!@(o? zzNz3k8#UzmA;2aj`xc!C!L!7!>Bt`fk6E4>5x86}IyN&kOv``|3_DVA5lW}YW`VqdX61jr#VVQjZtk515Z-KdF)IT!LfsB9 zHGQ#3G6F+jAPxn%^=hTZ8;ysdah4t7!QrR0F8e^*>ZFeqnFoT#eN1aea?ChQbTL7|uQ5|(sA{TP*?D8Y#Gp~ZpN7XHr z!cp>>V-np!bB2`M8w?4?OJUPy$M_5nbs_+yuh6`iGn$r4k*7$BFb}f^ zN9BV6Lf7W$=6AiGe60` zLr#pfx+Ad}-7(N(1@aM`-4vweXdC8=bBJZmOA+0<7qU;ZJV@9ey-SnKhd$96gD z^32y+S4ZWej{F`NRxfsxIi9pD`929RP4;7xV%RckGCl@-0bu_Ba-oAPpckwvc97j6 z_k*-Q1bWsje@6PfZ-)Pc+$CuYI&iYAuu~--Xyoo!y<|X(Dgbfqv-`3un2iOWEF6g;7epA(B?I0o^^P_e~;5LHG)ywBk-JYY?_{N<67%XJ{JzJJ8;`$X| z@RIFz(+RQVw>{RzqmC_B+Zjnjcfi5scfKS`1`*RYbIA(vCMr*_pQq}79Y1K>+=!Ad z19XG0%HbVlKQ(fj`U0p%VwNHq_9jbD>H9JjvP4Q>-rz*g=9~b0?G++sqWMxlX-Nm{ zG_p`A>U;XS9tk7@I5tLKYCVt#fJ&VdL;<_&gaDKYHQi%?6&>nL&JYSPpTir-tRS09 zwB}P`(dB*OP;}_t6e7k$kC*kp{@#62VJ8YE&tO}p|g$tx%-v@F|vlJ8;0H#9zJcuI1_%@JRKar=JJ^**L92coZw)n?Z#oC_1 zmdasLvYb8@;S-*3Mo6blH;B3B@~?u|&LtvtmiJvLG-88dK2O(d>tpgpuBzj9xzuA+ zaUTDj%ycthRB)7;Rwn4F7p$VMNH)yrOvYFaGeD{rvl;b#Dq)fem>t;eBN{Ou#O88n zv<>&wVaT&&cwRc0qo`lNF${)=^(&Aa+z&!u9Yke^s`U7YJD6hfro*koZdDz1(w>hmbq6aEF_zlSXIMK_gs z&779mxBaPfWcC>tw@d;gFM}u60QnuoOWOor?~Vn=4#_Bhk!vY;aC96>j_E?5sD`MQ zrAc}z!ip{T4bkyvFxzTxSS&+a(m9*tU&sC3Np=KR6U}YEt)Wy5rbABEd@qYyo)y6? z$_HeG)BFhTT4k`pV?MXp+(HkI(Q{sK%F&PJ9AL`>fGpdO`OwnhAU6ZNhfV3z2H|Pl z8`*;mJAT4Q$iWGa1Y4AFry?WX8_+hlyFP1xdcpi?N~|b;3K1Y27M)eQxmpHWt|+s? z4~V+;JuyGV>H~@-$C}g;p={dE3|0o6bo%fS>2*5O-p$4ey z&BMb2Pf+7*(jkCx-b(V_4~`8ri1ng6R{~l(+;Pj^)I1qzuL|D9;7O4Zd_eT9$8Uq1H>|sDe46U6x65UzZe{_Itad4tRcXp=;5*)oL zpl54APg>kWsNR?^z^6ozTZ@u-v8CA zV%kT>YkXLf*^o0@x&P%A-}3@j&?5e7)*S~=M!C~SQGj3#2wd!?K+nD^oBMLi#AG&* zuh%(G-QARiwgO1sJ1lqI?~ZrI7vy}LK7-!SUIpOCI^%7hFcu3p-Z1LQf7wN0LH?sqHxM?9oM7tI4S zS@R*@tWb?X>rg9&Vv6({$4Jv{%_L#!6H_0qNspOonoZr^Wj5uR8GE!p2=pH%m>#(F zo&OPa*!enazuJ+lvHMx|I?L1Q)oZMNRhFNRzIM!xt;Tw-fOvG$Zs=q|;?csK1M~mk zU9q#?i7W~(oPL|%Q~DPQp&x=`40TdGA#8rd`b&dniGWf-G2rH6jeV-~N&knm_KTnO z7zNOPVpz!)ij8kt_ocG62cB9niet7#%?E}=_lrW!=N3yalLT@yET?PtztjJIt;o04 zf*yv#PX~3MwB%_f{}Lfs0DMt5)b7n9r)$yqI{aV9|F0|fI@$v-%-N??dj;Fto&Z+2 zcC){dRW8x*b_Ebr?~7ws5^4I_NHO2)Y4-e>QGKc$YyFP(VtrBj7)g!{IYyLj@@vle zzv3_5Uk#0O6xa#UXZNMV%5c>)hhq6*b4wQ?yKzZPw4y%v$kQ}o$_CDzEB<1~A zGwCSXyl8XAWRHHo+upWu9bp`6olO;>xR~`hd*+_{~oN z^6Vj?v;FAVPoAr5EoTD9JCAXjgK^rS3>_r|oZ(+jsLt)LN!d#iaI^dJqq_k4YZQQY z+jpc%+M!l{6iKcp-B@*;s`oJ8UhLIVQCzb77^H*nKiUh8W|nTR*z&M>s#*EC$e;~s zcXGIEezdo)m+_+V59NE`UMQ0h_@V3L)zU*BY@;QIA3rVVh-Gu?54>ESAB!7f)amho z9Qz&whHid8V?cd8BQ@V~BHMemZZP`MEL?OenIr0;)Y)}>5HlK+h}p`EBozMDxsSUZ z^9!T?uzOa$+MxS+>R zK5h726g0@|5MX1?xwW-l3)y@iou3ZJ)5s$q%|~xw4Yb>CFbVPj8KE2V%Fm1(JUKtZ`$N8a?VTQ4=ck ztVSiSbYQa)c6-)5a%8p;wHM#tiGG1m@Wb;5Q(38NLE`L7HGzd6JS97 z*zQ0fZl<@4OBT&FxBX78@6H$OxoRJ+@ENj&%wnNzS=$OP87f@?p`beC!oUL~N$=a8 z&$OQwuI}djfG%tq#2c*m_}lL0Nk6_ZOPW1`hp4C&O0-J6mYXF_zXzCMnx`k?&%~Y$I)>wmZ=sdJrUJWQ6`B`9+m`ZH zz~?rWi-mrpfIsysRT-*>;Vlg7T&?KDJL3aQB})pUJ=$eeDX_Q)pH0EZ6O?+l&n@Eo z$YuN$v6O@&vBXp?A8FB*oSKIh<<5+j)2uX?vUU#f zD9fn3U@FRYYox*!YzN0?1U@T-opXy1Ed9yGKG}{(C#h(J>^uOFrQ2))2iZE{jxDtP zsd!*X(~d<}+ouJENE1N5qajIKz*9cURIudTR{A005c2!1U5|X;C+x2mZH$(437R#5 zKMg*32zL5KEk3vF`@ehQe83i3>k@z(WP{8w9iaI%Wp9~z8!_`w|L*q-fp|xF6~8%p zd_PnAFta(S)^$I(%3^d?Stg_Y*YH1q%w@Q_p(3@_u;f)@^Ou5_H?P<(Pk`X8ym}-Tm**ujb~t zuA6j6+9U8tRl3z1te4bpt9?WU{J}X@7>Ue;_#cMZ{m5Nteqy=MW%Fm%@&3zVVz)rqF@2Dtr3R#`L@ze_1|Cr$zaILjNyB4-%xP^fmmU&(L9*u@iPa9?lnbE(CoP@tPi8h6^-iKmxXCFMvtJa|9 zcF{t-S*F@dZIJFnePQ6ftu(pXYX9ha+0|)=&c~Jl7k+;J2r#5QeZ29YV+^37<}{6z zSuDj7g!ibv%_O7a&jbzB%7L(RTZq?*u(Re*uxso3Rv`IUPT-_CHwp+xSqokK$?ecv zyF*XZ4*vi*{~8>e=a9HZ=cUOIiI6H-S0Mnf2&rMK>RAG?h3AL2Ie<9=JlGnw2yB|C zK$ORy0NwqcC0pL#*vZ$hnER;#Rg3CNuD|Uw0I<(9!CkL9J4#z+{Tcl=pFAtN^HZL` z{6YD*wf=qLMpnwX`LO`^TIbXi?YPGzykCqRjP$?WVDPEuFRNc|C>^an2JY6mZ&d&v zBspLCuM=Cq3A~Y+&n>Im|7jPGTDN!RYV)TY=fb^OodF}Udumq-_F zWk*cg9SQA*?=_B7PaEmqH>VJDn zKE%w}?+Wp^+X4KDZ7%5a0QGouwd+EHCh&i_+ESM1^A+%Sedz*BqU4d?n>%IcdO$t} zZeEcAc&5sBkkaawz&;gUvca-HV22jkdOG?EuM!0cjGyN1Cu-WkB~@h>y7+)Z`xxBp z_Pf&i#|DhRPGoWb)sk{#%V0)+H}8p$u3OoIac=>fEme{e6wr-BaCHqztgG3*1bQ;t zBR5_qCg^p__kO)?3)P?0H}i2Te=u&e#`?c}^IGfnRD;(HdxcGOZA&f((fJptW}GT@%t^g1|t4;y}%sC9~62L~JZTM>*U#_oLZxOQRdJpLxu6Dxxkk(0#^ zV-)~P}+NPt8hO!r( z@6MG1$^uRf5409=BcGVuTHMfJF-JzrZ;RL3B?1+~g3cMYA?GMhFAd0v=MC(gK!A~a z&Nq$)^c0hh<$HT!`%apMd%10(Vqbz6e>{+Y`o-m|6eIv{?hR^is8D~k{k>r-VJn^U+#`VKW~$zQ&dvvB^2$rVIL)x$ zec5C7sdAU@4B=uHTOQL$f?8_ibs;ic2;s#NP<9pbMq4&z_<8^Opl6U0{w@q|Q4mon zV~z?T#Uws4jo9w`?6p2F>5iM02Oq6fVtS=#qLgmDc7}*-E6r4Fk1$%wKfI21;m6m> zFe+VB8U+}Es7R3}Mnu5jazCJEaMv>m(=F1>0*H|?U`0jl>1ILX0iWUlZ++wPU{+3x zuOgcUWYqJ!$ zyt+(maN=Y;y=D6&A->tvmZ?_*3^n!k!uLV_1SC&|em3JinKq`0R4o3N8$Slz_`@+2 z;mvcrSKCY(csQ7aYTTb@UIzpMB3`$!ogH*+0ulkh5a?&Ho=W;%mst-}H?In@ED{|+ zCxPS~YHgYr4Y~`mq@e~@FD#)S?Bz);fQAy4IJ7Tnmz#QD?gxqdBukqcGSUoyr902~ z?}rASZf9X4co8IM`F-Ohz+vfL8%-3uwVJjjAqyl3pvR+Y{VCN!VjVCS-82b)wVUh} zil>Oh=K!gihnZpnaFWgB=Hpx73S8g_zIhD^bP-|VlY<=bOfR$yQ~JO&(t=ZzS@;G8sM5Qs=$_jmDp6XfN~f`o%vu} ziRiRs6y=#ChLA~rg3ndAx8l-E8uIB7YdyyW_VcdN(IAA=JG1g4$|XWm)tNu})|Dd$UX zAMYHF7m+qIs9TNJ0Lq|&D!{_V${imlyK{rZXPtnB4hoR``0!lKCLly0gmsZvOrCVf z-sUIMeJj7i+25GVg-MbQW>+>jvaSYXw*-5JlT{*^CK*R2u&LFhZ4KJ~9@`}pX> zg`f)>DvE#DF3YAvMpf`dW+%7%0s#o(Y&~B!X_#LzlavX&wS`mTu0bZu^b?S;2zH6{ zEjGMo4sRVq_P3o_(1&mkm*cnijZU*7k&+<~!4u|ta&M-Ro0Coux&j_%(+|#*V?-YE z+}PJ?Wx}-;>~~tug(?o5zKR4Ax^2!?6y|(y=UD=E31yc5p3(QSq@ww;?t%WJZ3(Hs zUl8%X>KX}6{Bc3ila^3&22`CJwgqVKcAY9z<^sTQnQ32O$Iv%ky|igD6`ks*_BW6H zfi}GJr-#!E-A}rPMK}kHfsTUrHPV6Dkv0gkk1jy@K+Sew*W{0TV^3hfc5=wOVRu#8 zxxnI_TqD~HV#LPk&WeJnT65tJD#P5S^V^$T!V0Z(CEre>B)v)6(&%pt@NNe=tFvz8 zv2M*Sq@x_5-HWnw>Sp*9Zt6?)R?d{V&Y$9?7v|*lb7`jibp25c8MrF0>Im1*5cj2i zz9GzaIMniXa=nfEOUoMF!pthJ-EiG4F2{VIcUIjN0SAa)O}Qnf8h^s!yM@8qb;Nct zvjpiGKU!&T_7rybEnHu5IfCX7Qq5@q&AjCC%}@;8Bdtz6<1~kCA8e!h}X@AflD-P)vLkCHy&k_T)m58S*HUb89fFjvPf#Zb7mHVy@M zCb#GPVP+$@o#c<_i} zf_iy+xqGTzq33>}bB!DR`hm{t`+vCh$#5~n@yTFE=eg^#B{$D-0xxw9Co+meXEF4Nm6BwDR)~zpwD97bU?`a8#UcJ=GR1gfho$-zqN3 z^`zcrGb1N$@n4OE_2Cp&lB9Cj#$FTx1%NjuBZ6&B4JAH$T+*zka6EGWIOLcocWf2y zx*}m9U=?%v^XuiAn$~i$uU?#IS-kQ`+SdYguQ6cK_Be$d+T?9ll{x|(-ok|c-ujI* z+J_%8Gk^R;d#2%*fI&m=3~p1&^9E8-&iR11*8a!S+Lu#Tz1CVgk!J+ejAqZ1u!#!) zg60lDiEt|Tr@~S!tGws4TAZuE@QVStfG+vKQ}6>t5{?x%N%EdTN6bz(7#udlC|R=xNkP;^7g zpjIbE^lNDVrK->@J53oUV!ciyOT1eqE0tlPI0<{DDlz&r$hoC%;~-9YRupu>#R83x zpAH08Wv5p45p>0$c>Vj_7DMICKk~L3ZFM#RJnT67fzgfh?I2LnM@@r94mD0ryb{~@ zpk~m^&#|Q7$ynLvyCM^zZo<;fxZS7?1h^jQ;mjNLVvFqy1ao^0HZwG|Z$4zO*NqXS zn=TkKcm}AxCXUh5E(b^drdq^zX6o z@Q-cR&!R4lho5WP$jyk@P>0yiI)UyiH^X~b{J(fE?Z_FlyDiE)1S=E`)Oi{6?s&IT z_y_Gta-^QB-5kg=Qq(o$pf+Y|F0Ls+l$b5vX%#u7xn$9uBy{r9e1kb${KIjEuZ{xr z7@vyQHJNasj*@N{U*Oc0atYEP^Avo|U9V?NtVrVj6lD6h-h#z`bdtD2a4H! zG?N|iiY1V5nzV(e=K5OQ+EadTsXY^GlA+B^+#O1<8%!QWWZ>_QgX8&#*VUWLNw0dC z=K-?N<|K7eUtiZa4SkHm+3jUAh8F~Gu~-(mB+6*3yV2$ug21AP>Y7tHS=K#GRh%)`;&^^7HUI^y z!QI(Xo0)goK<{Vxqo3z?3tu0iKFKNvMjTDg_yp!MG#*Fkji=Oq@B2)L|x?D%?vpR@6j01bWU}O}H4~chCZit#(oLFik#0px(RV=P^kf~@a z#~BQ1--bik(B1XN(foMhzxPd4xv{GQ#wy?@JU3i#LRRL|s+ZSftv@jc8*80Vub&)I zPwp6Po297=&0lNjd+6_CoPaXL!;>65XD7+R{s@LQlr|Z`tYZm$Gbdg^p)W~7SjMOS9Zs= zFBrMuPqE(j{{?rB5Q3;1u zcD5>9A<&l3gDYiy2v`=`AwInApeXkpN;7#s*_SJ& zKfm||v|36@@bz`epaA6?ld(dx@kAEm?<%x1WhLeMKDupno_rSZ&+_Cg9X6wxFbuts zEF;m?`#Fc2YlG`>?9VBQ%EZqJy#_P0j=)+z7yX@Wx%o0Nl{=6s;}|}rf`cB?O=r2& zTrbpLNw4Q-8{TK4)q3!L#!ybLwu;`P#Vr_0ZdI^R;_xlFO@kJ#L5xp45{-c+jfp31 z&w&2Wh{Q6k_Z;M$I2ZK6nMZUBiw*G^8h2fuS`*s6gac{O%aJ6Z30XDeT-2z89a2Oc z%f&A*=ni?%4`|V($_hDwTazDp@23YuIcu13DOCs%on%06K=y4!#)GN|j@jsbv`rw7 zAiihOR7&sitG5Brq8VFD9@+2e%kgmRWF#c~Ioj)*%U$BBc1iyf*G>QSnPK9Crpi;5 z`qopoLSpsNVc6CUojkN~NCe&$`R#zAUAQ-u^qR@BV_F5#Kr zA!5r{2ScPqrv%7+1Micgu=V8bS(HAHjO2km1{YGYgFLl+Z{ z|Lbq1pMBBVDY8?NFNcR4r*f;0X1vlH@&tyhp@kijy~j|m<6PSfHe9v4lhn(_aj(^N zDTgVUpK&(UH4c=KFzqxA5uBv{5G@GWt_4b&VDK; zyrEGVQn*n8L!hNcM|uiWL3E_d9ot4{K2Gu!s+3N`RL@It>q;yplH~4?M#e1GmxWRU|EqA>LeV+ z4h%gj9<6k(oEyPGD&P?H0VQzN*297wX#*`zZu}+I?ADd&LvA5c3DixK?mCt1%D(onl;kCeX+rE? zr3nL#?lLV2KaatHj@Fm=fm}K4d*j2M2V8!G*Wr%E)-@$S#ATBv}li5nQn;h!H<| z>!+tn@@j~gkiXVy4t_QRJ?1c-7+&3>J*G)!Y&hWf42nHedm-r(fhEG*upu}?3U1n4 z=n5hxPMBgz-h(v_F;28X9~p>rTSX7wRg3TTQGK>^O|05We7MM!UU9%PXRml!|dKHbqt5 z8z5GfxTaU4Z(y2rwt-u1h95-6Lj(}rT0x4VFKTH#HQ!VBE7(HiVl(fIU&qde?hu-C za24GbPlAs&rQ`LdVRMu-BLoD8Q$l*g6;Hp5GkFPFQ2Awy?pdxWO>LdRN%XOch{3MD z)YgOP^@eyH?}n=BN8k(V*h!EETS78c{v>pBK3uj^Lr#l=%YHInQqpd&zYAz=d2ylj z4sW3So9sXX-udDhu~<^MICl6T0??-H&F^(09S^Ol+X8b^theGW%x=JLEw7#IdS;w}_Imr&|w=Rdpj0Rs&dfLhr z0|y4dsRDgHI$`<GGUTLnZm8 zjbit_6G0t51lLa3KI&bo8>;L?#$^e*lU3Ht{@^=~$>O?3I!(Hy7myr9up(a(IbNEA_pjW~gQ_m#UCfl= zr?hp(s!IMO7D_QI2rQ7@D6%J8VaIi5X|O&oYVVpK3z&WO8Gb3-a8l;Quy1|)S96Nx z&pO%m=f+Se*%Um8f@&w>{ZWw(&7Hv!6EH8%Pg>s$%sbi{dgy}pZn zo@aVUaSa_x^ViPWxq79o+fZ~?1F&P<=l0%-I*jH2WADxXp>E&)@vD_mmSibqkIFJx zvJO*-WYF@?k}aemyJ3uNl*mp<2s5Qnmo1c?i3TA?$U02MGS*=X#_n_KeckWt`rO~o z{rLU^_YamaGq2Y;&*MDS=kYw6HhIS9t5&_-mA5<3YM|uVX>#-}+(Nn6f|B!Wxt}|E z-oD&zjF$=ZPiPO0GRA6-9cDi@J>vSK<1!e5%Qk+Qhd5A3TOL5V4Has*wUuSMk$0Y^ z?FQNeU)Z0>srBwhCxNkc4i#wV#a1**uBo9;1fIt>+cro+9{bhD@Ef8J9-DcF_cJHX z;NO+Ta-lfyb1HI4RzM&HTz*`)u6KuCj@Awk#iUM!co5k2vxMr+X)EgRm)5+v4b2^?;K({F zl@r-)R;sdmVM&*yQrkb3x2)PEH6xn4PC(A2wXk_^-gcXLOb@xFqj^M6D94b~ zeoQ6CD%49yq73^m|JSqkQP0CV!#wz!M65!sX^5l;cJz#4*;Kf#6iR#aj2QbEuy)W; z8SW{$015gTx%!eGkQE?xT|ZrlIxgeQ_BAXxoT5rpYdUTxvaLa?%BU)CIU5nT{PiX@ z*VwM-g9cNjL>Pm)MTX9DwmJo4ZC6R9$<*cTHNTdMw?&JV4=4wwJ55Ggkro%?g~r+l z$ZF-Wn6oVWzRTQ{W}hZTG9o%#KjS8pddd#*!eXRCNGcJmajTBiErjhb_p{`dh1;SJOR$L{vdCk?ufNbXNYW~aL~&Nlhf;SxzX+$EpLkO{zZv88F(FxT1Qe{20BnP3!fbqcx9}ni3&oO(C;<_#wKaA`sBuXjW_6#NZ@_(>&x4vy8Y_Bwey|vDuPAAFOa|S6%ZHuFhQ)@iyO$2rIcJ+&==Pc*zo?WJch)g2>HV^hL05r^o6j%5#d1F6I@Z-yP>>LB*NAk+Rmped z%5ZK^rAL&qI)1tkaw2FLrTO;I5nZ*k-p||P|d=P2^!%|{=HUjyTd5Nwk zxYoz2!xvyC0^!`3FE!ls4J#%NIz@Q4% zg3=Mk$&eUClaH0?KOGufC^N6YICG^zsbTraQ@yR=LOzJa{K)$2Sm&-k-v+EL-a1nd zg&v`HLx$hr;wov|hpUvV{HrbtD4VVuGk=UpT9NlXt^K9pykn>Ay1E~~QFD`?Q80$* zR~sw}Mfhyx+OQ+Vnhm{i9A_sEEyr&Lo4Y1gacb2_w^~7|*8;Ipsn>t@$78el8E>6qgVB7g5$GBf-IHQM ztWm6S<85VnlOy@PW9O9iAA(ddl9f*qx$HPK4WZ3sFr{bqCbT&|rT;}j%`1;nx$dIU zQt0(F(Nta~L#W=bD{$McKk}r;mMgnfS-~VVQg}cp=!Ls>M)D-l+zXum47`4Rh% zI>g|3_=yLwr&!;|_;x*%XPcOaGCM<-QsO2$L7Ehpc9y~yZ^!Ge5EK&T6F&rFO;_Ps zpBzNlsHeFl;!H0}nLx|uiN)yeO?~|`DZ%Hl3Zvn4g0JW{mYO!N^ksbiU@WE``K0a< zr81Mt*fP{-BS0G4tvfCTgh(Ko`jXej_;i_} z@9oS>ysC9iikoNbHqN<8U=@zEH$7>bz*GW~Ro?20{W^lte9=@BqdM+2+9=j@h%4NG zVx8d2wsJN_B#ztWx4w>`!etSX2QfmO#`DS%r^ts0@r_7{{!63#U|8e1D9-5NPsd$P z0{#)FrFvu}>YRJ3>zEPwE$-6Qh~k^3*P(p>c0^bcbZfbYHZ^OF?dBEVYrB`VmC*z< zUnf;p=%bVK$-F<&J1t*Cb{I8b23Fm@nWT?DY^qDCvi~}IBVGuo zU*2lq$oPX9rDsqKUPw^_8Z-5+xV2zJ!H@LOp}N|4_lYsztz9tBls5Bumey#%nGnh; z8u?5VJ7h3_`!vK8ND^zrw)U8VRMrUyjjIhkKx})U^l*%@deKK< z^y5$JYR=@LvEmWWFw1NvE?uYygJq||Oq3BI3wwG6j(^+1)p1^ZXiGHWCK*%*nNn`E z_n_rlHy0vIS?vMbV1;?kUBdd+>r*K(_*2YvvZSCB!SA~KOlZw5B?CO5bKrWZg3|Qf z%3zI^`@(p_Q_lLD)iNXPY*eF@uBFaJ>^at{{FA!Rv$34N<^xyzzlmu{(OZPZUj2MS z5F;<+(1?)R<`)Qz+o#Dv#b_gP?#>h* zc4OgjZx$A7qlzsK(TdR>9W1DmhorU!5=2JqGI_@JrHD_YR_YnCbp)N0b&{;c7DG|q zSko{JzV~D#^)LAp>3XkBS%1pHVMW9ouu3vHOA0yHR=TS{30uLjpB~*AsW6{Zs~HT& zYSU|WsoDxVkgXX8CJjI2Gd%$&Bu{mZ1TTJ^9r1l^<5P$F%IiBe>8gx>28dlWHfzq4 zr=BX&a>pb|5PX>OGR{qekoQg`f?9TaC?=)PsZYFo@$^onIPj9^TB9FwpXGA*v;EFy zW*q?rICSIZ9DHQ`&XM^cyJLBg-t-_i{K|P9Xfd+c?@5fuv#B@_J1uQX>n`PlSiWU_ zir@K&CW@+^sM}iF$A*gmT`{vy`9^DgO&(IyVh3yEX3O>Hz0|??H=%mRMBO92P^<2N zLK}ioibtMu8r?%-)RU!V;nXhSF8l0)A zge!p?GThpnH#iR#KstrEj(X||zY8|Eutkl?QCKKv$9R}Gp@=-6gUD!ZFbwt)3Zf8t zVGu+I=&f|Fc%|2{!BeBGVWX8>|6-OR}5{*TmauRMiX72HS_bEQY?RqFU`9fg{tee5ZfvHBZ_$9U+y9T#&T z&C{u#HX9&2w;@;0e@lLPC;KBv8hPHM_aATeD5PFpQ_k|127KV~({@V-g3|pbE8(?+ zej}jGg~3D>sA2GX$fkG3rf+$onL)rHL7u2K{Ex za``c0>aT$Cg6gjeY}G<5-+$~Ug)gww@!11_DJz~@K?VtKK(lNwY6#_+6+1~>7bL{) z)Duq6tmGBd{n?9JMmVS61eXc5?Jm@Jqp$ez5zT!%7~jflau-Ur)$r%mT}^M@8XXGd z83m$!18I`M9gh%LiDh+N%)Ab}uErq!5**+6BeQCBpz$3G?W_BRHg0^t%>Mr-*&{(L zR!O~Qt(xOED#Y~SP1oZu+bL7ArP-U2cXFSa{cJ?;jI|9+v++WF3kHM7O^X+{>5tRZ z?BmFff(ZRbXiGnxG1+LtMh5VmD^XT3)GoRxJHB+clPmb}MjK}w(6K_t4WaQicFm9` z?KWz71!oj>Qm}b!C*nbbU<6-;Xiwz22)tjZzwDfAD$M#GBkSy~eIifz$_dno%aBp~ zA8}r#Riv^T5{A~6VrUPcGR1T4m)I#s@V~1K{H0?}Ij(Qp1 zGQXCNHdlhF#rD}KYEL{!c|fiG;+{Xg^0l{+6OhZ5CW-fLUvyGaq9fXH$jawV`GgpdYKiOl)FyQa}81k&_B zwLrZoBoiNdMcd{$<2rUqDgq*b<$m^Q;9)9}o+@IKj}m9k5)}2aTm3 zv#28N#`*~^C=A`sknem6LT7z8Wr=2CYk9h`ENoiz#tHV~#45%5mo+7~xu0DAF6HTH zIw9f8x)JbZKudvgBB9zC>1Jbc`@KpjQ2)T>a+4A@AXO2?B&O=I`~q*wGled%YR z!)BrN75DBB;R1}62jvg9ANtalcx=e10FA7D9SY^;I>;R^dPIX zCPNF$*IL%7>Lh&Pu+Bz2{3IoMObYLMNSQueG|uv-Q3Cc{!ZkFasX}}1vulQ2=&3R`U2z>n zFyoapPnt(Ns8Wt?hbj&nRhduYI_;nrp{?^^PWV!yF*ntap!*oiYA(8M((1r0`T)kZ zj756=C^r=TjIs53{66k}Q$wE=93H?qu?Q~vdI|B5t!F&r%Y%Q|1k{};vYLyYE$D5{cd#cg7Yn8eA> z1+6zq`q1Xgy09y*#9EJLj={$zoqehAp0K*?S0ZXOnF2P<>;fo|Ie?F0o(+NdecJ41 zaWdVIVoKshRmRM}AfMHQH+Ta^sLg>^q{N3b0P^i3AlpFZo;!R&oN{xa?7%VR+d#OQ z_L`)TEWM^|wJMrgwnV}j!gEFgolx|vf4%8q7GWy!-Y#;#88Hp4S6!3L8xI~Y$E zEHOfyeI0O)Mr0EwUVkcg86jccne$16cyB{?I|9lGIEw%8Vvm`a@#!lwFlIh{%ds`H zWQ;#UJA6zq?J{8@xUR3ueRzafT95f$o*<4jts?tIzbsrH`h>Jr+X>Bk{~l^k3*0~~ z)NiJ2o;0Q#tkIIY{qXHWS}pIq)h^^&<@_n8mJP_O=zMT`PpPcU;IQeUG8CS4Xfo_x z&cEe4>}T7?=C44|AlO5_Tt~?wA`InWI&);#j267UwrE zEzU$v=L!z3V~}5!EFqi?2}h)Wk}vm>#!9X{?UY%K-n@RhA0$3DN{(n`D|Jm~*v%Xg z302Cr>15CM$ad{NRllJy|498LCoVAUwINPeP?RU@%zjzdtGhX{ZjPxUc;fvnVS?zt zM&n<93+0|+l{@r?$1_asnvL!sV6KfZs0>X%7v$D;*%HZk&Zl#lpsEAiP|G^D{?>)n zYniVwAEU!c!N8@9>oVHTE7+Y|(ajt<_(zkNeQJQPyccXHh9@Ea_2(l8&CX-Plm7V5 z`F4@b6r0{}C`85PuOUtxce=&VBbnsM52?6?qIgcEg*ZAAkIy7fqgxkfn}3T1CaSVK2C*PNW2(feD8m^Bhx zBSU2Y=%#rPxDdo3fi~uKs_c)Iaya_?#mO&X<(~`gyn75iAB|DHw2auus;}c z;D^w@cu znfX3&(Kd{e;_%}pfw4H@`yqyJAfl`$Klb>(BXURO*suOUKT@x#|3VB`1J`xYin(&1 za;>Bm-W1DM&kB{m-j)DXLK-Q=eZp_r_>7JbLFXW{nuGZ;&r4^2gXolqm$0|k`(m#c z=)HmKxB$lf44$wz9tQU!(j^MJJ0z?ZDMLE)p*%VQp=&A>^)&}>;s+3TX))`LWyoZAAT z=EW4%6`kMjcXI-BU}D#AE%Z329?nwO={Xw|wW|H#TB`OR z?>h83=6y_+jvmS6D@Z%et_ex2<&?QGUYP6_~D=y6d<%ux|HnEE9SsOQh5<34atL z4ERxah3ME8(PbNpTLPyWWHHb??~%ydg-a5r?3E@jL~AI2V{;Po2%DxuuY5rq<76qN z{vmqOK6Xdn43Le#gNJe~5XVn&Yjd@5mGR?Pt5+>W-1%`_J&x3OUN2kE+qgD@=+-_8 z!50KvL&jU2NRNEsR0^HJGn$5uIU7)B*P|fSktTk1i1V4$c4~2>IPb7tKRLbVT6re zr1_6lzX(*A4yUGq2+b|v(i+Fq|X?uM7j_>lCexC+6mWSpt3y^+A zAo~;qs?5tlfD5^0+xnJANOy=n!XZEobAEm#V z|K_JJaMYBS@5$4uYplP34XWnSjQAR)59v_PIws1RC6V`qv$i79G*7aP1NQ;eCI3e8 zyTbIC@!3$VKS5=Z&YJr!w$o^&;{4?VTWrEv{LOiO_#m|eLKJ`@&6de8O{@* z-=uRg3(CtUa(s7D^Chp}QO14KhC70Tko@>XP&op5vblR;)~v!{g3{X^N2wWZTYkBa zVz?kDBys6#kZ2%hyyi6_F^!GH%y3tAPT|R05?lJwa#{i)Lk>=8X%&2*n9OnK0KrRuq13v#v-nW{chHv|gM^N+nT$e+9Ph?1HIt;>@%%*lL+`gxz z=~8NKfGnN#Yqx{Z9STUC zSZYI5QgYTCX*H?WGA<=tVi#&IU3^$c34OTVArPkmTPNA$K{B9D zsgIMb1&=?msslj^w%e`xI2c7OZnQ7w#49qjww!19R(Efg+jni-;oogOqaU45siY+n z@j00BLvmQer7Y?7k{Y?<$C;|2uNrp4)(sK6l5w`u zgJ|*ZU?}A_fRSC~4}L+sf~G2m7Wh-qM=&r6Tjn(n>b-BNkVDISf2Si7(b>S#-omN{ zl{u4Y3IgzVS$wCLl6!V5D46m7Bg}Z_y^6{Cs*0bp%6{cq1crDN7m+EXGOpw@GHF^* zt_o_tORAl9BNuu&E4$H$T9g5xW4@oEjTR6IMLQ|}iw-Tbgscjb*vJ}0gTg){Q}Sof zy_&(bi4sftOaml^t4N7b8F^DaEx#)H9yC5SfZ$e1nr(&PvKv9QQVJiqNZ3d(V#1G<6Vo{J3J-Ef-DsiZQbl<74RmnQpPX1bENzL}2_ zjfA%jPxy*f+j<<^hj>*w46DAlDDYYH>d=k9Q{e?|* z!vu=QjJ@1k10|nL8TE<*HeLkdbKr`9-=9IsOjJ^Dp?P{op?3L*rslf);Ggyqnff$a zjerZnp^hqwC+Qg}nXB^ZeV1Eg>Le<|Kq68FlWH`Fo8-*}P)+b~ZBM^3!lH*ciCjXQ zMAvGn3LI4SqZe*nSF7AhSN;2);HAoi{1jD-{u|JgmK)BWd5AKZ?~0v4RbxjTWl$wi z**H!3_lqalQhuvU%3c`>Txm?QvQII#U#8CP(`7^+Qc#_mA1|QXA!Kz4jH~S{MHoZ+ zy*iYd!wFK8Y?a1|78&8sq_GEvCQ3_i3a#c=JCo_|RZ6adYsksnjXu1;a$&w&?sq4t zK^vY=Z~iu5`Zc%d89b!YNm(!ZI_&r=4VKDmWOU~ehjV0xhE2yLtIxHWRtqS*S2xGw zDq=0lDvKoGo*;mVdizeRrgn3+`a<`^%FDCLl|#N4OLm=fU*M`eK5Y6^%H04l-Hs)) z!h7P+^lHa60kuaNHWMS&yD5c4#M9x|qxPhRb5ei41C|t${tURnymjB>Kv)7xX(;OS zxJ-|R@y+rDcTfd4j`Lsoa@nbiVE9V$gSZc_4YHF#)M$b1SmxVoskR$S5gkgZiht)Y zABM%vTE3LBG~A*P!`2#UbgJDph4^_TdU%Z3xY3C7dnRgGT2kXP+go_+tu7{8LFxx= z?bm_6a(A2Zz;!SD2@NCl7RoXk-Fb?lO3f{84gU4zNueKOvTi5MgV`kDzINm|>vTpy z$9Q!L-J&?B^V#2k5^?&CMn)fxNs3KX;Ll7=WQXw|6#>kN2t6&#8@};p5DrtmnDuzpJ4uc8cCSG()6KoSQb>Tf|aj627J5>Xvh-xtW z`hlcU^Qan$ec4~GDyC>p_ z3~QO-v$$u5G-ebzr4HV!^Y&U#QNvX>Ojes}?kd7IJ`fmv|4@Hjzk3@Tw6p;uRHoX5 z&HeC)kk3h)R7Nz6dekDq=FKRDxDIx7`^-w-1g^_pSsX;MT(q&jeU9Gj{bkCH*kqO^EvD5q?EG~ z>pM8 zy~C3QcRUYF(Y18?lG^mWp-cKEK+bk5yCSw{F2d&oH>r#Ob6Y!RKdfcJ;aNS9TpTR8 zZcyWaym|In4gXK)>093t#C~?&I9**lnnJ`sSL|vaU>ZXshwXR8@mdYy<`){qm*Tgr zAEx%(?Y{GH!Ytp3q!xjl{tpoBHwd{k^x1i^+(lhRdy-cp_Wt^wx<-H|37}l$!8?@y zKkqmgNv(`c`B`jXwZY?D*_9t@vsKyiQfk|7JLLG# zC4(E1JL)#4-vrF@*^DiQZ8Y|M@~GTua6Y5SekZ~v)oAdt&1roM{jzDlC1&(8yHI$- z-hRjaG=Vn%&UL6*o{_AR$mZHizL=>9*5@*vt;lmt1ulXq4hU@LQSy%8rZ}Iey9qN; zXBP->o8YrP_x~2(D70YR*2@;_Y#>UofYp3*l@-tl%5~1bM>%CfS4nP}Y5jlyV|xgS zFa{teClBlpJtt^>>Q*e*H^&W$ON8B$t!6jz-!{{ZsVEo4~MeXfDJ6 zY+Y9Y-~y&2V~0P{&+eZ6+hXrk?(01MHqU<7m!y{e9jffvU2vanUteWfk{ifr0d348_KMG)pk zk+rFvWOz<5{rpbN_2d-M+5fk#`m3ESDt9D;8llOg4f`s)4|h11yOBNPg*T_3=>L1qBvMYn5Gg4)n=cFp^`p3vKfE_V^k+sL zg}`@T2}j`3bddEzj(pQtqbOH@ujpjZc?G z`pF({)!)ug^{wCbo(om?TCPyXEPYb<-N~v>@(iYHiM~b$0VDO7-uVlIatpem3)KrW zFZYbzL^YjJb{NdhPzvgWmhb+u72*)_o_b$o1#l&`L8>k>)&48~FZ_oY0L03i`eE-5 z+ucI)zz$CBmFEF8u&O-047H%R-5He7Ld}*k{WDbRkX}5_1Ia7>!31$9hlm0+;=Tqi zi%!t7=-`$U7p9O_4)MD2ZYqy_Oh7hZhV#ecRUyU>ZHSd)zOad-L z(-T~H3QeONN#scLPo+MKIWxO){0wD-oqtYKK#_D@Kw+ehIJw|kA^GEm??E34=gzWX z9Os^H%?{gm(9jO@l*8L_)df=JSetB8^4lNyg^}7>LCwG^(6POG1BK=KFH82Ko1ulT*RbYJPkU;4XOXTl_)ypjUpWfxnRPUGcV(aUArxI|!)En{i%DuyJcs7o zID>B8w(7Q7sPUPX=uWO2FK}|C4S+ zM|GHiN!4`eWc6KVWfMAgmZk{3#(VL*-{P+))q(3J>d46$b<9?Udf?heb@bwExXDWt zFv74pV=Q#K0|#3Gd@d7`@qdZ_Z$Z-0rf>ibSx@kPg6$<16}P`C|K>Ve*H$!DAQM5w z9X~zV2-mrDu-(_<6LO(^=4SgnpS5o?=5>F9bCa(qq>2g&sMeo_V~4ht{E2&kOc#P2nYRx!{1s=|=Smz}%UmFYWA;TmPW z3i7UwS>CGA)K47Mr#~%h6hmM>Umt`3_TI`79A?^oSzt=?Obz3zK%eLx8sMe>S0w(? z{NcVVD^sRPPEY`IH!JEdiP=}rIm*dJ2kx)A939ZQSi|vN6H&@LzzOp$t%L8@d5nhO zq`j7>Zo$mQDUHz>DyZ-M1r-2m<9L3{wttF_IV7v)V0gT~3Q063m7T2~^LBZL2+gG)w&poWN_GwgF2ZZ1B^*)HL5`lTk-QzF1ZSI_eTvcbzp9##%?}jm zg-Hhu0_L3`AD^SG1JH59HGbXE;mvVp84aQq=3>Htc&x|RzX6RJAj_Mp@^7bOI^x)3 z=zv{igJ&A7)+&+b|5|$1tvB87c231jCVehWRu*=@fpgocqwC+maLO4{HS`4(MH`OVzX}OHJy5yu!Xlk=jMr)G@2RJ@nd(D(g`{=+WCk$A4 z)*7Y|)LYURw?chd%CiBGfeI^C_1n)c^sNXe2Uvn`0f_yphA<$&oI+sFDPehzrCArC5~M0Ydc% zOigc}#!!3nbo8?@0%P#%_HY~YDOZPahRSH$x;P`Bqw$)i{V4fOT0mm(oayyc@jMXw zsiKTQoO$#}(t~2~pasI7G$|M^7rjmKPLu{&Ey?x9-(f7y=8VKVGSTKXKoRznrCTz0 zItHk4L0fg^UYMwF@ec*hdW*JYOKZr+sLz<4gb+AI16DRk+vr~&8E&y3r08erPb8pz zJx?e^#$yWahSPRP?)lfM-3pzHc4+Arx`*jWxuteWeggn~DWGoqClk>|%PCOzznWHn zXE`P~$}YImQZ$H_4~pA-&qUDXbCN?8vo0e_kYy+$d>G!oW`oCcM>lcf)2F}MCE?R2 z1U0vU&hh0X8E0xzA41G18;I7E^_A>F+-HHJkA@H zc5O}3lW0SykXAV2;q?lF6X-B08mkv;Hi;qtN?7J(qed2UO}{(v?CO+yXY46hV^KcQ zM^}6I^-vYsB{pr*go~3Ex#x?+mD}8mF+ePlAk9vUT#@#1GsGmKxvp$)Q+Nd*!)>` z+e-0_IJS=+E0JD6-500VgDN{6kA~W|7to zSlkoq9B92a7d+y2+weSbqxSA}xXPw&GtK(`wetls#psbz@|AMug?JieLhA0EzXYDO zR#%h0)sWrhw%nIg(-~pbcGR=%mHSX-iZdy{0K(jq8LS>)@CH0d=lhx>PuB~)4BUs2 z4)?Ed-$u@Hvv8%C7IS@8b{m;A%2cI-1neBKm3=n8g*AUhSBDzI03&kN3%;7u;kCA` zi}~`0LruH12zsn54t}8Rm0ar}K^(6&-GyvD@;0fVg8mKWocgbew2_r0RwEo>l(9f; zdO<{iaRi2XY;1y?_#q`dt;)J4@Uwk+2rwO@Nk%ztgT5=2OX{4@-9!Bd!(U-%3rb$$ zT8?Pz2F<_vG&9A}4J0^7DBRSf6c4HyLUZE^AI^AC$`urBht{`(`#*%ImwH52W5k$j zK^q6=orcP~3wvNvzvd{bK1w3-W$tm~*EEPY|PL1>{k7Wv%N?HhCWKj^Fby6- zRu>AHae#hUd*USNm9BH`7{0?jiJF=IV?-xBB^>cRH zWXz?s&a3?}bK;|1-I-*&)$901HBnBBO5dr~VjEXd)g2CH*WUf*K~%SL`#$S(&o(9e z-Sx;M1gU+fe0atuowO)7R+?AYq6b=LhBFQsY^BY!jzpw^-ifje>xu_q&cnswd|_N! zEgVkLHnklx)^)AX6wqv3KyPLIurc;lpvrm!CycqSb5b&XIDo@rR@uEh%6grEG%Es|gD%;ywrn@mN&Ebz`#SWyR6AJ~V-o1$o9YV8+bf7{q@*911>yb_f1vcuy} zQzs$TqeHgzlApBdQ^?^=bsoUtl~a51BCXOd`sGGo?fTHnPD}U0%@(fpQ1!BEn_9QG z&Ph%NUE=W(L-3r}1rkB(-Q}~OAYVG+u!+){z0i$7fay3cL83X`vPgw8`8$1p0)5ps zrNELZd4jX>R;iNP^_)N{c?XNCM+E{({(-2jKYR#AtSHT`0R}#rS~st-{oX2PVXtn* zY@upzjkBRR0N4LJFt5*}6TsfxKtxoB6I2US>^_{gt+T(&eJI6e^nE0=+Y1UeKrrCPp(z%_SMcqCG6rf`r9 zLqS`uBm32;Z~m(haL?t_QDv~~*lx~Lvc2>d86bF?;D#~zxDm1oryIJz1CSo(mi?g{WUmZ$b-hb1r7@^z3cPE zcyirPbV5bxpZ)`Xex~)xbkqgS2x?&F8ngEN#C^*N0>fa1sNo!^>B&W=p`-iK{O`{qyM&2M8fJmWRVL!J~b8wm`gIac7!zj$2&5re@Utc&w06bQQ;mlNs)&$ z1D024`L6_(HfG@hSZ|&*MIjLKwxQI!<9ZN@=sHX{)9vTraq}mskw;0<{!;IDZy{VV ziR)u7!^Rp(N>0W5u>sy&@5=>L-Lhad*s3en?e)Tva*e8q^7E??-?nbd^_1e=`C`f> zrCrq@jGF;Y;1>2n&BhsjN?YW_0P^}WdaoWA!PTz*w^Z=k?fiLJfCOe=Y!6JczQ}N> zbNkAR?>;cCuxf-=8d;<%p_O7vtF15sr=&J4b!D zbx?p{D+c9<6{%jYZ1YEZ)oW=}((TRcN{%KJrV)%$$ z$7L-mUhw>3_$|-Q{jN3Ff?0ia;7>J(;3PMa?Hn(Ws{=OI)9thM-nh44Hn8(!5oub1 zT1sDvq+a2@Af`20$>7^Q>ei8~`+UB$z}Mv1Dgv>)6m2uzcB9(Jl4;jEH9%&#=2~0cnl{+;wbs?;2xdj8S^zD?0XVF=o+l0gKa;hRwS=`W zLuTZ?`NWNY4s{`UyL__ISQRZlQE&}@e@Uxpl+P;NdCC?`X!{!Be zw=olU1Jgn};!lfog@G?ikkh=t5`@bOvx{QGEj~BfWx;!&$H;v_-&cMHnB4uito|9j zww{7u57?c}I)?~Jf1dJy^|8Gq+>;MtwW@{q5YdR#xnZJIYUSgYF|>VN#a;u4-kB}h zr50LMkG_<)nW*De8Vn-svh|gcsfE6~kXnA?BNtk8)PoMZ?t`GkUvJ2|wb!Z(lk@x( zrbQo#{VF}HJd`3D>CZF`M4P&Asz&bF5nxp&Z@SD^W4SyUc>cuHJ*THNA znZWr&Rb$rV?vi2~ku$jD(O`Gd_g%wu*P+Vl)=Fkr$j3G5ZsIqlakEhcPo8qcUfu?T z=54)v4gyM#YHYz^9 zpX`Sk*O0HmMky+wq=JiHX`8TygE~sKkz~8Zjyf5XmF)}|dsXGitsBWHpS-(vj5adH z0j9!!CtL{QVOnSz86Nj{tPL{pJm&N>G;;7S(~4J}k?ZqAxx*J7jsC!V4?5g)SlMMI_mNxwzz@cKH7)dP@A5C=Hc<(ZgmQ6zdFFTl)K1Fj2DDWcb8OE?=HR(*lh1{ z85lTPRRV%Vx~U9Rju{RA2zGHm z!9ioE>aQ|NCbALWMr7uq!&V(EkS1+5b^bC*BNy=+_#@*k9Kq+tj4D7sNSS}Rz9hUP zRn`4g@}-DT1|Nh;&E|dbG?=uvn;NXw4RO%o-WemdRNMpWjpGo zsp3b~ViC`t5EvBw6GUw;xxUNu3zokj5;XuV@w+xDW%;{Q^ItVhiGLIYmTsg5k^iSd8>Re+@)OJzVpK=cc4WXp&pH~rgFEIQO8FY3|D5gd8Inz2 z>C89e3+9E=dRVA0#*Bi-uw@he(pY5g_B(e(1!%5Y-xMX4HrA3)?D(~t;v_v+29GOy zOr3|l`9jsWgBnWQ8C*%HwT;>UYEU=OKV?Ep!+d?YwpQA{TcJ9cP&2EF_L!Ab7wgG~*hrYDBEd~@NrG2k-$a@E}qH3ZhJlmn#ai07I! z*r|o5saGh6DX=zpGi1gmM-Xhoi5sqlEb67?C z7A7a~;i@VsUZw>Ww|Vg^fd<;SZA5GnrA<@&A?)1M%{%%S5TLlMZ!y9mw{LJs;DYNE zIfNX>!S$J9NyfDIkcO$ra=cp6cWfS!yOy^K!@(ojaG* za|&4!vUiAjA~N^bE5X@n4OO2nmxcvRW(dIiUhxGVAs%%-=y{)~y5Ni?hZSsx(2NN1 zGN}YE?D@M;y}mlHm8#8$t)Enj+&K@r`lnZgROVJ`Lkp;d?H^VZ-sg(UKvI$| zrEn1uBX-|?psd?)$1Mj^dSoStg`K|sdZz^*dzz9eJh4n=c6*^G4P7}E8BfZbu0Vjv}l=8)ffQ`P0tc_>53y_c4?a> zzCuWzdH?4|yG8klM#j84Qzh=C*=IL|8 zDHTsAlcm0r-G7SQA(oOr5@aiAsu_M;IH*;@ee|{sIV_4AIutzhk#AyvD}-lBtnSsX zvW1b^xJ;Gl^5Na-sUyLVvMqd2Zq|*!A08kV3I%<$Lnh5x|8`&gn|6H;q~Jh^q=i+9%^0wb8*CDrHZohgqS2&qBfpI9%`QX>04`DA~& zk>M_3%&@eE2SV0n zR?-9c%f3>#mL%{$4>%Q5urfbe@?0~?z|FG$L(hYS;zp`!|GY3LRjnrAS?8B7tBGkt zS#5{z>p&QvB>K+Nr3miBb@LB9tfq!5VC$)u9A4REWV`fJV0Tz-eg>n};eR<h(NLo7@0QG%_rov9|b4jI$&#z%)Cz zFRw=jDq_<=d)hY{qo^Rt&LcJ(Z^%kjh|)wiO171#Z_5{YhuE+K$>7O~OfV(VHch6# zRycz$IU?Y0imli1CyDz`22QLlJf14C@8hsWUzQu8mg+kzKj=~M-RYXhpr}rNEb$-e z7^X$r+6P)ky4#xHC6_K-frzFHXX;90!zZr|^UhnQSmV$?{K)&ttagLgmOwS4BH3AK z&9#lIFS0w!N*OTfN(dbmIrU^4WG5}tJ2Zm$JAx&xXw45;W?t~nd>1s{ci9L+(Nn#y zq#4tyJG<^R-93G|$qBBD#7unZMQr*PV(~tEeA>+HOLRm(@E{id4{Ki@*5uK(ueBAe z0$Qse1ZZo8A|jw{fdH>kr4*4?B#Jj*euShKi&8wqmOXZ6TocimskLiv{^b zKK9(T-lD+=iU+bw1yYX_h}R^X@a+t!0KUE25lfRN)q%Wo<-2RuIXjp=y!?1n#gLJ$(ko;M&KCV=Po_Gy+CH^;7mwmh2{rJy4_9;(hPf;a zKHcv*a6MaP#bZU`MBwA(H!-8QweJ;{6%*&9^123Vn_7=w4JuwhyWS{q=`+3NyclvZ~u6MmP zYMGV0V5S%!&HHgB_l_=!rYO;c+V;}NDJ{A5mr(z`S5|A-6+dkf#6&IM^m z8JW2s5s?SK-I==h)w#W$GM^TEQZAIc54zSB`TpGdI<9xuOqos~fU~4w0N344&v4Zgu_vds$0I@ptvxFTqjQ|5)DFw<|%}!&Tn^ z2N`Vc-FF$|*tK>pz*v~=-BaW^4sJo=X#27W`(rpE!&=Qam3xY>%Y28};QZ&v0B{ps z5V+@KcFNzzjBW_PaRPLdcBQJ_BR`M<9bjGT#E3yP@kzF&1L?kXuh+)w*IKVGPgYDV z&5hR;^%UCkS}$HUJua}XkD&C|1Q&xV670(KOsl_n8M(rzPZ_NXjIOd3Er=6>)5I#@ zL`C!3<$mA#$>qa~ySx4}H21Rj(Wq@3_IbIteZyt)IJo=cY9y-maJIsb()*15a39k( zzE+Si!u9CKNo|^6Ze+nhEG*dmueZBNVDtFw5`j7B)lPsG>91<`(_EGXcz1lxPQW{PkljQ zov&ax39e}jtza51x2^WquiBTx7AJ<*SBD%y!p9)Ug)A67teiW<3EWJl^^FkZC5`h<5EC-r1>*j6t9QlnyX${nO2Cj?#`f*G(8M zzDy*&^19>=4vb$pFj{`TTG3?$3q~Z|ZpQ!#p~FX{_!}Yp5B+DK6gbg`XfM6C_9`o3 zAai}iwHMr5Sbs|PZs_A{Tdi7ttt7VYsopyP(ll&U6-;R@0w+o+@IFlw6)yviCOcOR zTz>N0`}w2h45LVP&Ax=%pSMR%LU)ijx#`qv8Y4{NUB9_8x&ZEsh*K*q>jWcVl9^I+ z7AK=MdvEP^S8hSJ?XiEVt&N=W>wryA{vN?R3{uCVryiMtNm#pQxwn3e6<4#qRJ=ZR zx^J{K#d=>YM{_Y=_pEODIg^f;;8um6+7Y<-nUDdErwxGnB<1@&f0QS6yC!g)+n*V} z8>XwSLOi5d8XB?m=5-#eXIfUiF6Bax5OX&oe(Oico4>_G zZ6ug~OO&$sBgCiilJai;-mR&n)V)kFhGR`O$X^cq6*+$v2mXI=)0%kg=`S3+@fNYGmvtS>c^K^_|!p zW};7{B7{Fey2iM}-_hpP)vi4@UB4a?bp4aXmxLhuUx~lGe9g;@Ec(OwIlDcSx7Izq zwz@nRC#CifC0qt;rOsPYPVZ&%QQHxvy}RY~x_txJ!7V|;<{d8+)Q1g}#q8%S&ozo4 zJpLplv-+=aVW-o}jOIO+(f-T~eV)?xvv-a+(X-V24bFvt#}==yx) znJO_^1+-Uhwwcrdw8EXF6>;_JtJ6gw-tRRBm4MmDRpY;P z?I?n_yMX<1^RJ#LWqSSEp3L3R#g)3n1FCPG6CL?IQc>>Wd~k^P_@Boo`^Z#Zv;}m?blBddr#ijzK9t>gZ0>2;q~lk9B0Sqo4_$#(g@P1oTax&6W7o$vOY2UB`-bf z>cU{KEuY{$c<0*1OAn)6n{>X>_BI*E1;#GZk5t52M%bFH9`D zu-g#K1x79JU;?#|hQ6WE+4t>@t-o&B`iU(?6!F z?$(tq{bXI(R5ZxSHvGb&aBg{WXo}JU{^bp{W|!6ElT{;EYaGkHY^@Zg?Mw_KwySTK zJn~|2rVlSKh`H;(4#Z@Q_FokqY@+U~oOcF0npi?N;jtM5C%=Zztf+(4>?PjsPqT&_ zK5{j~{ozj;@v0O7L@AN(geNuM?{ z(C<^@9oSBl&?k8ee4}9Q=0l`ovNRGD)zc|=KB|R%v{@A zw0ExK0hh@-L7Jc)5rQ_xZ5j=5?sv22I6_Y>qc%b7kJy<_@YGBN9(jOte zHlrw#iE(F0T2P zMcia59Q{e)uvvV2q8Oif5O-bw>g?VJSSS1d7fZpmAbZE?Hmf+RemA#!U*B?3Vjsq{ z=zT7-=9e^o?tHM_Ss`wEHqEmr5BU@NC{hMpFe^Ma23E0Q8@M|ppe&Grl^@R{$1Q~f z!^^jroRC$yMLw*b_Q?OJ4%6R<MaA!6_%x(5-Rq|LqMqK2I241eP0CPdr6 z5w`oSIrWX%SqfnCf(G7Bao+b?@|kx+k1;S_t#Yz==3Ks6z96^_*EA=ehXHw=?R@b)MwVf!d79HR z>E@Gl^5(P}o*wk7uMvj306u+iI`PYCN#F?tYFTLd6(;fA2qOlgFUZ-!UeAkdOLLCb z7R#zskE<}JD>?XPR!fMrj!aEQNuwODKg4{FizV2%_%Un@a4rV4OP^_%?@y52_o7ya zl{yT`0Nbj1%+?#UB-IJi?QNJTdq6JEZQ+JwZU@w z(x7nhS^=WyZ<9n>UK8xa&!rI=D<@id+cM1tsvZ9&GNnx-BPH?t%O0{Wl9szQ5NXwY zcsblf%^y3@UIq7J6(}A_E4G2-gA=m$ip$}AXZ#Uq4Sf&gfXwca({z0@@UU)ri>GET zm_jfAa!**ef5T*-Z~(&;_pRmALK&R^G8#+(ktLdYk#;12dDf&oo!Wa+Ozwi#ETc%+&#BsI?6Xra&C5-&D$AL8+~s$t4}s!oJO2}6$yOMhu|IRQJe9Ll}ps;Ch3 zIC;v(pk31Ag-jJzLR?bw1vrbxRH0&qo*zWmB_6)Ao3m`pfvv0lXK=vNMIwe@dFEd_ z&S8^|D?t)_({E*$1GDN!JLq~=`%33a8gk6dk6Kha+mgR{P_{ExKJ?Wui8AveacVID zX;{g~$AAFNe=tIy6 zo!@e~*zjCEnXUnZm~(LkS}0MHpKeX;^+X+WbKwl{sPzrc%?Qd*8J4Z;yyI-s!u5rV7{Lm-7EcITp~4&&P(dErbgdwiA*gJWEf^r{!3yqo{*Ztm0l5%5 zFUZNsttJm+qUqWKo2GcOFrFN$r%mhpHeot6VUDb)pfU?4XstfmlW0uUExC#VXht$4 zWDPk5y~j0uoNb@E5g}08%!RlAie-T>EpbW{FZQgvlWe!Z~cRta)Jo0G1AN20~u* z@zkfn2zX|;9WVyD?Ro#2Y}+?YHcJ6PHXbJM zA0W!w7iDd|wV&4OfHAj%99`7h7%M6E>grrCE9kY`gL|)UPCn~&JFVK?ko?*|Nn3_x zlN3)=uG{5RgJ+HUM+X5JDyqsnb|YjRD_@{#bKL4KA=q)aq?S|#04^dl!!;yS9(Q&W zI){NbtA?MeAL!j$pkM-!QR)d~`5~Hzm*(&MX9T~-omOZCJ)|2)Zp7acBb_!02NofT z91YiW&|z2MJx^ow7M5>lfrHY9)Z|?rYrPO;8iWOf0J#;2?VS1jQ<-u9{kk%XZeCf+ z;AlywaOMMC(t602PjHc5bSEQUFCyneL+~v)%>T(=Ryf{p>D3Vh@wrO-Erzm)H`k`c zhZGE5>kR9V`0?5_-sV+${VOjYk#0lX`VT`85CYWBb6pvFGRkV>*q5 zd*@GQZb(fV({(j#$StyE8uB=dW(hh`Mn-1L?R*w0Ax15&(d`W_$M+Ffr)9U}&IGlT zI2y<>`dVO|&Wp5L>CtNOI8nl|2+X^cAcka0}7=k{VGlP|v zm3zXDr& z=mkNDpTW*(&6e36Q265U^p^6Q^ePrYYLkPo=Tj8snn=o)pL!5e4zUotGg+gH(lL5u zc>Eb9(Bw z00*Vg?*1HCSVPW-q0#%~`fp~PjLXqXV@Suj97CF9nc@~)vv$TgkZ#eA%r6a$GX$_W z^T;-FPG@}65f4&Yg7}p-NnE(*ZU2Z`jDgSP$oXfk)+!`R1_trnwW_bhaKC6$ILL)m zrFQC2`obUpT5URY_HuBW*SP7=YIa@hpRe@XQYYnA6T!;}M%H}#WdBM>2aJ9sVnt#w zsNb8l?X$n>F#ky*zH@i|blq~kfX0O*CpiS6lYe{;Xu2YBTK34ti0z*#jooh0-gTug zkGU;oXlvoWq;~y-)TjVHTJgNa#e1xsua?tYfbYnJ0y%&W<2Z#z+=pnx#6NC^RlBI*wV3@glQ?d4Gb+CqPk*v+HCHHJbcm5N>%JWZDy z?RknL-Fwr!r!)LaJim`9FZs!f(x%k7W7Tf6q2y{kU;|4PhPEY)fA2^_(2?^icfw*Y zD0v~wPk!Q?Yphw_$hHPo*$Q8J4Tqz3@tkl9IUlJr)3Ftq&Kcg>wymGb5j5Gna{B>Y zB-~x1CodklvI5j=-`88}!$77^?$W)}artnR^%gG<-fB|U$k5g6U;^ev{bxeL;`0Di zZD!f*KmDwooBXUCF4il}j;%+tscnS?qUO%zSZ#}7HrGYk2WhMM#k&GQolBT3D8PQF{4LfPX4JRTW1vhw>4hwlG4k#5?p+!64c z7arVTZlCj)4?jb;bz(Naf0E^W>KvJCpy{A?=@mb$vGTB4@&!gncst-`h%2@Y!PjI$ ze*1)hAswaXW5XrG&%L~-*e+Wkq?cVPn(=30YkP7$U5+^={7b!^+oayQfk1O)dZp8Q z2QIde`(A#(?SPBcarl|6FBm<#G0X%-jkFRW9!2(9b&z)UeM+yoc=U>JDUUB!4bXL6 zG5o1`?W&)AfO;fB5Gu^p`;W;5`KoIMkrty`XxspW~BTfU{r>->Q}eX!iaL;$nr zp-1(Dc09;D;e_^Tp<~eUmB_Vc!&si%gT3`)qzUVP;fJjsjWb*}?PeQxCC1Gw)td79 zHSV&i_Fs2gsqmSDeuwJZ3Atf*1SuU1MT2<2GpEkj*Xw3-ni21b%mO>iw-3w9DDZy9 zCx+FQ%J;wEP!`-Cpqt?A8~BBK^se9nAA(tJ`_iJN;4DA+Zt}k*z1ffeN!wJ#<@kHm z-t(6(WM3l*=5p>g*=q=@WYU|$0*{u?*2*X|mUf@7BGSn8T#x&eUuIh77L9Vr+o?!l zuXE(pRKu&GeJ<(VjR)+#_LdI>w1VAz=|fSI6+?z?1o(*of4bZ-;62jAU)*KD11c5G z|K?1lKa}>U9e9 z)&mIwx1JpO#rsU$uq}a@+3vz~Hy!%IUw;!GQ&o%VFT&K*m z$HjPt$k4OX=qcSLkB*Jq$HV6zNt?tKv=usdIvAXD&qJ#e$#SoY{UB=dCZ-34*`&^I zr1ONk(SiX_^-wNOHU{b8#6Y>3C(ERd=zK35gXO8Peq_9+9Wh5LHM(Wmy?<}daieJdEGOQj$sfK1|l}2>%NY=;qO7Gt}4ooUElK6J2@c$y{MS8m&sj;G6GnboptY@PtVhlFx4VGI`vS<5PU7{I&Be{ zs7A1K8TESq?AGV$-h%K;ELT+cnm1|%pGXy|mNq_Fu{VB&L0!6T2^iKosheI9W9*TDan09o ztr#%0rF7<1F9K)yHVD7O0>9w|hqM0MD=x^Md{Z*>uzE8MqYPkr(*#aLQyxb5iK@L8mE&3s9R{)Dr!FU@EnL9$pE z5~Hy06hk1st=xz18hd6KgejclvVO%Yev@|7?H-+b@2oa!x6zOgPTzJ&_>7*9c@mPe z6`8>q{^B1k%EO!YBO(RQ)Ptsu-3{(bF!trvM|WCQ>>TFkEH~=r#;9w{(5CfkS_&qj zR2%zh)0e_gf|V?t##f_aF}GLOzmA_+HEQe|J+TC9FND^{ki`RR-LN#E<&b?KKflD| zudKVjG55tNR-e_jQe7i(e%&jqV22hg89+pG!Twjm>9~*8P~!!l@pHrhkttL?)$+^KY~629F`F*ndA#hyksX*cX>Mw(IhGY5`QWrML&tS9j*%rD)ai zk~3X<>?^ZwuFd!R7FQIOoz>ni-d}-E4_YB|_HP$H151cm;CS#O_zaj%F|X&`#U~-s zkuPdH^xq6M{>@%L5r1}lSv_cQ@7JW&v=hccA@0ZGG~V)hv3d{-^>sLZuj^&e%FB`I zModdkoQmzd<8l@O#z#6X#HrQUc%xJk%}-OAI>@y8rLh83^B;$;PRL3A;nwso^{eeI zfZ3F*zrhcE6F(s^Qjs;d8oS5vYVfM!{F9K>kYA?$^2gUwBVaW7s-sbJGTN&vTFx;u zQ`gn`v)fSN4TGrg&O@jXYSRrD#45)ZEeQA0j8Zd_*q2!+r)c4|5;oM&x)k>WNqGF)1b9ikVnR-5Y^Y)v0-wOC5i*e+OolMyj!(kP4zo;S)0 z6I)#MRIg~3`gAxOxAv>#=PX`Wr5yNp zL}p4}r}HRUs0rV$#=j$OSdCK95%P)$*fkMdm63?hp7})k@`ZE#$IjNSuIOF;oQ(9| zvgO(q)9+7SxZ<4}BDmwh(o37B*w%keK}X7mDy=ET6D#bcYZeIV{DwCr=E??Du8!RFJY_@i_$i!`uO7ts8uezSK8jo*Sy+BA zGi0S@OwYvPS%7*4CsOW=8e1ve2xH9Zkgwr%tA|b%p&l8zzRFTs4N{+HV8?YP3znQ? z_*k9jPO}9`%Cq`u-uk7BtDH`irC$EhzEx)6Wc3QmIf_p&-jO9W(Ybzht!Dk;`lI!O ztH3=TZOrW{}5f(Cyg@NV$m z>pE_DrZ87Mh(~C+HSyd2)}@to`Bz;25tnmJyq*mnGi=_|Q~)(-02 z*O38JMms1sA}$#0QRnm1{hzY}6AVI7k!_K4^mU_gXBCfyD)xjbq!Gn1lZ87u362`N z>zYem7gxupSG|V<=QhOgkC+9RMo|IbN@={|0t+$53*Yj~3K=!iI@2fCb8fG;mEBMb zbRL)EPoq|v3&c1jr^YxHjjx{R7k9a?V5Ie3>FN=aZR(-Vdr&X0oy3H{ytcld<^xxb zVyTE3W72pjQ`R7->8aYC`AP0#bNxZhzm}JT=JdiLgwd4J4MAf^hatZg-1f4qvCry6 ze$r5)41129DR|``xr5USgVDHCdqZO*1Yb2gQn*miOm~h9_^Bd+FEzEBk{F@m_Go$` z*vc|;q$*uZrQc#GqT{FHv_(1+CzxT)s)vpo7&q)cQ7*}mBq-1nDBqqPzquY_>j2R>1&anbF3Bi zHqK_>D12oCmi&tTGG{wOz(yHKa2AL<(!1pP8-~<9OMaAFBG*fB)C>?0mPd1wdeZq> zzTDc#9hs_fZB<5UUZZS%uImlOMN(!=iOc0wn?+Nbuoo|6&bX}AqgEFCxA*Q5=fM;l z5`lH`6=Ca-4aW7K@Y833T`e=|)oPGj{8>73;$i+)d{P%BD^lBS?Ho|wp>qw1;!`DK zr9hFR5*d6p$vO7sVGEMVt>7C**vV@Ci8YOQgOJV_;j(#H!+;k?vE=cDov6der;^u_ zULHEJJuobG-9$R_;#9TwEBcwnJ_DXO@^@>7z&V63vh-O6RpoK{b>_7K6XK6lp-07a+;|79dDf%+;sQZD}Wcvfk}PSoPV zKrOlnRoE&S%jVY~jHlPM`Wk3Ob(Re=xr?3&$o?X9&HRu0Yf`8;j|zL9?W}(wzvlt> z%XHqM0;Y4|IX8>+(^({&04=ydhE7wm>3R{*H%{inMDnU%Y+p$y2n1H(|& zp+9T19XD&dt`%<-aakG|T3;E8(Qb4_n_cAJEf$Th2CnuLB8)zpHjWH)Y_w5}R}25@ zjf=O2n|2m$67#t=ak>;(recEwXP5)1^Rkmn?8@D^M&Y59@K!})eRgC9r5S^i+G|Xi?o4nh5FO=a1Ku|bg3M5>Gk!%s_S2ctKI#lHvyLFDR`Ye19Nedj=C~Myphz| z6|v_Ga|!)QZa%W!{nR>o$8=pu#F}+xY^Q8OhC?A+f4;u)iP?g=b|t^=4L??GCk43M zzbLF^srMvwezlGitUTS;_$1csDMPR;Sn{rs%R}s1XZqdXrw3#!uyt>;)k8bki5Qvs zWzwFyl`8@5iH~9QdUuzTvc~&X(+RhWsZYX+qxj2euX+!9@!9&bN|4S}jD=K9)c>Gr zw3uOFDrKcui}UjN@igu7?@VqiyfM0kmrUqMPw^6;=3bIeMc5u|%QA8uuyl9m?&&l{JEC1hT?C96f`M;f$l#|5>Kot2Ja_1?I=FnHxgT;RfSEd3m-^Ye0P8z1Yi0oH_kmJci!oL)vJK_CNox6f|0W_ffMq(v* zYvkYcQT!U_4%A#rLkDwc7ig(;=!tsw?-gbe?gkI2QH9=0u^qi7pGUx!POcwMr2@dq zzG)xTSrr!5c_L^BB`ZSa%>4Q?t{*|4tGnX-YOdJ#m->dWaFUFIL0GPO=;GF^{q>Q9 zH8Guu6}!>a6Gm=?$Z}G3QK#klo-=CCa*a|BQTuQvlI4-bNa4DAJne;N$Qd z_Q`da2&1(xuAUhzr!+nbvIM#n#I13bV}eW#!}!bx#iXl{WTrb8$t$Ecqnq$J{3coUfl%YY?u1l z6P5@OvX$#4DV7!D9LaV$FJB4L($RRMaQDb#`pANcf@&pSIodLB=XCuSG%3S9p}EtV zsc0iL(prFe(X@$LhIZ=uvl`^IW2P5r;n(1lg(bmt!_eVJ>T3g&tmL`_>NG~ZdH8yD(TkPYTb#c@v0l4h*!b#nSkz6aDe2nraE3K% zrLNR?nv{_*wl%*N9bs6RT)Ng*zh=hLl@jp;A14^SRpg#T-yI#_ug@m#Vc$_`{3G{N z$aK=luFy+_6TMr-=d_$9kTF!uF`14`iY$QtiU4^JycirIl;&L%`#f1qe4j52Ksb2FIeMnyo;rVtgSym7y)nRY_Dcp<9R1&oU7m;*>5KpGYVkeK@(-*l((Lc) zi;siB5FhZ9#CzJ|<8X;+R>!{Qc|dN^M|9Zrum1c%sr(mFiCQU2$C8aH^Xx5h_3R;O zB*4$fVI1VdOjxS9U?J-3&L}*VSj!T9~NEt-OFn7v7f; zRB0hVaSV=`hmD<7;CVVM4C2dNv$br-tIyryadN}J z+*m5wf&-O{<7fan#z)SAqs!~NW#BCx7!YFi|E~wFg&g-e-6ny*B`iTSDKR&XGggjbRbYOE$yBk$J8_Jsa>E`h`ReR*?~VR1-1dk2n<3tjW3NiP_4j&gY%m( z5_DouV8)le2R%jdu(Z2X5no1FN+xqzrhF%8%#!n|*gViLKtWYO0!{hfWFR0t+V1Jt zYswEF_!7LRjKhMef&ppe-%j2i_Pk`=&&gvqJzIueCBw_6fdTgd!)nS$zMC9>$RsD* z$CeG$I)EVtlIQizXoCJEsO6#D9lK5Cz_>vT$n!^lQG(;bTO9KHrI4~Ma}c=XJ}egC%5Da!q63(r#mQK0Ufz)whHkORIetA&yNGmnFfgJ`?)o-;R?1_@B@)y1-&P8r?)E%vbG>axy0I+|+<3yb3{%?WHod>(i| zuuxfyJu;ToRJ!s09E{timm_W)INoBAj~0X=yqiiQ;(=Sy!Hk2!q-hE^k(_ z3=ThgX74cAiz^G2z)V=u(9nS*jxf9WPUJ%9bR*yNHwI=c>&V7jNJcNM|2Aiy8T;X+UvEQg*3{Nb3$==Y zanQ$jGYW>?y_|}2N4Z!#+3|aLYid_tt%F5t*)fmDwR*&4Pj8QqD)7s(`x-V7fm2ON zAL$1x2WUg^7FrNspxRsgV5?v(#Lfq_6AQiSB);*14zugIjZ|PQP_f+6Q0VhAjbc1h zlT3QbxO-L4Y4KERdRO21DrNR@wO;UYK2URQt#g{?+Es^ptX3}oRd5HqDQcgcl-5Nj zIg3_?;E{IEX}RKw#PXrqjgJfCWYjk;Egyly zrjw1|o_&1$S+~}Qcf<^9gt48Wda3~v@LUjJ0`ncBsq2)PBO#>hz>Kb!BK{hBXx)31 z7-;61PK9kuHn0Q=JiPx9;)R4vpRdUd-_PODKqbOVj`yK!nwqD ztyE|!fyp-32f#S3twBRT3KJ=bcV8)EK>DLW2k`6xm@!&f3YFfr(LeASSjDy#;e$4^ zNC^R4Fpvtbs?LVKaCM*S>kaJ!nr>bV76hb%bmz^n2k2y9XD7|FyUyW`7U747WF0$DK>_c+0zLW(gwd_kKITt$z;$#di<4hlBL%)uV0c3*#Qn$GA(v?fun{QU%%Kf$c#LDK_#i60WjEUX99{|!NA=0LAY zZA#aDUbl3R#5loIWM#k%&>6Me7jLgQ&;AjYxLD{x* z2H@?DDR{to0a}H1=Bp^)eY_=@UeLTE1mhJ4-_NleL|j2woaYHco!n+<@j#Qz0FT{R zUEl#X?Mf^3T+rV7zSF?|l#2>`9`D=Y{uJY}TVE;AYr_t1tgnrA7zV7?yFPFF&Vs1d z)OJ(`=GnU{LF+dbBe1FtvhP+5P!TTtcQ$&IkOh{rMjK3IO-sk1wJQ=4_dQath0P`L z-s!E4lixS~)3y+$BW|ph7o{Ad-!mhY@~~b_%60e8*yA`b++q>;nN`M@O#i%|CWIxl zkQ`a2K%aV%uv+kWO?-NY*|-OUw)XkDJ2Fb|1LY3ECtsp62T{D8PDGwpYO)7_*iK4t zG_MoM(KSKW6<~t^IP`+1@L{{Lx3E=MZr2MX7La>b4gqBYh@R>+*L;b$kj?v1H~bw= zE__=@*)gu|dFs>Z{KS!(d}lkQB>D!R?o6HbAVPpJxvaA~)fFngus}~)RVM+bRjddW z%{z2yFN@Y?z?5Cg*SWD4X~x1 zb>MmKD5xaOHQo|<3$~b_q1=r3%BNl7oJu>~2l@gAbO?AI9{^@eM}R;Vh!6UmHUhkd zjWD~N!MXQ1c2Ro{1D5~Bnjg@E0If)yOt`~qvqyUtH1QDd+)zGx%512bezz(t6u5nh znn91`MUWiQVV>ZH@@`%E!~y`x{RtSaCStNZy^GX00;k7;@4iE^07jQ4k5DQvaCEKB zHuOxDosYaod--H^Yd}!i5!3^@InZx&;91%Fh(qqkTXl$X+i)2+SEKt*5L%EA1q$!$ zLO?zZ0Q;6;f`DWLe<=VSfKcDn0a6ZlJtZJ61YEv%ImA8(Jz+ZhWZ=vp6AV;iMKcYL z(d5GAW1t~IKy4Gvs11?VYRk}N^EJTS1>X?O&^h3}{RmAi$o29W>GuF4`aeqq-FLE< zp~FGs8@%oN;9WgF@*qSWTnKVj$nVN$&q2Z+cmmCM@C`6N&|<)2;^}@4`7Gc+468z} zCslmK2V{_etb4(n2gAED zJ-Chv(0D=(hl%oxHqtv97?{n<^qpsxo+(OC-q@!z^Q_4zJ~Mo_xnj-cO#4$^y1q|- zQ<_yQMTdwjzfhMQs3m3L*=lcREqFB6jpo+cCA+X(r*W>t1i(lfNI5~o8kS$;s%B^v z7>4;G1mT1eF~CapDdI3B-E1vBQhc5r0;T^@34v{Fnf^1S?Et^sF1^A%k5*;mQ#Axc zkTa1Diuc;c6S}3!XfbtfE1hzO3w~igCF#w;(W+=1+N?Cdrj*+ZfE^cSh9hehNG$&e zmK*Xa{!WGHpLT>o82Y&f99{+eaTJuzmdPBkVdf=e+43$jId>{N{Mu@>Z3FwlIL}gf zeNRcS9!UM_CS`(JCMf)5Tdl>)yaS#fO$jvFl1}h-XIB#Va2a}A80I1|fuaGrj>>$h zie}_9H3Z-vG7Vd8bpy?XIzpu9hIqzXWxo9nhb`DnZ>wODFeMhR+XQ4C+D0X8@_zfY zBx+iq1gK>Ceck3REM6+staPkoftE6x3``6OhR0xm=^`RC{%m)^CnqLmWpnSeRA~A- zv_k^Db*JJ=4{52h;KP=ykkP6+x|WIA9UWR%+M6#_n{?dGxj5C%^Wlj-_k#Nvu?McLPV(uoTS%Xw>S9T??5X;5&6SSDM{<8LhHd&qBj1#^Cv>t>(lcbsF!|uK> zjgA&JpGRA+lsr;2$j%mucczhkA?f5?@O%pN^3a~b=2|iUWWK-k+)r^+t~2#K4khl5 zq(4#=(2_+Do|={YnH1lA(EgriRqZP5u1%wMd{lv{g}ZWuG4z$3M_<-|ZI|eDFI%Vx z6-`V&?i-)0bw@`V)Gc~)*zT6^)3i2OP#kz*I&EjhhjVe0(61%+N9WC07e1Ie8x`j8 zI9)kYZ+^r10BDND2Q%y~@PV9V5o`j_MUMSD7bOM4fVRN%?{Uw+3}7Rm9NK3Oq>R7w z6hxkq2o;tcoMyROQX%33_#|Om8w-CY@eL^U5g4R2mEEDna5Y~OdS9v*nP-Tc`M2N= z!h!8F{owh1{5!M>DjxxW^I#1*;5=RVEkv~swBb33upPjBftiV}{z#mN8R_kLi^>4* z^d~ggvqvfqV1SEbfHBttkgm3=EHCk$4EjExHGpbaT5+!Lu$$|#e?mftFaZE_`W>tV zk|-208QL6*p0|h1Mh60!uyN(~f`$zjC`a%C10&cY{XG{&?iGX!LFDyPm@Vy|O9bF46&6&5*i9iYyb-{iVFS;L zAd*>GH(Uf^-V=XsX;FwDed{a%eiWixRgSTq6AsaUs|1+OYREnJ%qqK+o02{P0A#Tt zgvfw9aLK~cQb|WAunnSU&=xAdr%^4uUzeQ(T(sn#oCu$yToL=JQkZyLzU#yP5G5x+ zlye4rzKLjj`=@sw;12;pq0waM2NEJV!WTCrkJN`R&W;Uzd%nsc0a#?Ho(*U4UDp6p z3EUf;mM`ore6>qos;VMx@W%j20>5DZ&>|nWR{_Z!2R%Wn(g4ZP6Ce_lcN_L$l3QCC zRJq+pKo*G2WwRAgVXDTZ0Y4`UVoVs(TJ69mJ%`6udABFEfWFhn*|u2L3C8~`Y4r=no_o`8Tp=ue+6>~pT>ofKY1Y#9O8iU9VFfei>aTHp!CBJsIN zFTrQMTm<;j|J)H5_EUqxZ^i>DR(Tsxz+%K(Qt?{BY~1SMGlHYfO)MQdj|AXIb~ z9L)7F%7Oq&Rf)@JNO&S(inJMDE5$BeZcUC(aSBzg1=+`_Uw-+PNRKIxcjAh&F&^$<%1QCtH#0?pBDr6Ka^ka6Tc8X!Q7 z)UHiK^eYp;{zJ<<`wqEI{g2A8ynb2&j_1h7vd8Copwv{6OC?U-V-2mkpY* zS^-+~PRC@&=dd{e2u6P5tP^lj*??A@2Rvs3a5Bwtp{iVv9146@Wk#Cok>0NJfi?>s z-PoQ!Ohge4V!45)q>{k0!coAa?m@s)7&5I|=h+38JX>^Tiz{0Iih_uYz*M!oVRy$K z(?u`^OULflTIUwXD1&dMKP916n2=5m81G)xOKI^euoawzC`o1^${&g>0UE%z6ySq1V^ccc(0&ZlDX1eeZIa8`WsexcFk{1FDpcd*h*}touxtL5tgaM`!*vzysKlucd zFYE8lzOX71*@+k6dB3)4w8xYm$YcoA>V%D?2F}sx(NLLiO>TepgWgNWD$XS)r>8t6 z0ofP;>_j(zUvXxO;1S@+QQ+IjG)P>YL_(_zZd68jnw2LYku63~Ok+3dJs)_K;*Ham z2~z_NgL;)_r8G*>b7cMCK0z5!^fu<%E*wa)&u#Cmy5MBLA>vAWlMd^FfV2q!kAXW? zH&YK+b=v;>Vi`bK$l3e1{ODQI0|JC(xUJsOcXtPs;A7+$ey!R$)k#bW`K5{&X0fsbb!MEL-UST zqZUEIrVTpd5#$5djE5cz_i6%QKdHMyv-n(#E*N3|yktGGckv?-kK0^R(b#=q(O9sE z(jbDi{~2giKMu@S zx+%=`jO*o{&a0v=aLgA*!ZU~d^gT$c1sU3I!&~C3#0=SkCs6MZ*=fOEx zJgw>PAJoF@M&f3MO%!4;Pe<-yuWwo3H|xG0c?&sTc3XMy2m1O7BW$h755%geuX#?= z2Pk!^mNB=KS2{m$blC=F%`=a!q2hHO&&i^=W5k3Zuq(xHeRUyLrKs{7hd`Fq@Nld` zk*<84>`TaY41oQ^!!0Vsq>1u=pIyFni`lDZMn4&Jy~|p7Z8|G;SjGm zW+)?oqsO&UJ?-gGJ~|)_0SX-LZa!F(he^8vf<7+crWuL`?0xDmOv{I=F1_zN>=k`lx9Ji?n@~)h|YeJn`hDp!szxplzb{+ZJ zI^SV&G`}R2AyXJ46*a8^cBhLKBP!PXaxV3B?8$^nt)*PM%3X$3WV^2 zV4!6bjpt~j4cjcPc4e+=P4) zlaEV=8cZ&jzciTF|8AnP-)T6ZW^$jC^U8KGHTo`q?bj4`=<=W+W3D2~m6)yoMw_Y( zk^wEC(^W!7v4dlgx~j}%un$#(>~9DT#hXbVTFI$)XM;Uy`LxA94x7~lvInYc2tW-7 z-=P9CnNn!#3%HfvY|+d`WmOa378HO8o(DC}m~!c&zgv6m>x`Bl7YIyvl^ve_C-kM> zhdD3{*TN~)V-@(z=3q~n=ox|VCs8IYRcoG*`7{87$N*TU=&}<&7pr#(f-gUge-pb{ z?c2g&{TR>}aug~>oUxT2n$cQ#`)T#xC?cr$Sq!&iy8;d*za)FR;j;li^L9J;qp?Hp zd8`xe7zU6%G(_*;&HP6MSWjQ>?9u)XSkD+y0CzFW4!#Xq)?OyOpxRms91k!ND_{fPIlKjkA>^T3b3K5qKbO(kT>7Xe;JnO;o0C%_h+2A1 zSYng0z~@P%%7mXIUY1Rx+D#FTwm+=~Vgxa9&{-6aSUbf0w?RS{3HH9`6B3!X@@H}u z5Hi3vmjzcWJX=n-fv#A%!$U34IrhI~2s+7RzqvV`*vTL;NlIFshDp1QNk}4AY*KaB zoAQ5Md6j4pYd>A7v@$vC+4yk5@Wrs@_Vj!#-eqxc!RX8KWdU(`B~DFJrOu}L{cO1; zK+K7i;;;}zK^@ns!;1jhOKW!K|y5+grvS!Q4y#jL#B#?VG3b|B*f7|s{(CV zvJx!L8L%sMS&0?kbn^)Gb8~*2noOQ1hn>T-_LJ-{`=++AP>)T_PNh>U*|ga zlYUI4QS_X?*{VBzN5BAI>GRap!UD=A=v17rM1 zT(kf20vB0#gon*ox&u0)w|uf}?By-L6mX*F$HDNq@nTv*@ZGx4rlE*aFn}H6ZAmdg zR}hKfYNTrLvq@nHI6Qp|)ZEgyJX#XaN94ezr3{x`Z~oi*?K|x<%}y53VU6NZ(e$G| zGSwBLX@BkG1#H^B#Ibz}#I2s!6#e0UG}@1`*Y`dWRVI2u1^VaG|F^>a*=z>A`~3*` zDlGc=`yV+kZJE3;qK(6Y?dMsNtJrJOowwYy_y9W}%X2mn91af*3=GKi*F%I+-b~Ff za&r%}YFsE)?fD@Y_fB%cTYF0fb4gT1*Mj*R!Od%qV!l)1+3!_aJZ$ir+DhG99YUH> zH*a3rr_Qb2F_NC{7RysEPv-hZ|50yrL~$|pW~4R{p1es9T__#ZWSxemzQ%f*42}0WL#^ zf-nON%oS5QEHXkOf?_RH5fQ=`-P}q#_wKMLJqGU3x>h87cZcV~o^i%03XM)hlw_08Fm zz2s>SwsBDf>!kk^TQX2LH7?YNU`~<^okBW%2BR_R>vd}89q8W{7~TaZ*k3jhvC2EB z3!W8=Pa0otF7Ml2K$Ruj;v5e7-i#P(Os*cM`00-##NO!8k@!l3e)WqR&FBaS zE)}ueLI@(NQLB=odB+>0#HV&TU&3Ybhf7@$228$-{<~{?DxpYuHXK#nWl29Te5WF8 z>c!}~U51?%PY@uIT5d=dCsWO;m1B}i+hRrFyqbSh%np~>FllsBBU4dSb+{@@388QieE&+aw?N-|qDFXtiqb@`A~rRi$^ zzo4fMYcv*V2f)%wqmt?xs(=UYJk(jb-!!Y?ISN{!b{ggRD+=?#W{OdQKEUY~MtMQM zK>C3zKyDJ(=4QB*_}^~wxv8kG+sJAz%zFUVVjt^adzc~3F3+ByuVRp^xJR$k6j-`k zNl0HVN4AG&7{j^cDx2=Gii84^Y@TJyX3Oby?{7M!CqZFn?RW^Y)oP-V3XjM|F%1t# z)1ptCBo9A()hbjC=_~XkaA0%gYxJt|{0FX=W`2%z4^7<&2i;wjldK`btw70xc#+mt z%Fn@RLoHp@M)z z_qsrh&j6f}435Mm#Qx=^dt=8TNNJOs8ix+E+^iE>`l||1@961xfaz7Z-Gt>w9h$BF z-pw*~D~o5<;#jMCC!S{K9Ad3M>zC-PI$0&Ub8Gx;)(@0f;_lgQ(c|YrLPs4@cI>c9 zz&rk6t(h9hHhuLoX&00Cy^!nTpg$pcWRTvpXwF_l$4FmlG{ZhC^arhk@pm5GpkJ>FiE&_C~{Xgd1+RHv>mXTJm^UA$90-Q-mVS5R}H_v*Tw zWOGa$=LEmKLa)F3d;~(0Z<|nKO8XZFOWw|_JLMf~jC~JCL7TT=#k8Y?iKeB@4Q&+n zRibhacU$B0_?|j6mxz5@VAai;cE~=fAf6If!Vi4WBa__ zg2JTP0kAFj*C!NyAWFwQ=zUY)nh;=Iq0lMd?{eIsu2+C^`%kF>5CO=^7CD{%rzl`~ zw|*I7%f+++OMH~f%JtNH4&O-S(g(Ub0>(g40N|P|>wcFq$^%UQe|jya0Yhg51il}| z)j%5!)KWa41KH!X#OFZ7VgcBL2E0yTf}=*zCeAP#H&kiZ_JG58^rP71f8J<%tjw1k z!p1Q$FH~A5LBwaY0FEi|56)E{u2^i4>ML`1Dcp0R`aNsjPzGAk*lU|n-1SQ z%sraOUX+RG_h%}4`y~&dYBoc{twc{qMcAGwqq?1KyEepS8zASy}^_Ht@i~6kW zkd?0?bu`yAW4;UDm+$#25BK89;Ex{Rs84=c&2*Yuz=+2%=+=-njWk}FFBi$^Mx3*EIe!W$%Z+OW-@=ypSWV6d}4g#%Lmd*H@Yl8gtb*x!Gzls zFFX`#=hzZladn3WQ)Nb$CX$!W2Zro45E|gfxhfz6B1;7-)bLaRwgm*5S<$$82EQo6 z+Zi%%64HNh!6&Dg`eNUF-q`V$zabQMH9Ex;)(^t{Ig;x48`(bRoL1JN*K$C7bJZve zY)eelGxS(~xh>D*0mrv$B=bIZY>HgZ_juuy!Z%vxLjZF!_2!X$XHeTi(ZVR369?`4 z9dC9_wTTpO^FTlNlUM@xF6oY!w>|w6TmZKPC6sCyv#eVq3mqWAX>ZGu<|TFIl8Bq` zrAzyeBh6`<^FpJZ1NMm{=Vd_Javfbo8S43`%CbVypGt2?Yq`8p>F3`C>6hiwK8O+` z*O%wl0zyqY%QBW=oo*3b4+&L2Zg|E3@(fhTDa9Cwpi#n9J3_gU66%@w#cHMR(?VF4 zRie|eVi1q@S(%%e;$&@MI|$haBZvRK#q5$}q1&P4(F%5>rT@)TH1ZNO-O&|mg+5Ui`7v)zAAw60~VRyY6i6$Ps~ zQ-15_3|Bl%0g+`wCo8$snI51!2YghvaUROI$}|Mno0Z_x>NXUrTFSdk*Llt*Dms?LzG$y*Ky*x7yTBMGbyRN zuS0!SQcPCpxks;B;81-`qSC z#Qdk3xy`qhv-54kS9?iN)!aLgx!c(_W3)l4f_hTlS89$$MD*k-urf|;aaRlacPSmC z5{Y{JCfbJch}maO`b^6%+{h@BCRwaQw=Yw#RQOA6}-?TJXXkOcuJaRKSeM=kX z>Oi3PIsb<*{NT_V_ulg{6Zz~J>*{GaE2%KL5WA89Ffh0dv31a*FIx-eB|!%<BRgf8Z5yf{K7MvkBt`Xw zHJaKSZUHYonFOGx3qf-kY{RInckRG%_&={;aic3I@PAym`YOfG!qW&?(SI*;d@+#^ zv?)blBv|sSks~-9|J;9R2Z8{0{mVe;zlaovzS|I3E-15nV0D)>tYU@X3@43o+$G&^ zF=-&U;fSc>hRG(wagEel|F%gfPpIqi$BY-6>H7i`tC;8%aMnhN_wPDu!Er<7N&nKY zk6bsU)0TvqQ-?ZsKvGifXr4FbJk68^v z$GU*f6Kgk94FJuuOb>`x2?pcxEvSZMyI41?CE;|z@LMD#C+wobC}8^u62LCW#{QPW z7}$;iJGW!k2y%S{&{ji&JvS@p$e14!<`L!AHC1p6_I-4wt{S$nZn6S@x_FNTAB2eZ z6%0;eCM!JzYHG%67aQ;KyPH&%DqK;m`t-8*6~q{&0uFn{B;aQ}Q%eP~EOazF6U5~7 z1yPDU&4VA-OIqJdHA-6fJcZC3eMugK%lbk$lxomhNoPLo*%+_O#NDlR&(a3(r-I&P zvF*9Br*bJ{d`qnOS7jgh25)t;2WX!^wU?$qbOC8~m(;d$*=2Sm)+3pF(+SJ3Z=uPZFD+# z*`?4Gdl0}A|EpY6(Z(zy?&^$e=4(H)vyZtt8=?44kLHT=IlJhb!f3b^U+7>;$bn0$b&E@kaQQli( zLg=LV(Xk^2nft5uV(lR8Ux9Y<-|mo<>r%vNf6doZ+bm)j=RgK!Y)%qF)Tiu^QsHn6XPSJbRM z2);5ae9i{aLt6fXEFyg6d0M;|L)$|v8C zoivR8{js>%m`t6E@@y8yTAEaQo?i5=PJ^)FC7(e>Jp&peZ$k{6ANa_CGX+-S_q)Lo z&I}C^UFOpAZEcBYRv*~icsZ_{8~^reU>|(h;oO=Cz{VEisUBL2o99g?sXD-&%#+Qn z&T*HWMYU~Uumc_Kr`omXlD1s;@Y6yS<{#>qx2dDjA>c}lk!PYYb(Ymuxhy+b({GH{ z(_Q8y>W*NdLAoayA$wOG%N#J&rxm_j$S-Ga2F_t#@4Qr}Lott)AMLzq>hVhWM%Wr~ zyebo-+g{+u-l{juo4iPJ+wAYm88-5D@E1Pdtu}mTg^&i$Odic!h=|y812!0)A_zO& zw~Youcow0fQ<#9wGERjyMB=P1FVl!&quKNwJie-D#cs276}MZ~6cAL-CQ4$iLn-Mc zY*tC03*{Qq`m1cUvPMuqaDqYu#33|7fumw_{3T9JjZGg)&HV{%yn%CV5VG(~j;*JE zIDN)N1z1MglE4u=aHhvEjg;R2V3;;IipDp}C+}}Hv1)S)wamuCJqo}-04ikV+LXs7 zfb(h)inK-(Y?q;<6Hf6Bw7DP$l8ntx_pLr?_9lA{5gMOE``fjR0{{D5Qq^9NsiGCXjlW-%p}lW85ud`)rFbHK%1!qT2gK)U%q)r z_4O$U^MQ^IW;^83(N?*PvuTMzK%mAb@iaur1!~cz#8^}SXpaooWK90B5mx|!ql)ri z=ri2xcn=uSYN3s*N9GJahjm6r7*A%rsW_d!$8AbGAn*OU7z_rc5`$Mb4l$mQ@B9}Q zi$(iow%Z&DJOo`jPbnA;>zB^k(5uUkvNa0Bl0_S`t<2xmdQTb zZ?B8ngvUZHi7U7-9^O)iWo=4z+YQsOx7z)v$0Z>)>FpXykKJ0@hKGrFLa^l3{UY@j z@0vG-joQ%7+Rh168Tq22Fz2wEm}?ngPH1Z}!Z+SZ?0E2GH8LlfexEi*h#$U;B%r() zjb)7OTCK(H3w!~m?5*_(8C=(-<&@;lVc~G+Ns?PeHm7vVD>8n#N4+BpSAgoM4uR)R6KHt#1DP&0}|@!K!h$#%POuZ+35{Nf^f?5DJSg zekLZ0B=a{L6S*1DBQZllK{(OQN-W^KgAduE!)j=`s;-1Y<4h}#@S@~^t%C`5!G81+ zJc^dv#uZK{i-yoc)IQfCuXdr7LgS6;^Sm@4SfvYQ(3-Xmxo~`{CQ-*cjmD)%>(FxP z9PvW(f(?|$u2Fk54sJY{kcr?&;N4@xH9C(i@Eh}4auwR|VO^>$Y z3XwTj*`pvP!3I7W3LaxfaN#Vw3@zVY9g#R-Zh_3^GJe-o4}uX-tJ1*#^5=5wRriU| zH3G3-6IW=H+?3Tu!lIKki8O|n4V=VHHYQkE$7j*(3fsw6BVcfWe<29hGSI|!sec?T z>jIoeV)XJ4nnKBwq2!_BHzkh2b$BB5OdL(!OiV`5nk7>IUj7g|+il)$Xr!}62y-Tw zS<9Lfwmu36dk^DaAIWkeuy=aXH(Xtx5cdon$Fe*VrF9L^PE!BGJXnR{r9H@$t@a+b zO?YQS&1GN;y^2Ns%PR2h-1LSxo=cX_Rc4vb!Q)m0(DOL(+OkKh#N*AQAuW&C&#%_` z;}mWzt`y&*#tnTtdR~kRZ^|kYVmtX*Wnzp#ve3(?k2GF{@Jxz?Pd)(W%8~Qv5 z7P>T@U4ijFy@KF&lg)%;Cu8#%Av}n6l8PM~orVV`c(bc{b=!VaJ3iMNeV0r2OMM|R z+Ua_g=gam_B}P`$;E3zYvKkK>rJ}RJA~|&xtNeVbw_`bl;Xjo+#7!&+JxuXPlNxx3 zJ04>C&JH&m9&r}5JjA0o!;^yYFJKCrRt>0*sqdJXZb(VFZ=9NPmz!FbD&A8sqM z_mGBI#5z_5B11Sk_S1H{gnp?%kQ8KjgbZ891n(-cNN4ZHftM&JW^vv;S2qs4S%tWX zl@vw(@abpf>-5cO3N$5kc%R#ZsQuq4d~zIN@-r{5S%1i1(3x>Y+-HoJWX zTRn*ESbV#)=1g(dh)%89inQsH`sM*6pOEM~;sK*O-o4y+h_LDRPdkFu4ZQ6|)r_7P z`0jOYIigzB{ML0!YN^-3nc+EwNKLHG+K39jgyy|0^p96)p5w+zNB10<~|I0^5$pvvjzzl*<>_I z;HGzX_mE4pvBxX@0xk5LBkr&Cm3trF)_C3aewM8A@Oy5**N~(g&E0)|U-FwVYTwnr zd~mH{`$D7Nuxo!EK2+|a&9*BsCVE7 z7?E8`{MJKZ?mtKdjL=2jO8P_CX|uKV`q3i7)~Wq|^DADj&jbI$ZHm>fvTXdF#^<&q zxdCNYX1;3;y!CsJ@1$UR>N=+}#wyI{IO34}GtLcey2o8p3E-qNlV3Zjw=r4dv312) zyXqt_UGZ-BoYB(v+hFQO?srv9d$<3KcRW<~Mm8yWhM4-};alOINzRrt#F=;4>VfuL z&wE^}Hr#%SLh5Dq_pgl%R!*I;b0#L7!h3gFAN;(Izur=p10!y>B($)o*!NR!kMEnH z+mJ$3Ut?IsL-NZak{$$-()H9RN$ zz+lpBq;OsDoJnRkhQ}-6Q5WHB2dd{xOy*u`jHno8eUaKx7t?B4D>Ru*N_L-ASarwx z!FobSG;0&VP4A+iPjB3+w?j7rmEVilczj4*(v{;YXNzj1iC2y{7uJn8t`9k#D5MsV zzl?b^m3hH3y~Z1>8zuuj-`O;!f9OZ8P|GWrH@}_A?sqDCvf*NfO#5DE#?jmqyY%># z0WQBZjCB=I=RKj9leCWptjXDMom2DfSM&CVsC6yCIvc^8Nn7{ocqA4l_r18Y+6TO) z525qzMl*{_ON*SR$P z>CA&l1pIu@m(>#m67TWfUm48&l4Kys&hnAHtVd=CucDkRP({|5h`y-4{>Kr=N_S_` zL!RY5?jdu4IMBh9U*EH4#hAhyUBYhsa5yadY#7f7xxtdLmNxiP2+OOO3-__o31el_ zurCY9!GWKBvvq)tAK+ZFY^v}c@~mDQ+-=}jdky115`ei;?=@o2@grX2Zr+tDxVk?y ztGR6TI+eUI>E=BKZ{1C}Z<~Gkd_sE2KhT)Jz6}-LuOg2mFs)TxOak(J`if>MzF~lA zpO}!MF%{0=ezHTt_GnFVjT80gJ-aF*;0#c+*0b;PG<*h8=J%_@^n21zQ7)7p!Y`h` zpx}z{w@LVB@2%0#YlUqmrpOC!hZ425JZ@uJi`khMtX9Xb)4JnzIw)O?L(b`HrezE~ z+F}{Tx=?nd_U^Xz8mi4@N83{yOwhkX+%__AO!IHpjdX2KY$gviUfB?OI;-P~_G+DMGjurqzmy&0H{o{_Zu3IM?ycZYimfRC8h2q{zWUEV`b&A4+ow`iGX# zw#p+zo-KOTA(joLkPlwqqh}%0gl<9uK$TY#EVP3jfM>`H13eg&yo7_H{OG}H3CqvD zLy{+@LCZM?+an3_UB0{H61WpV^-y(!$i^Qsfy$c5bP;{XCRr!4{VHn%0jxBt#?!@m zL6Y+1hy`{iC#sX+5p=wly(T9q%5wJ^yUdJMOH4`+`)LNn!Z9t_*u19ryb15T#?R5w zTUKO|BKC$EolLzQ$|7hSK-_4G>oj};e`oH{ZaVd1wE=4Qya~`g#KyRhTqyGuVGiU_ z|Ihk_EY$qwCbtgaH#IMzQ0D6d+?rd9AGbeb0 zOq$Uu*zN14!AfF9TKV=VI*tNHh@7#gZ*&X1M>=XWKpn$^)14h<;Vt(5TnM~q_TxYy;>J4Xq zW=6N^qcl9%R4FrnJlD=|F4n!bFf>Hk5fqf-9NI%i&cg1@3AJ~pCRD5I4q9S-W~xfa z*l5J!oLd-Il4hN`V@TQ7>0T}JtYqwTl=;xr2q_SMIjNxRf1!Q1Cva#686iD&O1%5Z z5ub*wWg%>{#-ZKo*H~`;DwJi^l*E-;9W&%v;|q4_HV|>MJ)L{qhv$H{ zLmrie3Agr{?t6i}-f>f>qiCTQMKeuQZUS$fE!wbk|I-WxQct%j6?Wjop*R64ep0GWeNCsdOYhYD_5>i5|(e{ zN@y64nCl&x7Un*I z2_`zLJ6Sg$0MFx4MtXi5rVYu|JwrIwkw8S?!#|7Yh=5+e<4~1G-9B6UdUmp(nWk|9 zoXGl`aXBK5KDs`%qwyO@!cdn-YjS+=EO>9gSU4>Vtr+*yWkC!=7HYg-Vy)fYLHZ_1 zFw~2~JqybgW`nrGqYExxbi42ZA!swkXIib?%b+>uXfAeCU>RwNv>-U~G^nJFL|g)= z855ft@v~I|o1yeFQ?AUIXqAqwEWlD>=SaPbv2Z32G!W=!B-@E3PdpB^US$(;Y&j8- z6XX75`Pl3QczlcpNsL8VEVVTl8N?=6T!8>h>9hpy)8pp0evI|wgq|L9BzAP4?#0_- zU9rOP9}*LK$|g*r9cNWO@7y0+JWi*VA)A@~&2fo)*)dAt$*bOX*IL0#)$z%tZ2|Ad zCTY7FErIv|)+)!Mz*E1l6@s6Iu5hgJFQ!m^aSq93ryQ6Pp~mhh#%rt^xu`L${oF{n zv$vU)cZTm%k}0_mGHJ2u6SXT4=$wI6ohO`5&!OlnS0ZNyuI>H za9dQ*#Aln%U5PLfAJYowZ`A1-E%ke9yzd*D_!tvwdBMtwIHF59&m{-YdoMML119f$ zGtl|vU2a->D!NVG%~fBymPZk}czWv|qok%yMnxQfD|hMj?e=dA@H_OS+1{E>W&h#6 z1m&inSvwyZ7@R=%j;(pwTsxIs+Ic^EH}Q^ioTk-(Akk{8+|vNA zMiwm88d8p!Jua~-_;6DF^Ka+ z8Q$QPZhDl+euW~`tvmpfyWohxZn$(>*91HFBTgheO3Z6~F|dN=CEL*GnXY(HgXgcq zHX+`t{j75b&f#>EV2N6ie}uW*iElvdUx^6O3K`9PtOu(?GEIf`7QC6Q9vG zf6+*C@$zUMDZV%dW;35%M7>0)D(V!6G3U)eYEAQDQ{=xq>#Avv$GbWv|8UwMNcA&`&dut)c5l>m7^|3FkW0^Wuf?v%%No zwv1P7&t%!P{8V$Rj1i@-CiL_r@YRCb{?1R|0$*s`mXXyyt>85T3)2K&Gf_xEd2(hA zNVol5atBC`0g{WJ85%%?oWPY=$!+ETA5z?ZmwWiDyIfF9(mrU2ZXzV&Q^o4f7%)2@ z22zf}U~Xd!Qu$zAB$I@fdP(S{@c32l_nAePtlhZd>DTX)0geM8LyiXkX9%c3=*KOj zi%5PcW0M6krz{u3Q4qp19LgvOZh9SyOtxQ69Z(rdC@q<5+&U)55~HC(P{6_TLy!$ZVVHc+>W?S{U^#gyKz6-1pd_HumGor=F5%Aj z;mWhtvSq|MTRx1X|6l8AO^g1qcxrm@mA-LIPxjW1_s@<{m z;~#2?~i3(wzZf6@WfXMHDwJEQOaIBUBfLd_$z!Bs@VF78aja3Zn}is4|Dfhh)n?i zKUR{`FvfCiTqL+zXz5Ap#6A^_?{Io4kunja7=l&1~zxs`l8f)+Gd8e8%~;+42GdO>vrQgVSk zQj-J?26?gawXFa$b_#((13?akb6dtuR)Y9VV@-po>tZeNPY80RP82!m?dY z46<43AclMh%kTK$8iCFchc9(CRB}q!lADujQt&pD90!#RstFju@_`R->C2k_IBG40 zOJfpb2%-C>tU=u@M&dxzY=QZ0ktnH@l#Us8=jOVUlc=(WYGBC*;+E$)7hfiQGdscb zP{&T4NWKwWND1M&NWh^&e1gFe39|e%kCgHmfv7kkdcO3b%F|wB8@*lWW@EJUJMYFk zA>f5XkaOu@wT4#r^3#u{#)AR0{nE$JaSdcl2lP&?p_!Qd=biHeew*;HW7*T~^k`L+ zg!zB{CRJNn#)6L>hRoC6TgiKYeY!V*7B|{!`zvypHcJ&>?%*I$@#UHMVI~FqbH4<4 zg!W6umOG~R>c@f(l7&KCmg%v;zHd(vhe|ox7;iR6!aE! zEd6EcPz@Obxur>v#7!>uF3b?w1Psd37_svyzZhZt%juD2OMu-?UK-7E|7%Esb}nqc zt z{~VtG)P6#zEU2u1YlCENCTPC#()9$uKfOTSg$SsC;mXC>Y5Z^Z!5l8BLbxyapI{h6 zi-mjwe9S@FDVZTRG7bk%S|{gB$env=a#)}b0&_DBaRSSu;T*JkB_9rzA7;vu^(H~< zkZp)lJ;(vFz{f1MezX_yXsu5clqEyo)pit{y#Irl2P>6BwOIbZvb_U_2C{d|gLSX| zI$(>bNdr2#t}qiiIjE8q9Y2@(9_Zp)v;LgAYC2{A9Y1r@w4q z@0knbcEv&K#?5oqrOr^}K-L*D@LpM)Ow&$}H~q4!Wk%-YIsl=y=ETLMAU*bT!HHpCXHF z1@Bw+Y`U7Ll=#e7>{#Sz9&&L%-Y*MTw6!%oUYY{zrsP_(rq>NN3$!Wkm0nO+2_7U0 z+&)JOYmfYKb{u$)0g@iWQqsVmj>u7L{v1a4uNjUm032xv5}5civ#rpbnMi@CR4+(o z>}mz_{O8iRns>bO16A~cO7H-glNXOWyR{aWy3dbWK(_9}I+|2{DEnAKhS0&aM0T-_ zdR_V_9My?0{sf&KYU)XvDHT9S(?gCwGXiEI$QxIT8#BL=T}-{yM!l3^JZ3&L4t)iQ z5sIjRR9yCkINaVG=#S`s8Qw!H_{0oU`Ma#3P|zUz$G6kFoG52E|ouc{+E^R8!I2nS4{gN^Jy^DHNbXGe|w zrNJvdAXiKr*JF`gGlW!F6a&nTkzE!tTfy-jWX^&Mm<;hXqM%;^Y#?Ew^cQ4M(qqst zvlUDsp^fUG#wjL{*lD(hI>%GM*cRGREddW8iRWE^%2)1t=cln48{goEX+`DA?}uD> z!y$igzo`gNo{&vRJOP|n5~fCw_E0YK|i&$6x0-+p@>_=FiR7!) zrHFC^Ss_Io2Wp^FYvSuCz%atFU!CEU+E&sRFedU=66k?Sl-NcOwn}V_c_vAJt5pzE z4!c%mZCGY_Mq@g>2$1djxU>V}O%7i`>zP1n24o`6eXs<7A`CfX5kj;213mW%P?E$E zmdz*Z1Du2`$6qkU|Ncrw{E(-ePaQm8j+r4q6lb}FaE-L`o#hY-s?ZXu&L)ongZv-H zF!w{Hn=AE5Txb|UL(}%ZntvN+5Y#O4wt_NST224Kt$;j`|FDSv&)e=3j5!REqcF^; zxa1^wC7_h}>}}43v3$vuPbTxM4>kF)9Q_}Ol$S>_7~xCENA7DGEg_$YxF@l4p8kJ? zaa7_5rv5*?1RXFga z-7wms_g~@bM<^?+I`jz!1sM8ItNNF}J~go5?~VWX0g!kv*b3hdZKduFb&(yN?5{3Q zkp|0pfY}3_YhyML@GM+n!7bulxLz5XG@};BXN+BA3)<&OnzCj^L<+dvZlg>Kuhwk+ zX+F+1{>go5uuZZp&czOT7Hx|zAG4MPHxCGHkd~cU3jpxM+gT?EE{vFBa)lvpy)sfYEo{_Wycl-zr!!`#IRToBKh2E_X2;8O z+dD@rKt~R^3+4)FdCNuP*i4IL-Z%q^a zD{2+lR5FpH7Zest(u~qPxm_|LC}!l$`UtW)mjE=ySci4MH7%-dOrdkZO=(uDSsgCiavvx?}jnHNCG?$1GaJ?E{Mgg9Hy!QV??D0kXc)gXfh9wWTGN5WjA@6Zu1pxo`Kk z`hgDHLNvDB4tA_)r8#8d8PJQ>P9bDl>_prtz}INHlFG-RB3n54YC6YS4Q1OSZ6tPL z6VTmqb9Rn#KZf?PyNoGqS)l5`DUL61k7r9DunxvitRyF}G{)F+yVhaQzm!Soxm@`| z!0pQcjoEJ53A!6j%%QpPIvM9K=d?fbiPvT@L0Q0Xn6Qmz+$U-CQsZq)@$ofViSlX8 z%?Zq(n;XZ0x!dC>AyW$48DjY`Ay5udeARhSk-=23NOR%J=gyE*d`NDUV5!h;`Av+c zxq4vf@x(&|3BK&8wdKF!&N$_|0jb+w??1obO?P|>y0*dC z#U54@-2Kkt1+FDd%mv#NRRS?BZGM4ki*@0>bBZTr39{Rz;GZ&-1~tcLrqY_Y*q1}1 z2Tsg|jbfW=HdRHzSQ#n{zF2wIj!G!dt#S4U8X?p0lbv=pmc#;2^~K6IZoVpb>Zum^ zkxhyf&V*^8OU8smM?RX8)zo7xzBICwf{m+<8@7F(5rBctU><#@xL>2Mh;jpk?~F97&q?W(~Yo3Rw!WN;DX*TO;e&=toF zdFC{{Z#{R$vdu;urb*Q72ZiJ!Lf+=~+7eA>K(*%_EV!+bQNc8E4Ad!>RRC3ki!dB% z3OP2~BFV|qP7^_`lhZWPVKc!&FvR3ij*yrVI3xrK1O-rCc>jtC22G^}Cy6c@VLs+j zKAL-uE8>G>P2;WNClTdipU%9SpAr5_oPX2+rO7_+uccjZv&33)@b;xuVaeGymw8AC4gC$V{#Rn}e{snM?+92tJCC?f4-lY3zk2bKp#dC& zoMr)Q{s$%qrnOnS^#gU70Er9!1OhJ0KJ{M+ojg9sF%PngHI)#>3B!0G;9ye6=K-g=j9%zZTo>-a}I9FUk%f;h-FJ0&T?=X z$&foJ%sykCq4nWb!Z~P180b8j7<-TlWnfVSgxnJ%XAE@kqoL|XrmEMQQGDCcBj_D@fd0xkGr?tpadZ;M5!U~=Tg(1r`T*kx$P=(N z`01~}=>PvWbQpxnxaxRF^2Vahk9wJeO@p2M6u)s0%H&LKhD>plO=GlbF>j-U-zdYw z!A6${AaMR(7ErRbHyU@jYh0*{GW*ZgFV1mdSW?~3_s-8&^Twxs**8mPxGs7V-5SH6 zD^KowXC}r+xs>axNPdBgM%SK2? ztPsvA>g~6`w|y~l+_qC!BK>Y0U(~d7{hDY%h67^!Z;i9Y*o?TIfG=6RozJeQH}DB( zg6icqTp2TDhkN*WcV90iq%}!EgoB1LhN@+7%szMrv_;2G(&Gq zfR}RpOeqaY$;Y75C;BKFLtB9=1-`^x{H?67jpo0d!j2Th8-(@w7|3R;eR$Lu&&01Z zj!%zx5Sg7w-SHT%t7m$HMUhkiJ~>-_N#d2u=-qxYl38!Jsq~-{?ApeiI)xrY{3`?R z-fQfgGFfzDBE@qxiQLv!HC3cHFmQcsK{S@1aX55#ZFTTF`)c&UMA`cv5<$P{V2d~} z1ar-H1!ay8ywLs63Xv~Xi9v71#>fZP-y1eC8omXF_mq8QBKfU(0t9*}1mRL12M6oc zKG|nREdfWnYEM5*x}s!Byg|gt(B;ICR0QQc8dAuPXFB?oN2RdKNHwx9D1y#5d(y@h z(avexsfA*iV$Zz}_a4$>=pCu7&wPCNH@G2XCfK3L`Oa7wilJ7GY63FbyC;x{bvoEt z@s~tB;0CVwb_({N(*l|#k5SXr14es8F#{Yc?`9^in?RAYOG?ebxwk|hVe}Y}*}196 zgqP$nhiI9FncmX8%k@Y_KXG<)BKDw&*_$GY|0UaK?sB$RwkXYe0Cn8-vH#wEKuNn| z3(pi02=@4AW*Ge=Z{E1K5P)_Yh<}L`;Us2uDA6nyC+g>$mld<$kf=oz^~v%H6ZnB> zN{3>&AW1rPk8s&})gn-FSnyVRB%!0lS%{#uG0D7 zUNolBB!#_$8$1r~I;vDc3`+?ZNm)2d## zRn7%e-+E0C=2S4-1tRh7wcj|0Klh5U&>UU(t+}8KrQme6#z=B)H!O?Xxg|*Ca)IZR zu6jyrAI)~fj1G4fs*gASR$%{y+RqU*%zbI4uljhePh^aWQGuFyV80f-w^0F6Dhind z#xmroTT$m?dBG&=F$xzvF_Kf$zgGA73M@WvOu4vb6<5^5jqKnbP?vmhWt!W-1wsjf zAx9Rf&&(_)+}TN}(_3YGX9e)^|30fh-*Vq4)}mOB)XCN;10egFZPy}Y?$tOkWoJk6 z)g?aiGVBX|Ae|k5Fagdso-zLsp@*|A0y6MSY4xh}?C#<6vfex+p%2iCw|~KNYheF! zmtvk!0AwtP>xilsnk#<`DWiCx?h7m{7q6z=YBV+Q7f^y1>tZr+>z_^b7Bkes0uxMK z-kCqP9Rz_7iYsX_FI{4)ho5g)uE`P0*Ipe7`Ln%46* z_^#?VTK^qF^3pt5{Cd#f?J)n9y)i58+#!u*i#G!g9P#s>?txROPW=GQ`TT1a`;iCr9g4r1=j zWmNmj<>JZKKwiERWwt_kyat>SDEqoz?61wa&$$3ESc&%0oHSf#9*5i=SX-J}GgVoz zSHi-LgcA%nXEE;zHNt%9-y?9YeSs;htnilcexv#yaE;@Q#8tvR1-7aIv%K9~(pM0O zpP!wRu9mJ_T5JDil-{dIkAZUmqd*C1al(1kvo+}m<0=2o!gWv1i@1uyb)AaN$w$ZZ>91@U zQDwil?e9+6W0x$3{T(w3?Dp4^I&os}i&{o+ zU5C0(n~%4Ha`^krhTCWGC;d&eOA$L{%q-Z?qtAXX=*LVvR5Xj znud=}SdSOw7FezPZWa?FksxisQGZ(D1D%?Ng55?92Hs&>hpjBX8O_gUde2#uXFeX8 zO7Sbr+6ng{<-2%!b>){2Y1w7HFw~_bZMS;y=!I!0{9dIo)IEzR19&2l$gZb+X=-n`nDTO=uRqxXq zEzi$1=d48$U9PK7AYKC<@)dXVg+xD=NAM#47-j&GV#=qvo9qZ|J);pCK4)5-QP<+W z;P%U8c(EVCG<*@Et}7YL)pj%KH(bM7yyVpB=6o7{5dCCvj!P^#^tdVboTSdReO;K! zf3-?k7E>ypAHQ}sAmJK9`0@mDzaM<-?x_NP)=pEuM8(Rk{VcTAXTHPD!Mw$}E6KHO zdDq(uRbae~{h#Az>rd@ga7{^gxH!w?B$m0m8xJ)Y`1K^OVz%6JKPV2EH70w}SS3%z z;?<<{#RRmH1T4RWX6`LzKu`T^!gdeakV>-QURprev1gwViUW~jQa>Lf-#Eo@de7*n=9)CnBcgu=OI?@A#d2eq1B4ymLw@Sv(ld&QbH=G3j7maF$x(Y4dOLWfGz!Mv#|*uq$u$A12G{^=JFg+DxSJOY2onJH3q;su% zi2_j}>B)l998ic%4AkBmS={$2`h4+I`uLs~*qX8IRYM1wJI=bUm8wK4sulqqayJNv zozetrLix>*&swq2c#P}m%Q4a?d1=4(dv)7=*WlB8Fr=EHyXae9WraVf!G5(sK;BSq z4-BBpf8}7L^fqxJxLv_9QTw~7nyLXKgEtbzZO@c&Pw_RR$iXb63|J zn8)e@P)op#s7@tr+U*{49{nPCvNtw5<1W`IeBqb9O@-U+-G`;b`zU)!L&P?iN2PFr z$2&hF7!b!J8x|C_HOzBYZBjG%${A&=JT*Ywd!=n))x#^sE4^8qwqGY(HhIj#Lh}w7 zik;B??@P8(YIYtpY<|wIMXhp)!@ag)WZal#_;q8QD7qck*4~+5`b_%7=pNtm_Z0sq z-$W?>e(^kCO-wkk0}O5aHjOz%Yvy8FPYeFNLzC2s{eHkm_xBBVNrana15U*rZHLBV z+#hHkVSe}2=%Q1kD(w;F*5xNezcP;>dWbcK5^PG({a=0vj_zqtxKEz*-EpJ)R(Vgy z4`t*j)&8-jUGERxBAHE*@wdI|{ig1$R`3RAVrLun_;u&lzAR0?#xP57A}4> zi$7opRz`R+IQYz*pBeL+8Kooff; zZ}6Q*Vz_%gVGgJpdu1giAhv}MMmoc=u6fxIwa151@P9G&-tlbz@wa$wN{wob+Ei7| zmfC`9jZ&jpsz%iIv86^3v0BuwUAw5NmKwDs)SfYGi(0Wmf{=*((tGdke(&RveSI_guB1M?IXLzvA`buH(WXD-m=P z=jU-c#{=Z=Hlsjxz9ngs==a}F7ykCm`K|+}CQ6s~HF(T@l*jANi!Th&&Lm<{pCZHw2p2qFRtL%6CgvGRaa0_?7Rg(OiFYxNAs865o-9{NI%@SxV zXf4>MfVZ%=G9(aDpx)rW`=>CYQU0JVZHe?~nK#P4&TFYk#PRMUu?SbtVW(=|nqe!NLAQiBtm@}A<@mTpT zB44G2oVdL|t}Bq?KSzBkPWaKxvT!c7(t@#(LeH3)_Nhm}zpW_a5TiCBlajzBMRI>G zUPu<(2k_K&S{~$Eu29DeBs=wh$vFn2uvrfHAqGKtoqza!nAd$)mwB~GeD#TDB&03JawN=2u<`24PEclvi@EAXUMM0;JSDG?r?8sO7xYar z^4EsyP~}P#DkfflopPzV>UHjN@u+II0N5ko(>o*0^9>8zXDt5^*Gn?u`)(~mIV=Nf zlaw-spY>#U&J9kVcAFteSCqB(0|VdE3|KJ~crP`mw>QEhf|gHgL&i>Y1LUt6y=w5w zkP3dI|N3=4J{N9Ra}DPFxCKS7@b1 zWxI7DOj(v~ZLU|#0Edii9)CxWPvLzB9TonW#vLIzHF&r{eW9xLK!xnb-bM_X)A8?t zS8gf$mD+$eGPZ@LHD*uf4vPk(H-ICis;UyUw&*AqUYBi1H?_up=fbt@u6>(t5C zH*v1Xdt1nswItQOj|S2;J>g*_K;f4;(Ewa5+5Oj(?26CXGM(|Y>j>Z9nIa<*Wicz_w4>R7;}{ds-I{XCRTyyjG|j0 zXHaoa9+Ts-%CP-tAzJ!>*-GNi%%I*$RoU}d zZYGo@sS6IZ^Yyeo;+H=W7|c>x-kt%;SFDo&3x|_`@X%83zP9)a7BaS(dpY?9eO|iB+DkB_SPbV)aCX*Y=+a44pG7B{0$MbT5)U} zumhdSjw1((s80Buw8giVN21cB)?iIZU!0N^H`=+`4kLeGk;xFP-86X&Qx@3r1U z1Hut&R0hQfXjEBD-5Mm2PGQBrOtzsp!AV5)esK){(+Ge5_d#6$%c|+d^a9PbPGTzz9 z0vZwd>In?YBYDblG-~3ABCRNogT-bkI!A`|XW6oy(oHLoH^K)Nrl{X@Wh0u|?Lx`> z115dx+t@tB!fgxW_Jiivk9Ai>ej5LbdwImQXM|*eh*OunT!1NR(o12 zXxT5e8|FeQH?#ViQO1zM2>*90S)^aj#!mxwwi>@weNo2C#Pt`jxm zJ<}gemzC!+Rz3(ZhDzez!;@=5Wv@WK9Unah<_(c&Lj{8)>RRyI#H0)6P56%~nR9IJ z_!f8g+v2YAPr3o=@=SKNK5z z-~v7)3PA0LY;E#08N|XzLPjY%|32LPv+$EmaiNIJWT1t;Uk8`n3xYZAHrxpMk~bwj z6h*-Po{bS*%-yl1K~Tv3v8c8wq&$&yLbNVfe78<>9*-h-IcUu0yia}w;-8=JX7!Jn z4EH~C6E~~NHdaFJ1&w0%s()DKMy|BE=j9-$_6MikJzEDHGR3mRae#n10n9#!^OiOVw}KkD(79s zf=dlXB61`E{-Tt}wA3XBhvYmbDb{&{7>V!f3*mAhZAyR2b3-Yp)MNkG>7cYCloOL2 zFWqui{o*=PxmLgS0~TS)g|54vI<%Mg8MqOPp|x2@7cGmnypO9STe!0Tf0AtG?>Fz| zSU!>P)ud1SKP^Y;(pk>n=Of_cD6kKm;FNHD~lI;t@ZLS4<a9PORJjD$=pH54c{0zMIpvBK|c+VB;B(0RV0h;5=yr(A4&RN%)9CgIyK-;)t z{%zgj3&Y<NO>bdoBhEJATNvSkAOq zNR9mJKwk77Go?YaT9gP}2D|KdKbAnGfy&|UD1sM4Y?RZQ$hq7dhiMoq*#}pq6TAY9|&MCGCd;IzpVi1#> zkjzf(?f_tsE;Nw-zf3K>dDLgfHEYxZJtMf7-1g8eg523>bVwi_4nPMKu*!=tAK=eH z5r3m#7>RnMS(-`Vuk9*8q*sTJt@O^;@X0;gU&W6|<|+|lv3L84`P$>j0!m1XWTO#M zHcEGJ#yhwqZzK9{SS43br#BZdVZ)oNH!uQCbMjj`6Io+Krgx=Ljm3E*i2DqYpdYW# zJ=Yh%f)0!ifUwGANLKZ)1Mjm+#sYR$G0mq%X=Ot!p%?mBX+zd}B}u&Gq^(Jv#C%t{ zI?6s*VRt}&vWbCumfR_5UmaTx8oC_h1kOEJ1wVB2ldY=>0H7)uACMcBD~5wR+<60)$L45NQIVjq#8vRfOeeJ_Spb4W!*O*nJnl&-&spPvhWbb-NeUhc>P&N2jXj%kMCQ zwO~+TLCebdi49tUwC7oel0JCt2N{KDj}ZMi)R5k%2a&5-lFO;y-$Lm7ImRdRC>i;X z(?xqYcBBFEkZ{RZ*Is>tP(JzYNX;kFI#ccd0#+4UbfWaZr~-bP;TBPX=~o8u~GlJ{`U#&!%b1>DM$1ZOI) zlOMI(#bip+-||fFwSp3g-_K7rx;PS9GYe3EGh_5e(`W_yzVOP=f-~)+$ z`aM;ysp%PXrI?OW{~n0hsK1*9#8fa9fcUtV^r>QE9_h>wI=%hL!@Vf%$|9GjfkUck z-ISEmfR>Hi-AH$s`%Kkw$5TsvW3aLUAw?{oa($rIg*P{X>ylR!FL3Xm5<~Re@GlXX zwW_=RKiHv2CZd5=>)L!d^-ove{j}WEmvgoyd=?FB;D_9v_bBuBDmAT=6mHdh8d@=& z1AGC7H*iHfaCT&8J@Rb4X992*n8-v>C4{TdcaPgTDcsrLt|f&x@Do*zk62eWXVW7z z)gsi92$O@ZqrsNDlOV5+ghxrn1;^a7LwPOYh<8M{K60K?_E5rtTDFE^4;)e znQ_Ev>(v4?ByY?W_sd%ko2$$ z%41$%yegNgt!Z{5!)_i39f)F9L$U5uU&ZFnsk`V8HugcIg}WWfDOqS^$|<_@>#U9n zGIej8SuiWg1Yh`lkvxYK?>S(;Re2FKGE9v2ifkP`&&G-tF_7zSby_}Fo`x_i18lLc z-I6mY$~$^$BP+N9yxagzsP7F~z4XGrP~wLuPpK2Nr|v@}ryMGg$~$qIf)Kxr=Vu#7 zgCXy*rHgIO;lZ12Tk$2si&0=WE8#G|yf6WMpQx6nYOs?vlm)8JBBT@$k_i9Kq>Q!; zSuG3mn=4^b7B^OK5iJ(}W^|afuu_e9z=JRer8@@sAeL`)-A?Lb#y#qq>PHY9l6xoV z(&&H%Ms?mk?0fEcPJojL@+;wC21yU0mlsp;UK{}0L%AKu2_K?fJm<%%zWk#e#m?oS z$+up8ExbZCvYQ&*Yzr>rpNT+jwN{>&M#97kg9`wKn;^ib3f~y@GvVV z3&H&fMD}hhI=)(_q500D15@Tbf;&0pi_V^ccuxLQLaAuD*Pr{M=$g*&Pz^FiW z<{J&4m9@g94*vCl8t>L!Sbf_$SrXwn+kfA!0u#6lsaEAX|G9W>g9A9a*vj_bc?O#a zKrdP3l7@yXz7(l-ViPS7^FXcW;_Zq&^2pWdTRFRy&0!vfXc9V}<2m~Am^#DYE4jio zW`C)U3_GV>L961R9Bp*}fDL(Ukm8(cR{mY~kfE+0`jiEIW|wshe*UMB1~K&Dc?>y{ z+SQZ}ngd~Gi`EG2SRvyasC3myQhuw4tLb}?G)!#?>OaG>qT$qul_5X+i5`|eN$;E| zFMB<9p!Y)#NqB?9i`*z@G~7;MH_JJh7+81BY?!&7z`4e^jl79$Cfz+~MKk9}!Rp;} zPM^4N5-#B6R_DRON!y%U59%ELAeVsXsfl0nxHotx&@LCbTYYoUf^1M~UBMq7KGja8 zbI^G=LnIx__G;6yG;Kt2U!%DNP>aRNn$d&-!Ls*sEvd-;q*}*exUDpmamfVGY;aVY z^P0JipI8aEE9SCx=oo5g~m8&1?b-)-jC1FDZ5EKX(ny5YTzNP59Zud+~TTUYY#>#g<>Gm{@`(w;i=?}nvIrSyFi zH2?&kg5kwd=0OF?ZH7VCub)W~4G)M^!_If~_UYJ7IP1&Y${)zWnKnNrIXpMVlzYCG zCX8xA=ljwhvJ|l2cF-wN@zy-YG8i5wngDqxS%S+Ksd={PR^|_u+glg5>L-+3aPC^RK-K6bf@H$$yI>4<+YsrwtE3;CIhR$P^vnb-}-7i*-r8x^HF z6>bNT;}}>9%mjGTR+KxRAnD!;ukefA9eH?LbL3%p;a!#j2Wre$-2(~Aei#$=N`=H} z8UNEn^mVP@<}O(v2*aouZc26LojWF4B}H6B|k5LfGKyBL$lNqo4-c`WCvv59+i4jFc_%)+S%3Cf-fKimkc*<;xVS`7S2vMJkh0t&Kw5BhabFEqy}&(s1_ zvBgF?*=dD}PQ{2s_ffE4^8KSYKv2;8eaTZ1QV9@}uLBQ#yJGix zlvqMVx7YVx1J|+w=6i9-OJ+nbkE?ts|1#YvTE5G9UdwdRf1f=>GH6u2v-X_nquq44 zqGG_3DlqiO>68EH^&`$3GY%aaeKHH9EZq)u@wJ>Y34zUW9rg#2Jq>n8*IQAY+dYuU zUoRm1YIM3E+~vpuA2)MiQKk3KCr@%LH;O6`;I;oWCi_q3kn=4+2r2~=E6Pf*FUG34 z+)g{73aum!KGGRRmD(Tmz#RxNMWj*uFLKD{P*9gD)+r`P>4+u!M>yR* z{_m~{6JuXYM@}?Rq!A6Z&3_|SmTQm5jIW?$eHqkejT9w#pTd=LV#6bVe-_NTs+C5| zmZ)}1eG#g5!T#O(g}3s&6%AsXYkNRj{2XjzAz#H^V+r_rlwzq66vWAKO$f8bU-m{X z_*RU3J@*fz=D}VXk!Y~%`s@W9t7#M!^)>}X1&HlKDTk>2T&pvyxl2Op+2>+TMfRadi1TsU z^%L)7#&vYU@-KIbKtcgToM%H6I$?iIG3MozqCsI9>9S)p_dP+!CS7i_f+g~6sC1P3 zx-p#6sl?08*er&u(DP#x6y_R$4cj^Wxh?-h4)#IChB z?KDWhL?QCgF`dtdxS7Pha}^zxWPki67pxBz(olCokA+reOC4?7!4Yp}`LI0SKabqWX21 z@pf0)XfymA@LU@6cx{m(x{`Zsy-)oSi4|hA{-of-_K4>ohw6MINLlq79W_#iE%|f8-A%up5nu4<@Vo z&>T2PqD!8@%s7<>61wv->`~QAJ|$ZJu6F&ovu6@?WvsHB%j>+&qZc}U2OCAKgPK3v znR?C2FHU`zrx=LP=x9gbm9j?`_JE}*pV6sj0X-o2}tWvF|rl_7ZKXT z!GyuA;IarwyFI>Ixs?T+aT(+gBH!nVGfDO1jf>7}cb1$= zdW~w`-yr`g$}-npD?wK>DyqbcU^6;N`F1lm{U4t<(FF?!jrxapj;OR2_wVLWSjGi1%I@Q2(LTZsJ!4%=}VK>^>C_Ob?b>hlt!a?z!?$tS0+=6?+-h!)|YBwhSc2K#@-94xv z=kMzB(}U^PB7S-NOfS)Pdkm6R`tH;ru#WbTAt*iG*!rLEQVBBSp%6y`uWOD1Kj2nN~F%3pOG zJ`yw($&c}wuwS&0f28bk>#LJZnwZ?x>ML zvMwMu{-D2%M#S8W&9qigcMD>D-tb`2v|?bjelykjl5lcjPy zTbFM_L!0%JicU#kct|oJt_?{m`zj(N)z4SJL>kXYCY07DWK3)&Fp@3=GiWIa@VDSt zYyaLlPIe|@MbQ6A$|4;?-vDXO-BfDagG;^76^*`06t`wP{ToW(fgu(HW5RC}z81v&cBvp?M`anPs|?{tDKP}dLP*Uzj& zV1XA^8Fgp79RQ}Me*wZ8Nv6z9tN3w?CpP3ovT|^dEhgH}g&#qs0CD&8{5KpAP~FkV zp%Za*=RWy^D-R(a>zzmE>Y_d)MU(>JC|qnvW|&@5wJoD%9x0GQ=7wo0IJ1IXBWWRSttZ1%64`HPa^Q9>;e*(@RT zjCn%fwQn9-c4)N06^1>#p(M6t;Pc7uvbd`cE~Wrs(>NbK2=Qql9^_GozcTEozxche za>wBdCf*=xfzRyAmn$+(&2NsS0)|RkCk&lw8Gg!Wbi4nlYE+G{O@=vrTq2CUU4X4c z9cUbB$1s!QRc(JAH6k5ml5V+c$Ah{CK_aqk=#r`{UVQWc@Sp1$FR~C)mai{f)i*Gy z0l$|G0qYhHU`P&W+mDp^@L(k2@1=axo2jO_3J;N9Lp}6u;KgqP^8A9uV|6`bB4h(c zlxK~|A-N-GO{{mt-5Qi4Tv(i!u>kO3^Xji@eS;I9Grg+`#%;T>xA|U8N9H`LZ~TJE zwB+Jmd4W-iU7a9;e63@A91*Rw(P7 zp0q%!18Dlq_zQD?GRnH`o%=8)(4}C_VcUVSCrLxjK zU#R6xfIKC5B?h~Z*`I7gb<6c_kq_@qbc|(^R!Lqh3T2%Xi~9Si zW$zep?SCX1m+v10**4^2uUzwdO8JP|nch50>v|+@y)U;UkTyZH`L-;D#JsJORe>hA zR zUX0Hf@l1FZ7#(#ZH{g=FJF`S@1`InHp-qxqEr}uBMDSVN4&9NAYiHmw6JfMW(4MoeHxN@G)zSxewHjz^k z8(pHvAaODXP>`XGO!xQBzY^i2bE^d@8uqyzZw^0QlL@GwbP@#()7Eki^~&&VD}o+R zJ4mx)@hIpe>I-gyk4H#Vb*$Q6p8h#e@>Cl^ft|kyd5nmFxiT7_i{OaWG z7%k_K1I)1W{yY`RkAFKIpilGm%hKvHtc26QPR_w{IKwkLtv4c3sCABcX*VtG`IG$D zZbq1MStD0wMX{W|^fl$vP14Sc29YqZ6{4^f;xv(1^E(qFo#dJ~SytoFR?80JvHIHl z=Ziph`>ZqsZ>^D|yZe&EzVU5W<_9&!q^H;wwaF|+IXZ>ynJ zha`;(zP_+cWzGhhxyj;z%Wan>Oe4~*HCsR+!@K;%y`=N+7COvoum6vZBSuwZNf1h) zj%W}^#fcRj9JG^)4+@-E~G$IxU;FA8%n@=9_voC3j z*k~M0$(86sz;zTz@cYo%t$RaNK7|IhUmf2AW~0%I*a0!92LC|K@nBg=UqMi9$Z?I> zYou`XcDzyk3(YLzJH|7(->=+6$iqvcu*b|}pPGUbPMgZK#mx}hMGBONW z8`*d!QNdH#&%78^tX6aG1gKMXQ&_HUv3Cfxy0>BPt=1}FmGy0xsHh9$DY7T(X zd)ZJyBNcO{MqQGz-=#(dn<`ST(hrKaW?a3zaP$3Ygq#iuE82NfKrm{Rl$CnAgnlK# zIDP5V={+vB_TtVLIOAGB$SF-*u;UR&Y87!Z6}5iCXUXSY%Fh^>M5{X;IYY@g$MU4W zLdS#Tt%T}#;cfgIk~O8}oo8`!mZU9Y_A=M$lx1P~P3HDfV_w4L_%kh5>i_+U=YCRv z?Mtc9hC}S_Kc~`Pn1+Ew1+fq|q5fE5hBMm>5bTpfzYTzs9R(?y zmU^}T=`hg=4vJUNyui$47v0uwe^rt5@mLO6!{h&8#dxu`?}ZA`MK4>xZD1O5Fz~iS zZDDf7{yK*XYoF$=Yg$H^6(JcTESnY>oAZoV7mT3Nb_U6&T0~oZqI_xg;K>9N0|!9$ zkpS3iU0J}|nhJc-s$#zwXwSZS6;Bt}jpF~jK4qb&a>RhSsl^Z$cp)#$Jn%4~#G;k> zcJ9YLW((pAi=LM$W%Hg6zgdZZKn=*aK&=^*PDp&hg!vH_p0!;3_f4Tj$AOQThGWPc zbv|$ve;gIQaB<#>x`Ioa&uZVV5HtG-$zgMB025Z2yQ(~8)smN_Hp9PBLvs?_EVy93 zD%`8ro@)(s6XQ-WxXrl}*~x&J8GG^T@>^B{H8uf=%nZ@@PdHc0=dIR1Kc*q}eyP_Veg~U{vVAtu zzws;A@_T=k*IKUE)R0c7g`=ozoegU9&PH4h_|I&>=V;&W>YCmM`?zuii|blF*&{a{ zwH#-xO+DX~;D28{3J6wo9#)o1%<(nh3`u!WQhf#7yRc7eeR7u-kJNe;+`vIXG{c2b^~aT>ca_c1^pK7uc2w767cRQ+Wy(9@#OQ z%PidgGaTjdisV)TaME!6%Ff$g1sw1!_`|Y!RpCw5$FELFrpfmWfc_q>aOEyXTF=w= zAfaaWk{1FXAw)(cpMB}PZMjsm8u0d5&K=XVHuo=Di*%pc;7VL{kE?A)RQGFujVDGS z0YhZ33J-hV+iZh#*~!4e9!{+HLSw`iY^vI%sN(mhQ|pH0@L1(Km~+>J6?!Y|Vw&v& zx2OTpc|C2}Qusq)##OM7ZMBMI4E<5If5uDyxj3azU#_k@ul?4(LDfVB^Giy zJH$@DiP#B4JA4!!WT=fTuk&4RR!DQ;1sjFMW5T`wbfX-`#SkaCDs0E|4?RFMEJ`?z z4v`b95`9DQMRHl-H< zr3v{ZPYOiNQnOdN0xU;q!xa74Dqf0hOy9@Fs;LN7qt8$QFX!(M>x!afjeeJB^4bMD zrlKxh-)7S&SdKX|kN(Df-c8fZjvp%=eB3a7)W0fsu@eTIsrHC+cfK@j z?0@r(jV=Pwu$z@HSXs|GuA>WSt_Oba>}4W;xTtrt+rn{v5kX`OaCEa~%y}0g=VU`P zA3y!csZrsuN7!fA(Cnzuy?^aw>`7q)xi~G7v)^%L-llKpv+aJ(>QDb_?Q#BUYR}&> zw&s|ctNU8FZi(zw1nkvX1z4yKPrq|iOiEeO##j$WdqHgXOBd!HAB7EERnWX^41tMN zE_F#`qm+_;U6YEm^{>2r%&S9Iu}5f&s-c;Dc%1SrO$|`?wuU*;9UCJZ zLGw{Z{|(P=cwDl&Ok}P8_oP6gP<;#p*BE)Yw~X8C&z94572qDZdHiKj2u4_;$WHk3CmDXZd_cPg2s5e@AKNYobxj!26>IzK?3uoCFa^fa{E5$0Y*4zyr80m*d=72fm7Gn_tQ?e8+uF=-I?(gF>!_ zO|e3PJ^-tj>yz|Z}+q=5I ze-Ay11O4Zlbmdo3PNEu4t#9yipM5;&hh)4aqb_3uS<}sUd%c-vqiJbieKWg?-r9G1|5aLgX|()yUHNR3G7Hov{!#xmn)MzgkGEh( zk)wIuK4Ku^BmK&m#C|^c&0FN(O_r1f57_I-qqsV3bl_(`F+S{l=Vw9s$p8K7`2YY* zA8I579xj|U>s=i{JqbN5BfBH#HUot}rdagf0C=CjypX(n=*P?;&wiHGZ>~Ra@siVz zskyIk!&~@2R#&||A&L?ydzMg*R`FEs3m<=xd{&=W)zY|-^FKyrYsQ@5Q5Ky`!KFn- z&Y&cjZ%a3GuLCu|rIo}!aS~D5W&lS`@m!a5^z(+Il%d%d_7a2qkM^ECkBIgyeG_5S zJydT+`T)@jM`^R@i<}7WPj^BK2V_q30}h)j3wOQ+F#~O_uhgiq&$8#o<;3PQ`BzCp z`bY1%0UuCLupPzaO$G)Gx3Ip>2#q+tVa(NQ(i_1%m){*$L+`68CQu6xBizoOJ+e`q z-tTt;y@ba3d`V9{<4_zI^U!{NML5G#kk=)uw%cscm|3CRa7d^Fd+x)(`Ovqx zB(E0-Lbd`Odh)*K_pe`Ce^v_+HPr>igZ}7KYX8id7xfamlr{PTAMP3~Qam!<5UGj5 zHP2pcA328MA{Cwr3dY>G7RzdyrM5c$^r4mk^82(~A+c`WAGogE%GDb_SKFO=%YisU zd&5}-k?5drdeTCT^%~ItxM*y$GVpm=A^(UoxHCQ~sJ0IUJZMWZ(r8?O6h^iDTYJto z10!$Z31}qS!(RCby{fab*Vvhbzs>cAyEiUXh4{Db`D->3mg)Re_hH_Re>oNcHO^0H z_xUyvY6Ki^i?;Jqq_7nkQRIwi#CjM-_2l(THobe@`RLE*uGPBQL}f>g1rB|=h->}Q z8I~Vp8%^>O)#&04;(2jC-y}OEF;bb+b0k&gamh)Z@gWBOM-Yyujz3fWbw3L6VV2n( zb&#J@rOiEmzTH7cjY&xl+|9PrQUX66IIG;rj#I0?((_T&;8UFrl4E~R24k&U`*=G$ zsz`1BxY>ZC?Fnv27(wHIToG0w^5skG=FZ;#z0)LVZ-A3{b@`ds`1Ak(6@eTUOUGOA z1Qu!29K^vR`#B2n_s>_@v4|9I`V_!f+J*LI=y}{P)h6f>`rJJ>S>;(mwQ7Rdy%>#t zmn@}ydNazTxDilRp|2}E*t3vPh z2Ytc&sZ$x>Wf%D@*CC{QlxDS_%Kc2CDG~D`K6+k37Uk(`X~g85&x}9C_79C|vN>w- z<$eJ0msxuk>>fn_)k>7e!x+J+><_iaiac6L@ybkump1*+H-t*hj^Bp2T~{=Xd8KV* zt0e{#WL{PErA1VxU7nzoV7~)bxd$KTeUNMNo`p?By`uh{Jj&Jw;XZB@`N7uck-7(2 zkg=+mw36WX%kNY(@S*i=`PFWp`qe>J^j*pv{+{@hw(I8oUu}qI2KoH6VuBb9UmPWeoL(iDK{l=UII5iqmch*Pm?V zF5M!O>2fyTHaQ*u6-v0zmIFZWZhAkSLiCI#9UCr%9$?2x$51G~+iIwZG zTHdKFpt~{A#U&`C%XZT-gEabw5OQCC-bM!P0T=dh?Yf z$UnhOngVsHs7-I4nE&!{eK6z*UBJ45dgh?5Z|8Db7lXYrK!^}FBq@wyOj{;u1rgyE&C1v)c^$-nJXPg!| z#63F8lA2Rc9{sCN&ueL?U5bA>*2n(R+mYFl)Ran>ihPeIBZ`I!q}&-i(O< zi5~Kj90)%-D4l*%U?t0VIwmgWv^C~hlJH7ls#@&-zq9OWO8P+n(TUH{grM`MdIqiS z`?BjKExwr1>W$X<{G{)XKjQ`jA7?p~usH=lPKjCo6;gXZ5_7UFI!aQ{#P1NtDyB{8 zY;TVt?!~hE9qO{#=ov-#jb-@@-$G_21J|1w#l_x7#2yf0FPt9p7B(hvnBRfl06$dA zEI=p6qGl#;f_tjZ)>mouUIRR|mfr4#`KAECUgdJNSJ@V3svo3kzK#9rFL1a@(t`OLw#&sY)K#PJi>e`ok#olxm zfvx3p+^Qdkg=oV=U0S;gcpvRON}wAbz1%VYD*Z4CrTx3u$Om_?hTY>S5I^X9@A5IC zjo!0{k5~cS(zKdo$(<5-m>JG=Qs7mSf;TcWSRja9y>Wd1`s?pc)Z(S|VPnwaN)ll$ zk+{R@+Xw&yc)Fi>(3mTQz{aKK^@eS>X7LK84-lmhQg1bi7)1P$ocIc$Wa^ z9vHs4F6jrvM9yZ_tlv(AU1KUnr;AhQs6+Da4q-j!^(i;Eh4FLYzJ0XY28JQlybYSj z!PKA$jfbV0ibi{~@bjb}wG1Ga_Jdj$7W49ln<#7nblPRPR;{rvBS#Wi{_VJrt1GJlf zv`&=WGl~T!FwD6m&oe3SNhQ;NeScnXRL7S%`tF1~~Aj<)gMF(4um8ExnVtU;1EkLs0D)JL^ie-zXUyNYzR@ z4!j(#4H0aYf$J$r8~I#u???J86Hf&h9jy5Dxcv$@Z;B; z9Z>fcY}I!ransVw^zAZaMq^1vF7Rb4>bRr$1zour;ly}AFx(7%i>EP&`Tul+g0H@_ z9_tohway=h%7VJQS_>i~e-O?x^@|gK>k}V_FScAjH4XTbw&}pl$VaDB+S4$AvQDExYi+YI!be$AY9pB=?UP}^$|eU*gn*~^FYZc z*HDj@`sF~p?COFr%H2XHUof5Roo#1tmcPbCruT=%U3hPjMnYbh41OhQW;D1a^}{o! z0g%S|k5Fje&v=WV_lWp}8R%O})vjOF(2LgDoCohYAupi%Mk#lTmsTGPRyD*nf3-5S zldhhhMBwB2_it6TY{ie!sz#m)+l3lP}TSDh~1Vrm7X9fb`+L zzB7B&{EtxNxJLU}9W`^bTPkca0&o%$yyvKMHw&Uy`$q0~xZtF?*%bU%#5CNidj#)5 zTRb|tdz%WRDLOe-!QOQ=E08wmB7z^j_)OX<_y@&czEyP&2 zG@uUe@t+R9r5RyHv1CMIml&>LMCLC-$2cjMe)s(T;P4-tC{@b?O63@i(d8Kj)8{`^ z91F!7tJalYt-MMR>Xm{?KJJtHK5%D&o6Q>C@aU7( zie>)3R*J@Il1Yg1)2XLZQ0>JbyGJ7d)om-uBxJI9LJ7DR=R&u zwLVj+pC#}@78w6;Rqfryv|V@r6p$(KZj?^IA6s?p{*VEah|FPH|dI=XE~l$31$>W9^L5<^lsU)o=mCX;RrUp;*N)t)dY1Tf z$(^q@`}RD?a#{K5>m0gcItq`)?D^&m%6D!q-2EvEgQK_G?E!o5B<+Rt_UCrK>_`b*X4%Z&lF*#JF{K$N- zvqz5Js-;YVLGzik8dqoMA8DrueE=m z%dzk{5G-FBShIz{Dss6Eulee|^zo8TPj!gY6@)|Z_!dU}uhE2$_{CWG)w{VqiXZHj1m}Y&XdklIm z8gK|b`u*1k94w;I${HeXT&v32Jhh89&cD8Dxi$?c<54hzUYB_aJuV};sIX;7>3yIF z^OHfwcQ$%3?6f1Y+n&kI{V4AC51gds&zavKUNv`$*s)({nD6gWhJ-&jR4>BTXgI9% zo}E~RU*jVJ`g{MbboW2ak8^ba9r((e+wiI;1t8zpiE#{Y>~Z7=j9ai$47f*l7H@&~ zeG>9C!}G^D(&t}9H!e-^UQaS|dd-=F;pNO}hm`lJ@1R1JHY<>Km~YY{SF-^F9nI-+ zBSPHE65?8?wT6q^k++T?A@N<2=2jmP8e-S~Kla`;tf_5#7qW1p*3!O7A5QL|P~kdPxWfKi3Wnj)G52Ay34@IuW(ALJ><>#-O-s@@Fo47&en>&EU|p0*<| zFnX4ej_(v&WMdHwRsNdx1{c`a;N59^?Xlz93RCsCN65=!VJwr(gxXoD3vq6Cxd6;tCmk=6c{iJgzR^ z2YHlxTm|7g*LyB_nN+tO6zPV| zV8nmzwq$;zo>PWL2`4DFl1B<3CdnG0s&ImVO~)lA6_)@5iD<~Sp@b^G9O@FH=6ZM{O-eQ=LyebHUdftzbyZG5>G|bW@e{3vCUV}8C13PhETbJ zudrgb1lLiU&UnM^rqP#0Cz&0Q=FQnPMdN9P==W4J%q^rrmo63*) zXBOW$pA+{|=h%6ra2c502bXaguH#D-m3*N@B%`sD-z))e^Iv11e( z=0UH2%bEU-W`4cp37`SJMf*BlZ?XZ9a4Xd1y)_{hQOBXBHu}q_KfRh9s?@7QIcE8O zh2gN*M~-rPCBGO;_${w$Kwy5v8VToL~Q2I0a`(eaoQ_2UL4jowMTO;g-J4wa-bCp94xl*XMme^Aa#&aROgyOU)=c8_5Zs0H@$8n=INhyL&DICG2C=T5 zpHPmv`G1_q1`%#>K4Ly#buFiJHazU!UdMw0|JvRbv)B2qeQDFXtee|sL&qDy7(m$y z6UGm&X|0`gt7ce)+CBUAQgL`QPmlElu~Ivi+g24c)bVQ6zC(MY)Qrz_IDYxrxS#8^ zQ_{xHZ&8jP$mx~5_I|Dmz0E=MTbT?u*LIK40#o>pi`!h2N3No|^aaiz1y;c9{w=3p zkAC4g9uEE7-Cc(2Evb(>gjHQ(SAAP6xlnbIUG#15&-4{QZHWYekFOnpJ{A*4ESIim zG}3QvuV-?Cckcg(CZfLz((m418@4;2JC

    5r1{J~8Fsw$&3s?$tlkiT=M=mvyELdMt$rl#yar0PQ%c`j|`dBgFFe2ArwvvH-ev z4viqlMa8NUJ}+2dwgC0G-xuLt`n{F_?oIh|h=J%*tWHmQ9`aiyMu@na+8&3b+QAwG z=FO2N{Qj>F+u4@NsOJUK_$Z&0s!9P)-WQlfy6N}*-`^b>xI_Z_hl`Ugjo1t@a?PeJ zqy)BB1H!b=HV=6zCG)!;C!>@BeB8L!;jAw zmiK7=u)s;@E(eBh-?y|Kq5zj%=otWd`fYDv=G@%~RZ~6I#+S8=>1w0gq8i}t#5Oc4 zHhBq<-(e7*omL~Af;a^ZLC_y81Fj!>fO|V+pL9U>!4c0aC76>=Pk_=9Ea=~M2nZ>< zxrDrh7D9KYCDNKJzdrxQp5(A5y+{Egp3h%-?{B-JIh`Jx((#9_2)7&e`v)<8Mzg9$ zm(K!%Y07n(yJY$1;(y#1b&ug^Nz)zo-`e$}cVv|A{xp-3vxT+uUH|IyQ_U`fK0^Y` z5Ygzo)F*+45TU*xO67=Y;s-+p-LC)A_j&%%9RrzH0@b)Xd2iPm@c~5vnq;7wTZw z26U&6|B7t6HjDDoFCp78q(a_UYe{*^Zt9<$c%asSBmzxeyCnVgu>V;l8Kdv& z?$FwhD`%odBrlDL6R-R%MIAHKS##%+CcKP#bjpzhE4tAuvTygd^uqgY zgESvHyGNUv?v@RUYB)83a|wDwzI&~%RHpPw8KkLR(&t**d}8dA?3@Fe{bPWQWK`PD zxTwjEeo_Udl+zG<0;UqK$=&4DT{c$ReZu1*y)BC-IAdz>fS4LyP6E;6jwCt*_SijB zq|3uu?hM4~k9WXoV1^bRCj0AKd%7gLT#LtQM5JMcoQo26*1zrNzgZb!x)6JloC|l} zd2Md`#2Fp383yYXUd{l&Ho(tLeo&NLj@4#~6lnevyuRgkX?pn9$KS=dr?n}QAInP{ z|Cl=MA$7iY)pktI_G_AsUqHCQZ)5o{SW;kh<$L7I;~yewjB+>Nvw#b&#`N}BlKCZu z@BE51TTAa!!W1M{V3x~FWbVG6)7E|=sLZX}eny6+)1G?mPsU2O8a`e60>#7VvkWpZ z8*p}oV)(H&R0(}8K6D&c^Gj)atmHOLtSP2@;84eGvop$eU9-w|{P{#Ey~|#Gs%ypE zN5%6V_-K+{Lcla^bbW*(XLxau$k_$jzXicnZ91-K;B=8wqkgia+mBkTIivp9L~7Z9 z7y%|R+lAJU>3?7LJ!wUa5}nD{DY$-R&p9r^7_sH#Ir1#xQuLkZDw+ad~Bbf$V1wFv3;6d-c`vdlieJp{>R2>nXe1_ z9#=4eLB`Gp?q$A~moMY}g2&s5oJm#o6wY&Apii*s1YF6b`ES6jfX#KUng56)D0!3~ zXc!8C0e+`qq;!H}EnbaKP=7mH$quxEy(wI;-Knb>#V;bHxvB}biItoy0J3q+J zh*H~1-}bKJHr`$FVhCp*wSIpj;BI(`ROZ?}bz^)atb-5s$Yg3YloiJPXxatTMwLBX z;wgVlGB$rw_PQi-Xm0%m-+SIbQ+u=?pfj1a5-S0_H+^;dPG%X8ic*U%C|cz`bTp+F zUw^OV(#~1*lem=83#QN&!SzRYj~YAe3$pkh$XU5!@7vyu_+6zcWA(hdZw*EpnBBD= zJx~^OG|XHa&`4o_s{G_#d}8_pF`Iw7clj(Wo9*5oHNlD^jzy0U$b;|!slt%DGFIVej9$`k$=DhNLfA6v1|Ug{{CFe*r7l%+BuKPcCPZ$~hvFY! z;%q%7X^0-!FD>G;v@XG#hZO2<@Y2kkBJhdNaGhA&9Z=~UBFI6zb1;7S+nq%NqO)~m z5XE(5KyVb<%Wr)jlDTod1imKBaMoK~rs|}2OXa$Hh{&I?*RSJeinKg#u@Re&-3XAl zix3Me0?U4;DX`nvt<&kZ9SquRrm{Ai?0WIInXxPQq&@Sk^=iRX9gHByDIy^FGEwEt z=;fn`x1aSbd{lk(RO z;+@_uSK-*d9`xhe7J=xi`H{Q9<4joDeiaSB3{4+nbXxKd#HdRB08>!y(xFA&*{Kg6 z9VhPqxM)}jNh=04-aKV)ZZ;y$_24Yru!D9!X6j3JD%U&qOcflB@bLO$nD6f}6FVEW z-<&s?*VGhU;Ln%*GR@FlQ&}v4xAwPCoLkTNx!7rG17Uryy>5N?dEN9Bl@$BW=a8Tu zmNiXZ98~|_FyYtwtLj^MP5ILbhlred|oeNJR!`TaW8%2XO@b(q=xF>~GPxm%yIft~dd z$ZF$*7%d^yOQ)SDOO)1Ca;S{JV`_na;4kw-v99tHY}JeliKD;St6=p}Y#+*l@BNiO zbo**)!9|K4AfHMqGmc?GQ6>U8!fFjTO z@y8p$rtBKZ(~67nE!wZhwQpZMh~A349+?=aJZfvAF^%(f!D5|`v1BL;79Z_9KGoAJ zL;`V23PvA}OzASRaDD+hXG7#4$-;72?D*p@);OOnRoVBtU2j@aTFz`#B3_3bKdyc1 z1$)dTBlt@0Sk{r1)Es?FgM>!zP}s0MUP0=mVxL}`->EQKOOC)-7Is)~qrhIU+@q)F zcrh?jA<*{a@$39J7ay5Y9gYT87PkSv35>O&IPKK2jWcp>6F-g^Og%_9l@70~^nCXH zxeA19Ic1_Eo9%z>Vp@?dS9CSPB-G`ZlF=n1!tapb zug0^@20bG%&vF57d4|WgH*K+?s&l`Z4`qWZGTANnkWg{g%xT;@;S;{+#|W!R6yFv{ zjmaPsDy!$)%q&nt`cdI0wz9oj+DzalpgE!H_qD#9ET+w0wK7$@5Mg}5C-a%4b(Y!; z9xBd_)%|D1KMLFLq2KQFE8Umu*5u~c9%zL2{k-tnz;&E-gz<bBRK^AZ8d1kO^`}-J=2-hbtnodp_(&7UD!38%>Yyy*D_*xb>gW7JZlSZbusLm- zb(S06pIrM=Ro?fcEI>HlHALr!shL1eJpQc3>Jj`|b0)Lrvgkfhe`g>2cXKP2#m=?o z)0s1aeGjmH{8N~HdvSi06kD6PV8tZp3so^n#A{{Rcuk)dW~#$;%|>+xt)}$s=s!En zKQV?NCTi!GsI+~8u3F+;Fob1RHh?2Rm=Cou*R* z4y#D`af4Ov^4@7L7GJqS3A0x?lojlvx9=;?$mOV2;-}5%3YryVqhkH&!b7kcQu2c= zZNz6nNj9g3Yf{WHYV>X~Xp`IZ`ueVOPBCYlPD1rkC8YEudTT?DqO$a&HeC#Y`0Nq# z6Z%323PGO;@^>7>0(%;Ti+z-lQZfsX-629>Cd|s@QMQY_2p_@ltRsh~PBE@y=6t`} zN|5!Bqd=E6OOj5`*P5Y)H}nM~VFyvjGY0jMlqXQm8ShnX>-Crxu5bxvwy>yj%fa@& zdyV;>&KTpy{!G6Y;#qKVtNN|5wM8B_2QQ6pz4yuo6m+p?BXNKoEwC+CHh$>1c1dEh zf38GZ;+&Jkl-u${%{zE_gR%2JS`YA()d?Bd2jnT0Gulb2_@pusWx zDfELZ#o5z53L7ysr&DX*U4E;IjH`GQ@9%=W;xWQE@AH;F&`$KpjYq2Q@GV+M<*4W5 z{^ou$r30ZEA0LS5M$R0Gf5)|^p^f*{I{P_)%EKeelmakB;d-_r@-Dd&>$zPG;N)-L z)jPLh`18zq-_^y+GaM9*pKdv!6{IKpW=~=m3YvNT(os!tZhxOqt%G5@C5UwiW-ga7 z>H3Ds){9HL|Blml6ZrzMvo8!BO1<c_^qt}iCf4UjAohOEeq2VF8fxX2n2py!1X_w~^Oh7FBh_8nKNjPjde zOXIuLJ6zU&tV#2ZF@;G(O0XWsOr7eIPtzZ!RgVFKDyCTX(F3ZL@Q2 zhf?nI1CZsYfNwzgW-ZLfB=`zJx_-CkkXS|$k}&0WARzOKCI4M3j1J1huk>L13;szi zJ<%3zQ*a>d(!MFKA#WgpayoB_Mwz_!AQ*TZ_&9o}RSWkWR@v2KUwYc-N>vd*3MbB0 z^$jMQ);$yH%gVn={&@NJrl81Zm{$3nto<(D4f6Go zc%aZFXskhNna={!7&>pXY`k1_)Fd~II^N#4(eQvDjio9T?XC2 z7tJSY7WpNLt4wiwaJNOh;)}AZsr_=EZfg2qIh(_P5Io?gH-B^2!WOr z{v;ayDP#GeIZ6JJ#Vib@E8Zx|GGiK)!SxYTWc{AUwU`Ruk7u`8Jky3xc+; zunzWncu>tHcw4z$f#=2!+K(iU@3^rNqAP@baY6{k%0J2ltk-l2-zQx)%f3yC_3MA9 z7k^^HK*UWqWo=3NmkYwQbY*QX$X5hCDA;+D_)eiTM2 zB9kZIg%4i+7I{#Tx1FK`IegYqRKOf3iG9+#PvGwEoz%KV(Son>1$K2bgaqMs@m0TW z)y#DJnBY&AA@IcEES+{wjBww5o@Mlh>DJ*-u+|Y%E7=tA zQu0e+v7t{yl$ci$?qrqUaO=(cy?Me{SB0vy13{D^V*{E;hvrPj41~UB|sZ+@DstsldYtL@kIwJ<6td-86AOIEPMYa zVo|lRDB{M4*hP`5(zfFbXWJi8;^p-=EAgA6X@1n;j`jEM3A>f+M7}35$+JfG{Rcs1 ztqa^6z_LUD8;=@Ecjn)D=C?DyR|0I_annFgBPUUJhGqBDh5k?G&XzP@(LJtXF&`x| z2)7{v_-p8LAo;*$ZFD2Zpq36HHUwZ z8k9m8-Sp$ZED#M^Z^Y{Q9RMA4KlFDBz`op33P(@S_4;*Lv{Mt_c<}W>F-V8>n3)JL zI${acVgkI{rjfQiWqOJ?1((uaYQ=eC$+ds(fkaM26L&ll=Thh$wV(7*<}(k+d5=C7 zKW84KMN6kh<6YnPclis?jZM{!z(U_RtC37;}av0lckcT&5#*CT0>$r&Nye* zzr9K(iJlw>D%w^+2}<6k&&+5`7wojf9f4iE#F;E2vpb%kXB13MS>~EpFW&cfNw^** zdzAU^tkG?-n$@75&R61gB=SJY+e3G~szwDTC*wkc^xgVZ^sLUi%p-(9Xn_RMaOHr2 z`o#>%n#y=3Iwgrf=`B9Sv|*Tj2WKoXwaO6ctDMR^SFa|r2?|0}i^)};GW(s(Ia1_~ z=$nQ&w}_wP7Yx!PJP6Y0@2^JPI$}Mt)i7gYXB$34wSW<>Ou8~}5&SZ4Fi$Tr#!1Ra zZSiiSO4INnxmhaVZRbbrvNA3jAM=RM=LWL5l2d=p9X+I^R)=J6l#bz>CCD~Vyn_D4 z{H9=KV8yUF?Do4tw-%hgot4^?(Bb4}3YO%`pm(T`HfE+=Rs*%Px&aMWW4Q@qSz!D? zDeT4#hyBZTqKVAr0sf93O0gO$fZUi2QQeC&tnsMx(U87;q(j}OWS=b^37Sp1d%6vF z1a5cM!89#Ohf95`g2y>`R%{JSuBgefU3c;!pRnGj4JjUx`8;42a1z0=3QeV?Isf?O z@#e=S<&&Lu$3(}U{!r1*BF$8^0_+bDgULw9dv~B(uK+8NH$uN+OWx49YCU^YjqmTQ zz-qe|nf{&^6@YT$1@qQQZPqP!H;e#^VOeU0ciPihXYN4 z-?d=eUu4UFB|oW$oIt^k!}+ETHpM)JNvS8C!j}7;Y&rveacv=aj=S(fam>%Dc!W?# zq-?TSoTM+OSJ+aHcKBUR50(>oVTzL!bvWe48AvYy*nrJNs3qN*NJR`h0$aUvKv$vM zxKAor4kjfh6(WuP(?dp?bfc}99gq=B>rjMAdu5xd?33@&*Be#|XP^6Mnhs0 zG@C&!Y-QMODtNe231i3fyjHTLpepItswXQ`(uy7T|D@zn*}~%^3?wUYeh&L1pUjl< zs~p#MTChzwUVeq{n2mfwk(m}C$QVSQ4#dV%;HG(7E5ZgvIl zqOmxTNw_Arc78^rnGA}%6C_0JH;NK+LbPF3Tk6_kt?-eVnjtpzo|iS6f^mLphFcR6 zTHTg7Kc5Sf_pv&~HSbk&C&!}e+_xJ?4nHhVyQo3+wy`+P z+lUsO_Ke*pSXKCfJM4b(wcbNBhv6g1mSaxoZSSy9wdJm!b;qmvx@yauUmV`qb1HBK zZM${ywZo4q|8ezy{&w|0zdb#r*mHjH6|WrX65p$XWiN#x9on$V+RZW}mx;zZK)Hj| zL#wreoNCE{`0jgnNgu`E-rjnTU-)?;#ZwU|FV&>R3Ga)bKs(3>{>fyrCQ#<2OiRDn z>6;Qvn49R7PMzuCxy|_v!0Nt*I~X=Fzs7&fBU}U|214eZ^*eL=+kE!{aC@zpXX@!z z{EouYqYHoWQ2v$FXsYH9sgh|;gzZ+b&$=3)(7vB|oBJ48{-bSh4iK=aOXr&L?Xy2c zH!prVEkQWFXj^BT> zohKv`TH*&!&8^GCzP19FYbCw|3b7<}KPXMEe<3Z2S2a-)qpa7 zrJ1kZXVRz@R_TUQuTGj!23^--Zv4&=+=$kTjPwl^+wL3>xSYthM@QEL<@dfWJRaBs zeg*le01Ho;G6k1`R$It_95pUQbPg{KQ-k82jJTJPI zfnbUnFdWjHOczsS_3=kVV(#ZWJ?(Y;b;LZhYnET7`qVZd+*pFEz1igWs$Y81oBQ~+O#+@i)iXwW?D0-tNr>=V2o=qke{Z_kZ~*PIZk{3Z1@9609-vtIe#wjTReqdJ=2gGe$S%IQxx*L>1b{_q;dhl0@SrBq%(n8|4B7WP4)3$5 zEJwoEmY$gvtAiV(F_8X03m)*!WBsg673aFAs&RfZ51wM+&s^R>=eze!0EG1loFDJQ zcg6R?^3Bq3)k=W8cJWvNOu<@LsMl~97#931zzXM~*_u4jq^8rO(2;X|YT5kIwXWncl~0Nx;SE#0dC)8S6wiGKu6D2@N-= zQojQQ4W1wb_GOQq`a)Zbiz%l8+cpQeFDhl@%Ws z`MVPOlp18(p21(`j-0xLN3a*`(;a8A(mfwDjGd{9BO0t{`x>#ZYMD!$vDmK}b519k zU0xDynJ8Uj*MNGEuWk8e3V9trD;2*vMioHx3IpNs)UB0uH%;vy%u=2z+1&5ytsg5= z>J4pVa+h=|w@bx}5BH9ArklhhoS%AgR6=L68cKVDQgP0{Hs;YG`P{W;Z}@oIp{w`L zAG-c!{6o>As%cBrsG6witA(zpJs#hpco>ytTc{6FQBuQ#Lu!@Db~yKy=e>8|lpk?w~#PM63+-*{}MZa$%u<~{H^wyDmS&G>F1~1z|P*yY`?xtS()j`p4U&}?&X40Jf>WpJMP9V&+Z8l z)*dvQbyYF<6o@YffAeDNGHKw!L)uV}Ur-;o4kV}lZs3Q!zjr~oq92%eQ5onnww-3l zI#QbGX50p}GyZqTyVER63;uCX*XP^$%ZX*_=XfuMdxIKUG!uT{G^gHLh5PhwerZ)F zS$E6$gVY{5zAUH`SWh<;gA7h3?0=;HNbpe~{vG+iV2>YJ9YXp(c-3rd(G?=LR~FN1 zr%5BvIarPR$ti{p{X$qJE>=2h-%Od5$17U6j?3VNt_!EUT(BFAwpP`e3{d}TnbiuM+-|M@d-qANk)cgan*f0?0I8>^fUg=ZIJ4 zdVDK)3x7h1!FPADiaQ0*iv;H593kC(CJpjpfri_`PwUcoT&kL`gg!A@xD$NM{yGlw z^ou!2X>y1MNwCgI8;z>!{SNmShbV&6Sq4Euerqwl^P*=_y{`L>`@v@~gDR}XA@(wMr!CLK>z<20 z>?GS+Oap|LdKRBOtIEw_n6R8&j5o&oqrVM8ovyIRCp=-@wb)1b0pU8t#G60;0+Qjm z4<5(rVQ>rflV=P>7p!!CjJilLUeu%eO2usNrO&zf+vb8>lE}QNZ#vruMG=iF056>~8buFtHEGvz`Q=%BZL}HCJG&@48Hi zqQ68--PrdP`}*~j)HZQM1+&p#neVKevP$j*R9We46StL_<*cvPuoqHcp(TB(AKDzA z_7ek;gbm|>=BPQ=>@97|LzPqCLPcF954wvJBue{S*u%8G?}3Jl^KZ2QqBp%)Lug9q zrjVhSCa(?ev~;XuPxpSI=R$0u+VnkqFI(Rm=4KOZgi>&}WM$~DmSbifaoGnYk_ggb z^cC?;gxP(?hl;tHx(Y+g&G7kQS%yGb-53oQD z)#+yKdWLPQ20&5VGqx-TkN?%zYJ38FQBQ#WJ;Fcp*>mS_iDs3_gREdrb~Y&yJuM*v zh-Vp}%5iMqJtY!;Z_Y&f*UfjqZe-@_V)FA~XwC;F>T_yuPrTt{+St-4jAxe|AmKaa zWz}*jRNTkVRFziPftNQ*IBTBPaVunPPtXZYFW*|MPrF?F{-h3OS>gsce_KD9t8Y=Q zbU${-1Iz-{)1q|-+oyP(xELp+zEB8Lnm%DcuXc^K)qjb2KH`U!5F2i3_*;n{u{Z=k zoDO4&Dcz~dPUoDoLx7ls@O(VAOKs{o3o#m7bBM&W(Q$qD$CUqVapsne?K}s|D!0~7 zXKH`xC@>oXvrr?wUuxnQ3uAL9lHchiG1jG5>NeANGNBdmI_&t-LWXXheM@D9eddj+tW>RrtGY+c7ey$S zOx<@!yh0$KmUuIx&L-VDYLX%SL99;L`JO}r%2|Zp<f>(c4 zKsw$!EZNC6*T7}ZAJ5NyeTmD`*3Y1tiRFT0D1Nu$-_Jr}#r0Z#Z%Dyep#o6vdiwC^ z@e#u2EFhQ058nNJ%RzxC&)E)5zJ?=G>xm5vf+(r0xPCYUyEaQ(A#Gd4V3{7DgipFl z+leo%)g+rztB3*AmCw#*%jl5VT5mDTK7~AzgNif&dHIpD%z^Ji`#N59?u*Kf*{EFo zTAO@qK0_lw0y;DmJ>bm#T{(kt(f$xQ(knj8bcU=)Z0DR7cer@c|On<<8b^CK( z9RE~{UvP?EFg}T@na9#PZ=raOY1QOXdS<4EAvX9#1=xJ-qgtVSQ6^SsC0?Yb1M~{N zP=nC8P&WFGcO~NOTE0?GETF;l{5+SbMM2$CgGZ+nCTE|ieur02idCoJDyZSdocH^n z^GS-~ni(mVzOux?b@!`L$)Bl1d9e&faKJwgT)r^3Ec={&vfj9!o9N!?di}CdP}laX z-x~@1oKZKQv;3|>UQ3FZ9nW&7AVOpUZ#Kh`ZWAc!oZFTrwEW3v`7}x4o^h5=cvP2g ze~RaX^NHfxCo{Hg#X&8NxetuxVnfT`6BG^Yo!J zd9Z+<6NFBx(!L^NeJprvK16BFnQU=-iGTX=)Bw%~Q@rqT>L|fI>^Z2pSu6V6v39|< z&(M}VI?mbOSKqz7k`%rZlM}{X+S|(2YS9ulpBH))JDHf~By1f?zv6&$w0i=4&OOs&-JOg6G(`c6NzGsj0>Fd|@@9iB=HE{Y~ zU~1!RX*=FhB;#=O=+X^ANkLz=ZR@8$g6c92zVPNFnsj=d>Oxw`Fi|YyM?|PxVlFi z#V?+$)+l1e?Nc=*Q9irrGo21bLX|#iRhTe?Eh%!vw~X3y9z5AJ|M82BJ?yEnvr^iY zFn4&Ku;jgBszq5|X)*6vHr1RSU-<>e+-bWu78$W^;XgupYJ#B__$e=;z+r~<#9?WRi7j>XVa@o8$CApEVDiMCw~*~G35YD8zd6G`U?WLrqm#+ z``%~+A}Q7G)kH8C93lbtg|lu|!+tl3xUtQ8{|cpqDYN!YF0{@pEMFTYX$sr*A2s}= zttBFkA+p&NA#qC_B6i@L5a1L+y_P~*LU19oO0__-VCaW0_7T@z{qa(w?81uCD(2YK zeb9hrCRlA!dLwaaeZizG!4`zh;6JavG#?<#ECCv}$CAkoUsDVDg$obP43$2eUg?|d zM^ZL8>K_yt=1gXf=ilV&5}3297ejx2n*OPro~AggTZd;IErntxx1EZDAFjQ-U;Cwi zf96K+Tp0dzoA$@ZG!zFv)g>or6 z-S#SC-VVXwky3k(9;@Ekj^DLEvw^8wdaz#G_G$F?n5d@3aLB9M;K0$w9YSYC`Ess@ zL+(k34~$oK-MoHJBoG&K2g|km@6K?!a?NZRn0N&!qihY=Cqe0wq2rcc1 z=#zP^`I2mmrF~j(wTZs1GIrwz{3kWuZF|?N+!K@Lh4uU-ccBDD2iln}^kC-8R+74k zSJ0Wyo>bntFezuj6<9Rs{QzKNH+v!AdZ5jb;^D~x;!_j_b3}czh&eOA{ae_>Yovj1 zc?G-8ek8HpI+w|*#BiD*bx3~jM<%^aRAVfZK|DgF>e<7S`>3J;dhE?Zu_cXDUzaSp zoaQktDI?>(o>-ESjI9^Jvu?A8)S7BPgzS~c1xhGlS7`ag)SlYa{*_L5?|E=q^8xW- zJ%^&h|7>^vGrt#he|~|Ymy~MA(JSM;${E`;wbovZ`3+_WSCu#&&Jb0dT2#ZRD^3i{ zN9lx8HZ|&4kWCp^3Zubs!IM`yWaMH2ACNoX?C011%A{+F!W$jf_;-2tu3EKr3 zE~QR-)wm8!ugPajG|-2w2Obe(!Vvbs^r4lk=6EkzIw#J>9y+Z^T!C$saz2F4Z;ewE zU^(<61gO^*(cEtcK{sA>Yg`K;tF*(R=}7(U<e*)7tX}uP*-GiZ4X2#t*x$@d_D?46y3^yvlVZcdsqTVE1Bj2erk8@F(|uE&kUwLss8AK^fuh?10MO%v`5<*+Us?I z1y_w2R0$k!oqU+a3V(-NuWNL%K>I?aF&%+aBtMKut%EBZi$YTO^9OJ=7&cbbBp-P7 zCQeQO!@%_|86fcof@)pwwbj{!tl6kd^@TXbdK5{wOMD(3G)rFr1kt=9-vNOpBPpJ*kOxd7;{{x88RJviTk3VcCnYG?8LOh9%`8gk3yyQpIc4c}BJA5)X55{+agFj589&}_xD5U~`h?s_2>FGu;>v3g&( zun0i*WA0^m1L6&M%1IlyV)YM9_s6{d~r z;_&k5pj^yKu613xa!rmlrC^=tWru<3+EdT%>?E7jMfGv;jXN}>YhOoH$e~)lUav4d zRadn93AKOwxP4G>W7U~d2p=B;W2ny~-gh4)=i!6l#0)qb59Yv=PHDPntYvJUcb}X@ zaKe7e%X&`UO0v!L$2YDzRc{Py6H3?eGLRwg;XvU4OMAHU_Pn!M1-zcW;|tJF^9yFW z2tmBFzIeCIy|{6#6B{!4NLEw#CIp>7!YrdUP@l^s!${kJ{H=A@m znYsBHTjPz0c|apu4F;L?*=~r{sb1fW4`rJW`}E8M4SE+oSpr&SoF$L?q0il}lafDC z{^CijknNSA92PK_m`ZvnixkUdF z8@M@Udn-}*^1-Xj#XuHgHQVgsRct)J(WbHSQ#=SxT8|}Bng~(W|G ztyne^mql}SY?P>&iKRa$;qkg_gC*mE*=A@NJBkj>10ln$K`cW2`~V>!%M<5d>(a8t zn6+6iPFVc1I_XiW#hdJw5u$&ZS3U?%jnib4R}$MZJ+xaBKS4kUr|Jd3I_OzLX)MB` zedvHE=0`QLlIX-F)vYVnex$7EvdIG&PxwSRvI?&s5l~#uAROmm?zW&vST4{6)<8V- z*`k$ao{Ho>kUO?{g;_=A*PVK6^fhRU#Siw#nRA4&YKguPB#5dP5isn95AS}Lv!)Iz zj&XWdk`#(o?2xR@V!?aD@tx%gnL6Y;N)O_<##15Du^LIZGa+mbyFbtYY@~zQHAumT zYLXHnc9B#_^od7e%zJd;gSo|x&LLuGWXMPD4gOHK11ohhR)PcGN}(78yEPR!m1Z^& zIC$2@R*iq4Ol*g2&>f7?oFJF|bGFZ(m>fP?9Tvo0hh3UrVr1gLSfX2m?nAvr1mm@s zPp@O>@=A@nmC5@vkBecM@UJ?mDp6nUWC$O<)2{x# z#(0gxo(Yrt2@L08lRYb$dP^%(0T8)o~>5KFC}!;+%6C%q?$M}h}S<#!uABRHkZ z`RsT?9|6Wu8l(0aU1P@88*cy;S(-=(8^qZ8FQkwFOs9Q+@+>r+IH6%W99uOSa&M9y zE30&Ov!cAmT?+(36AK$W2EAHpnM373Y^mN)U&l9wh_Nn9elEtcbKgX+CQo|wda5J| zuZ4843xzvO;Sa@E1luXJb6{-`^A2Fx9?tlVUH0H-YGTG z;rOy{VJ1Cl4BfM_laJ>A(hMtx>= zorS7jh2PHt%uNkmi#pX)w6obLeJ8YZqK$L{xRtm}bes<4wzLGuy-=)WnnymfPnR}- zjMQSHtnq!M$RS*Ms$XRh_9yXqY_Cz(xnpaKwyUs%#7#;h#Y~S`Pq{k@1b;+b+!o-= zYKE>y$Y+F@$~;1jaCdpapR~Kmq%03sO^?Ej2iBFa1-f&WSM)eD6-KY5RVYWn)8o7t z;!0h$U@ijmbpCU92NgES%<>vz@tZcdHvrn(LC}Vbe%Qx+uE1c*5shm22mX%rJak|Y zf-xpch}yeRunaiDhTr{!h}z|I`zV#ogwo}S>qRm41=UYX&L>p-d>f1;pIpsDBa0U3 zD+sE$YyhP#lkfhPW9aVsah_Y5dJ+qdS$FKIe!agakC_n?$okXf8$9+^&-xdj8X1X( z5)%Pwx}rWr*dd zyfg|>S7T$l$z(dYNYxFR)SOOLeompVN_`qsRH!RaZUnZ~_#((??Ouk^3mf!=6G-clfy!Zr>qR z792jXW)KgafIZWuF@MZ-5i#Am&-NIv~+j1fTO{QF{Av@B(APwcHpOqRD9xf`xEbjxWasX&01yjZCO1(Z5VxT(f8f0f4TBC=%2RJ&o__3`yWysU6A zc#(^0Bat*0aIwBjHt5;JHY_dm)<)CHmhMjEm%3#)c+fsgfu-dl58I^3>G`n#Y5t7R zGB(REz*F~=T z-&t$V6gNTtB?SjSrwg^_frEV7vC1_&(N$rit`TeO2)1LAEXg-w)J0(ZBVSEW~FF1OHEHwx6@#t zl-g)oUI)sFx}DO*F;rAUZiD4XrPO1dEFmj32f$Qt03?vc2$ZeZS|H)I*kGn$4^6|`X3*vRw4LiXqNuu75VGJ+oSDwrN@P%ABye$!gU*VeySPFPIfUTUp zp}(->SN@q@S!1+##~a{1Rh1>5oCQdbU6vQUaFVncq(}DiwKFRq=Ns99Q(59uZD>ec z4US?!kkVWFvifqmD4AOjouhoo*PxVzxECSL;S1bc7|@+!|ALA0qg8qqk8&maTau+t|#u z7+;=-s{sN}ys}5kSG}WskuEL~eV!GNGL`-Hrq+O)(MXd}L=m#vz4qE=281%C{LQdu z%$lX)-8wnLKiyc7;n)E=O_aFP+5q11Qvf*YWD>S!{HBMk(N?6}!r_WN!;VZKGgd%{ z%6h_+W?v~M8vtSiDs-f6u2=PGe9(|X?DizC-K_WD!4{xT@O5NG0UvW<4&OgfV{)0# zHebj*jO!mbX_nPsw-1~M;<5nD1NfXCA1Z~`0uTexOVNx2zsAGsM&tqu6}V3VKmk$R zL*gmK@Q-b|Fw9vM;3LlSpyjnw4$S%c(hRgi=+Cos#*gE|gh{eAqZ$$_ilPlk{|HG4 zz=>@$5r=BPs^9(Zl!D(N41S-xdV8o@v^87IsW4BB%u@jX9=w;6I7LuSWi|HS*)aUns#DgW zXru!p;Nj|HDLuYjw3KCNqMBjd;@~dVoM1CR(1~s@jj$KDH|#;sZ2Xrlg1ps@Xl5i7 zcxu>z_-ZK^??}cDzqVm$r@)u)ueLeTm+(>2!nsA;^55;q?pgDV$->F70y3l{bR1P1BzYuEI;r@!pBo3b z1`Cs^rpU1OTD|&+AdIV(krPAuE&$Udibe4koIzqhY|vh`PoifL?}K>*#Gklebl+(K zV1T~~uc~uSP@;H*AWSmsh9C>jnqBgK?xy(1Bcv! zoqS|MAeL`4=9EWDMfCE)AY?53HVWL zezvX!cChq>(sZC*4#ZK^mA$(G`p45+xLGCNhKq@$uHK>-=j>R8lhFHsi-g z1n|kt)C7i9(kb})iQgwr>=a*+#1Ulr$UbKH#D+JhCg)I*b){fe3Ypy#{qc_}KkUwQ z-V2#Pgjc^WAx(Cg@%Eh-Mv5ZpMnE2jQ$cI!LvQc^jP$r`aF9<4z|S)usq09l=CSn(uMP=KEq@ph8{v`K(sdF7@$ z)4&_YJD$+bxJ1Vu`r#~Ve-oqi98|Sjs;IG4-hya_P=*cfcHk|x?+PMSyy$ zoV9|a0|H(xhJ3|ZG_>t~Mn+9~BEYNqlIPwS&mF_c`X2pyUI`F3Qvo$kU|@PPs)3fM z(s{H)I3VRzr&LCgZi7%Zt87~+mt}51S}dC`HgzG|w_tx4CCrhfFTaQh7rKJoG{2M( zdss^4&UG;G)5Fu6T1+j{H}|?ZM~r`ZfLg};si-LROxo7XelN=Ni+9^wx@c>g=&VlM zIP$OUOApW3rp<1DsXJPCOIv5{QZq#^yTuYGz$)@tUvr_+bZU29RZR_-Rts+gqsv)u5hGCeRJdnix=Pu9@z*JVX`k#GoSkA5#a0y>=m*V-U*&B+b4IZw zqP4zxH%O*y&SyS2$5qEpo1LOz{9V!M_m;w%;HJO&Pu|<3S}1V$ugnZv!Sq)Br=-aWrvr`Wuct22!C2TG zpDjs0@*>e_bPwRArP#wvQz=M4KkTIfdl(}6WoLh>JjWtBj1)sdAhQtt5zGyTSXU+w z8uLFeUF?%4gP>TonTGf(h@~Zq=7!TQc74?5lKn%jn&euv-MvY5nJCLovJ5#4(xK(= z_j-eSHF|3P@u#Ar$nu)~rN>oWrVSAtP}v@#;*5$jk!;wD3=9|>plh8Yr*)OhSz22^ z-}lylOU(>aJagYO0AEbtxU6L+7)VUvS;RKy<8Dow{?BQ8Z+l&_g$Gu)kx-HzB0HuU zZb&okGB0aZb>Xh#XBBW|+X+r1{!(hJ^@^ZHXp-1y7hK-Q(+VbYNO(t)>!tSGcNGz9 z@&kIG$4~qJP2JuaEXr|S24ByudspJQej7>Z-4rwQ`d67RWyz<1`HMU|)Oo?lNtKK0 zRQ<*~@T`Hp&p)%Vyatv;Zy-n=qi|W4RA0fshvk!-i0%_^A<^x(LJ1@gF;(gf27POA z&Y`FznNOOk@sMrRDF}^_4hV#nSRCGb&bfJR@h6P8x3}?euRm{lhCXa1ruRD9Pf1|7 zPFME((vxO2*L|sSLquf>?{(N7BDm7-nOA~JIZC$hFEfs`3pa3$#)TyBknH&;S`@1+ z-rOx`^R5LHUGh^sKrO^Wr)4jyzh4{JVEVoU$VO>}Wg{<4Wq|bSUZuDcex{gMl!?u_AzRlA` z)9f?{NYrV6JDu)QBO6w3V<_lA)R*6GeO*62NCqgV?9_AP4jQ8TDnfB9y0zhm-mr=B zQpm*pXC{*R<~jMN{c5dLLWjHYME1j_v)}GFKA){*uX*mM@LfzFf_3o=_%g{cF#9eXMz;&$T?1nugHTb@~QdSx8hxPWc(% zzm`uT)vB9Njd}HFE)ReuM(U&K81hjP?PlR+WKsE}fufOIi=BwB0~fI3>xKhyXGeF{ zR3+?P^u&sHqe4A5@smhCc#9ZR!>>P>|LEiDn`9#gcg+1eO zf#*cNZ1F>Rkjzkeh^a?@+ck3;wOADD5DO2vHEBsfRY4kI{Sd>@lm|b7=MufTsrwcT z^)I5P^|ujV)=!^DaGGV)5 z-7!fwRgdc`=`9a4OAahD$@RNP?wqk<1=R=ZzJCgFx@Y;$rh-wzy**Oad!oDyHJ>X9 ztA#tO*?Bh0wY~6A+Csa!qNag)V9Vs^>9ksUm#me9-AnQxpiYjJ`}6RH3=X`bFVkVh ztV`wy^OTITHL`+xa%)S{Z_`)4eG0}hED`R@ls-uPSYlZat3k*1sOSQ-_3rJZk8u&O zp0(Ve4PwG`+xvgoUX29c^QV`+&To0nqxH4~H5K{o=C-vt8@pZW5U;;va>SW({*+_! zyE4DQ@tR((Z+X{pF!o?OgN!Y(#|oen;SVL7Jj-O!#|c T+Q+7;KTe-M?&(HxxpeE_0>H6p literal 0 HcmV?d00001 From c724a097bdbfdae8ccbd9f066896f56d89621d08 Mon Sep 17 00:00:00 2001 From: Jacob Valdemar Date: Tue, 21 Sep 2021 07:09:11 +0200 Subject: [PATCH 040/249] Add Line Break element after tab widget --- .../docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index ad3587ad41..4cc9d1445b 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -88,6 +88,7 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes {{% /tab %}} {{< /tabs >}} +
    - Verify that the download works and has the expected version: @@ -184,6 +185,7 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes {{% /tab %}} {{< /tabs >}} +
    - Restart the kubelet: @@ -264,6 +266,7 @@ without compromising the minimum required capacity for running your workloads. yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes {{% /tab %}} {{< /tabs >}} +
    - Restart the kubelet: From bc4b686a97c0e873ea8b169b6cbf7f6203d7e9b4 Mon Sep 17 00:00:00 2001 From: Mauren Berti Date: Tue, 21 Sep 2021 07:58:41 -0400 Subject: [PATCH 041/249] Changes from code review. --- content/pt-br/docs/concepts/configuration/configmap.md | 6 +++--- content/pt-br/docs/reference/glossary/image.md | 2 +- content/pt-br/docs/reference/glossary/selector.md | 4 ++-- content/pt-br/docs/reference/glossary/service.md | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/content/pt-br/docs/concepts/configuration/configmap.md b/content/pt-br/docs/concepts/configuration/configmap.md index f4038706ea..e8514ec15b 100644 --- a/content/pt-br/docs/concepts/configuration/configmap.md +++ b/content/pt-br/docs/concepts/configuration/configmap.md @@ -27,8 +27,8 @@ Você escreve código para ler a variável de ambiente chamada `DATABASE_HOST`. No seu ambiente local, você configura essa variável com o valor `localhost`. Na nuvem, você configura essa variável para referenciar um {{< glossary_tooltip text="serviço" term_id="service" >}} do Kubernetes que expõe o componente do banco de dados ao seu cluster. -Isto permite que você baixe uma imagem de contêiner que roda na nuvem e depure o exato -mesmo código localmente se necessário. +Isto permite que você baixe uma imagem de contêiner que roda na nuvem e depure exatamente +o mesmo código localmente se necessário. Um ConfigMap não foi planejado para conter grandes quantidades de dados. Os dados armazenados em um ConfigMap não podem exceder 1 MiB. Se você precisa armazenar configurações que são maiores @@ -149,7 +149,7 @@ spec: ``` ConfigMaps não diferenciam entre propriedades com valores simples ou valores complexos, -que ocupam várias linhas. O importante é a forma que Pods e outros objetos consumem tais valores. +que ocupam várias linhas. O importante é a forma que Pods e outros objetos consomem tais valores. Para este exemplo, definir um volume e montar ele dentro do container `demo` no caminho `/config` cria dois arquivos: `/config/game.properties` e `/config/user-interface.properties`, embora existam diff --git a/content/pt-br/docs/reference/glossary/image.md b/content/pt-br/docs/reference/glossary/image.md index 883c5114e9..d521557500 100644 --- a/content/pt-br/docs/reference/glossary/image.md +++ b/content/pt-br/docs/reference/glossary/image.md @@ -1,5 +1,5 @@ --- -title: Image +title: Imagem id: image date: 2021-08-24 full_link: diff --git a/content/pt-br/docs/reference/glossary/selector.md b/content/pt-br/docs/reference/glossary/selector.md index 6d0d8baf71..4fd72b58a0 100644 --- a/content/pt-br/docs/reference/glossary/selector.md +++ b/content/pt-br/docs/reference/glossary/selector.md @@ -1,5 +1,5 @@ --- -title: Selector +title: Seletor id: selector date: 2021-09-17 full_link: /docs/concepts/overview/working-with-objects/labels/ @@ -14,4 +14,4 @@ tags: -Selectors são aplicados quando um usuário faz uma pesquisa por listas de recursos para filtrá-los por labels (rótulos). +Seletores são aplicados quando um usuário faz uma pesquisa por listas de recursos para filtrá-los por labels (rótulos). diff --git a/content/pt-br/docs/reference/glossary/service.md b/content/pt-br/docs/reference/glossary/service.md index b5a415be0b..cc6d976dc4 100644 --- a/content/pt-br/docs/reference/glossary/service.md +++ b/content/pt-br/docs/reference/glossary/service.md @@ -15,6 +15,6 @@ Uma forma abstrata de expor uma aplicação que está executando em um conjunto - O conjunto de Pods referenciado por um Service é (geralmente) determinado por um {{< glossary_tooltip text="selector" term_id="selector" >}}. Se mais Pods são - adicionados ou removidos, o conjunto de Pods que atende ao critério do selector será alterado. O Service garante que o tráfego de rede pode ser direcionado ao + O conjunto de Pods referenciado por um Service é (geralmente) determinado por um {{< glossary_tooltip text="seletor" term_id="selector" >}}. Se mais Pods são + adicionados ou removidos, o conjunto de Pods que atende ao critério do seletor será alterado. O Service garante que o tráfego de rede pode ser direcionado ao conjunto atual de Pods para a carga de trabalho. From bb870812fe37a08337e736bd77d9e7a8f6ecb99a Mon Sep 17 00:00:00 2001 From: bang9211 Date: Thu, 2 Sep 2021 04:45:50 +0900 Subject: [PATCH 042/249] Translate tasks/inject-data-application/environment-variable-expose-pod-information in Korean --- .../distribute-credentials-secure.md | 256 ++++++++++++++++++ ...ronment-variable-expose-pod-information.md | 157 +++++++++++ .../pods/inject/dapi-envars-container.yaml | 45 +++ .../examples/pods/inject/dapi-envars-pod.yaml | 38 +++ .../pod-multiple-secret-env-variable.yaml | 19 ++ .../pods/inject/pod-secret-envFrom.yaml | 11 + .../pod-single-secret-env-variable.yaml | 14 + .../ko/examples/pods/inject/secret-pod.yaml | 17 ++ content/ko/examples/pods/inject/secret.yaml | 7 + 9 files changed, 564 insertions(+) create mode 100644 content/ko/docs/tasks/inject-data-application/distribute-credentials-secure.md create mode 100644 content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md create mode 100644 content/ko/examples/pods/inject/dapi-envars-container.yaml create mode 100644 content/ko/examples/pods/inject/dapi-envars-pod.yaml create mode 100644 content/ko/examples/pods/inject/pod-multiple-secret-env-variable.yaml create mode 100644 content/ko/examples/pods/inject/pod-secret-envFrom.yaml create mode 100644 content/ko/examples/pods/inject/pod-single-secret-env-variable.yaml create mode 100644 content/ko/examples/pods/inject/secret-pod.yaml create mode 100644 content/ko/examples/pods/inject/secret.yaml diff --git a/content/ko/docs/tasks/inject-data-application/distribute-credentials-secure.md b/content/ko/docs/tasks/inject-data-application/distribute-credentials-secure.md new file mode 100644 index 0000000000..0ff081f99a --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/distribute-credentials-secure.md @@ -0,0 +1,256 @@ +--- +title: 시크릿(Secret)을 사용하여 안전하게 자격증명 배포하기 +content_type: task +weight: 50 +min-kubernetes-server-version: v1.6 +--- + + +본 페이지는 암호 및 암호화 키와 같은 민감한 데이터를 파드에 안전하게 +주입하는 방법을 설명한다. + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} + +### 시크릿 데이터를 base-64 표현으로 변환하기 + +사용자 이름 `my-app`과 비밀번호 `39528$vdg7Jb`의 두 가지 시크릿 데이터가 필요하다고 가정한다. +먼저 base64 인코딩 도구를 사용하여 사용자 이름과 암호를 base64 표현으로 변환한다. 다음은 일반적으로 사용 가능한 base64 프로그램을 사용하는 예제이다. + +```shell +echo -n 'my-app' | base64 +echo -n '39528$vdg7Jb' | base64 +``` + +사용자 이름의 base-64 표현이 `bXktYXBw`이고 암호의 base-64 표현이 `Mzk1MjgkdmRnN0pi`임을 +출력을 통해 확인할 수 있다. + +{{< caution >}} +사용자의 OS가 신뢰하는 로컬 툴을 사용하여 외부 툴의 보안 위험을 줄이자. +{{< /caution >}} + + + +## 시크릿 생성하기 + +다음은 사용자 이름과 암호가 들어 있는 시크릿을 생성하는 데 사용할 수 있는 +구성 파일이다. + +{{< codenew file="pods/inject/secret.yaml" >}} + +1. 시크릿을 생성한다. + + ```shell + kubectl apply -f https://k8s.io/examples/pods/inject/secret.yaml + ``` + +2. 시크릿에 대한 정보를 확인한다. + + ```shell + kubectl get secret test-secret + ``` + + Output: + + ``` + NAME TYPE DATA AGE + test-secret Opaque 2 1m + ``` + +3. 시크릿에 대한 자세한 정보를 확인한다. + + ```shell + kubectl describe secret test-secret + ``` + + Output: + + ``` + Name: test-secret + Namespace: default + Labels: + Annotations: + + Type: Opaque + + Data + ==== + password: 13 bytes + username: 7 bytes + ``` + +### kubectl로 직접 시크릿 생성하기 + +Base64 인코딩 단계를 건너뛰려면 `kubectl create secret` 명령을 사용하여 +동일한 Secret을 생성할 수 있다. 다음은 예시이다. + +```shell +kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=39528$vdg7Jb' +``` + +이와 같이 더 편리하게 사용할 수 있다. 앞에서 설명한 자세한 접근 방식은 각 단계를 +명시적으로 실행하여 현재 상황을 확인할 수 있다. + + +## 볼륨을 통해 시크릿 데이터에 접근할 수 있는 파드 생성하기 + +다음은 파드를 생성하는 데 사용할 수 있는 구성 파일이다. + +{{< codenew file="pods/inject/secret-pod.yaml" >}} + +1. 파드를 생성한다. + + ```shell + kubectl apply -f https://k8s.io/examples/pods/inject/secret-pod.yaml + ``` + +2. 파드가 실행중인지 확인한다. + + ```shell + kubectl get pod secret-test-pod + ``` + + Output: + ``` + NAME READY STATUS RESTARTS AGE + secret-test-pod 1/1 Running 0 42m + ``` + +3. 파드에서 실행 중인 컨테이너의 셸을 가져오자. + ```shell + kubectl exec -i -t secret-test-pod -- /bin/bash + ``` + +4. 시크릿 데이터는 `/etc/secret-volume`에 마운트된 볼륨을 통해 +컨테이너에 노출된다. + + 셸에서 `/etc/secret-volume` 디렉터리의 파일을 나열한다. + ```shell + # 컨테이너 내부의 셸에서 실행하자 + ls /etc/secret-volume + ``` + 두 개의 파일과 각 파일의 시크릿 데이터 조각을 확인할 수 있다. + ``` + password username + ``` + +5. 셸에서 `username` 및 `password` 파일의 내용을 출력한다. + ```shell + # 컨테이너 내부의 셸에서 실행하자 + echo "$( cat /etc/secret-volume/username )" + echo "$( cat /etc/secret-volume/password )" + ``` + 사용자 이름과 비밀번호가 출력된다. + ``` + my-app + 39528$vdg7Jb + ``` + +## 시크릿 데이터를 사용하여 컨테이너 환경 변수 정의하기 + +### 단일 시크릿 데이터로 컨테이너 환경 변수 정의하기 + +* 환경 변수를 시크릿의 키-값 쌍으로 정의한다. + + ```shell + kubectl create secret generic backend-user --from-literal=backend-username='backend-admin' + ``` + +* 시크릿에 정의된 `backend-username` 값을 파드 명세의 `SECRET_USERNAME` 환경 변수에 할당한다. + + {{< codenew file="pods/inject/pod-single-secret-env-variable.yaml" >}} + +* 파드를 생성한다. + + ```shell + kubectl create -f https://k8s.io/examples/pods/inject/pod-single-secret-env-variable.yaml + ``` + +* 셸에서 `SECRET_USERNAME` 컨테이너 환경 변수의 내용을 출력한다. + + ```shell + kubectl exec -i -t env-single-secret -- /bin/sh -c 'echo $SECRET_USERNAME' + ``` + + 출력은 다음과 같다. + ``` + backend-admin + ``` + +### 여러 시크릿 데이터로 컨테이너 환경 변수 정의하기 + +* 이전 예제와 마찬가지로 시크릿을 먼저 생성한다. + + ```shell + kubectl create secret generic backend-user --from-literal=backend-username='backend-admin' + kubectl create secret generic db-user --from-literal=db-username='db-admin' + ``` + +* 파드 명세에 환경 변수를 정의한다. + + {{< codenew file="pods/inject/pod-multiple-secret-env-variable.yaml" >}} + +* 파드를 생성한다. + + ```shell + kubectl create -f https://k8s.io/examples/pods/inject/pod-multiple-secret-env-variable.yaml + ``` + +* 셸에서 컨테이너 환경 변수를 출력한다. + + ```shell + kubectl exec -i -t envvars-multiple-secrets -- /bin/sh -c 'env | grep _USERNAME' + ``` + 출력은 다음과 같다. + ``` + DB_USERNAME=db-admin + BACKEND_USERNAME=backend-admin + ``` + + +## 시크릿의 모든 키-값 쌍을 컨테이너 환경 변수로 구성하기 + +{{< note >}} +이 기능은 쿠버네티스 v1.6 이상에서 사용할 수 있다. +{{< /note >}} + +* 여러 키-값 쌍을 포함하는 시크릿을 생성한다. + + ```shell + kubectl create secret generic test-secret --from-literal=username='my-app' --from-literal=password='39528$vdg7Jb' + ``` + +* envFrom을 사용하여 시크릿의 모든 데이터를 컨테이너 환경 변수로 정의한다. 시크릿의 키는 파드에서 환경 변수의 이름이 된다. + + {{< codenew file="pods/inject/pod-secret-envFrom.yaml" >}} + +* 파드를 생성한다. + + ```shell + kubectl create -f https://k8s.io/examples/pods/inject/pod-secret-envFrom.yaml + ``` + +* `username` 및 `password` 컨테이너 환경 변수를 셸에서 출력한다. + + ```shell + kubectl exec -i -t envfrom-secret -- /bin/sh -c 'echo "username: $username\npassword: $password\n"' + ``` + + 출력은 다음과 같다. + ``` + username: my-app + password: 39528$vdg7Jb + ``` + +### 참고 + +* [시크릿](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#secret-v1-core) +* [볼륨](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core) +* [파드](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) + +## {{% heading "whatsnext" %}} + +* [시크릿](/ko/docs/concepts/configuration/secret/)에 대해 더 배워 보기. +* [볼륨](/ko/docs/concepts/storage/volumes/)에 대해 더 배워 보기. diff --git a/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md b/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md new file mode 100644 index 0000000000..033e0afa79 --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/environment-variable-expose-pod-information.md @@ -0,0 +1,157 @@ +--- +title: 환경 변수로 컨테이너에 파드 정보 노출하기 +content_type: task +weight: 30 +--- + + + +본 페이지는 파드에서 실행 중인 컨테이너에게 파드가 환경 변수를 사용해서 자신의 정보를 노출하는 방법에 +대해 설명한다. 환경 변수는 파드 필드와 컨테이너 필드를 노출할 수 있다. + + + + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + + + +## 다운워드(Downward) API + +파드 및 컨테이너 필드를 실행 중인 컨테이너에 노출하는 두 가지 방법이 있다. + +* 환경 변수 +* [볼륨 파일](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api) + +파드 및 컨테이너 필드를 노출하는 이 두 가지 방법을 *다운워드 API*라고 한다. + + +## 파드 필드를 환경 변수의 값으로 사용하자 + +이 연습에서는 하나의 컨테이너를 가진 파드를 생성한다. 다음은 파드에 대한 구성 파일이다. + +{{< codenew file="pods/inject/dapi-envars-pod.yaml" >}} + +구성 파일에서 5개의 환경 변수를 확인할 수 있다. `env` 필드는 +[EnvVars](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core)의 배열이다. 배열의 첫 번째 요소는 `MY_NODE_NAME` 환경 변수가 파드의 `spec.nodeName` 필드에서 값을 가져오도록 지정한다. 마찬가지로 다른 환경 변수도 파드 필드에서 이름을 가져온다. + +{{< note >}} +이 예제의 필드는 파드에 있는 컨테이너의 필드가 아니라 파드 필드이다. +{{< /note >}} + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-pod.yaml +``` + +파드의 컨테이너가 실행중인지 확인한다. + +```shell +kubectl get pods +``` + +컨테이너의 로그를 본다. + +```shell +kubectl logs dapi-envars-fieldref +``` + +출력은 선택된 환경 변수의 값을 보여준다. + +``` +minikube +dapi-envars-fieldref +default +172.17.0.4 +default +``` + +이러한 값이 로그에 출력된 이유를 보려면 구성 파일의 `command` 및 `args` 필드를 확인하자. +컨테이너가 시작되면 5개의 환경 변수 값을 stdout에 쓰며 10초마다 이를 반복한다. + +다음으로 파드에서 실행 중인 컨테이너의 셸을 가져오자. + +```shell +kubectl exec -it dapi-envars-fieldref -- sh +``` + +셸에서 환경 변수를 보자. + +```shell +/# printenv +``` + +출력은 특정 환경 변수에 파드 필드 값이 할당되었음을 보여준다. + +``` +MY_POD_SERVICE_ACCOUNT=default +... +MY_POD_NAMESPACE=default +MY_POD_IP=172.17.0.4 +... +MY_NODE_NAME=minikube +... +MY_POD_NAME=dapi-envars-fieldref +``` + +## 컨테이너 필드를 환경 변수의 값으로 사용하기 + +이전 연습에서는 파드 필드를 환경 변수의 값으로 사용했다. 이 다음 연습에서는 컨테이너 필드를 +환경 변수의 값으로 사용한다. 다음은 하나의 컨테이너가 있는 파드의 구성 파일이다. + +{{< codenew file="pods/inject/dapi-envars-container.yaml" >}} + +구성 파일에서 4개의 환경 변수를 확인할 수 있다. `env` 필드는 +[EnvVars](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core)의 배열이다. 배열의 첫 번째 요소는 `MY_CPU_REQUEST` 환경 변수가 `test-container`라는 컨테이너의 +`requests.cpu` 필드에서 값을 가져오도록 지정한다. 마찬가지로 다른 환경 변수도 컨테이너 필드에서 +값을 가져온다. + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-container.yaml +``` + +파드의 컨테이너가 실행중인지 확인한다. + +```shell +kubectl get pods +``` + +컨테이너의 로그를 본다. + +```shell +kubectl logs dapi-envars-resourcefieldref +``` + +출력은 선택된 환경 변수의 값을 보여준다. + +``` +1 +1 +33554432 +67108864 +``` + + + +## {{% heading "whatsnext" %}} + + +* [컨테이너를 위한 환경 변수 정의하기](/docs/tasks/inject-data-application/define-environment-variable-container/) +* [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core) +* [컨테이너](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) +* [EnvVar](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvar-v1-core) +* [EnvVarSource](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#envvarsource-v1-core) +* [ObjectFieldSelector](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#objectfieldselector-v1-core) +* [ResourceFieldSelector](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcefieldselector-v1-core) + + + diff --git a/content/ko/examples/pods/inject/dapi-envars-container.yaml b/content/ko/examples/pods/inject/dapi-envars-container.yaml new file mode 100644 index 0000000000..55bd4dd263 --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-envars-container.yaml @@ -0,0 +1,45 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-resourcefieldref +spec: + containers: + - name: test-container + image: k8s.gcr.io/busybox:1.24 + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_CPU_REQUEST MY_CPU_LIMIT; + printenv MY_MEM_REQUEST MY_MEM_LIMIT; + sleep 10; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + env: + - name: MY_CPU_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.cpu + - name: MY_CPU_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.cpu + - name: MY_MEM_REQUEST + valueFrom: + resourceFieldRef: + containerName: test-container + resource: requests.memory + - name: MY_MEM_LIMIT + valueFrom: + resourceFieldRef: + containerName: test-container + resource: limits.memory + restartPolicy: Never diff --git a/content/ko/examples/pods/inject/dapi-envars-pod.yaml b/content/ko/examples/pods/inject/dapi-envars-pod.yaml new file mode 100644 index 0000000000..071fa82bb3 --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-envars-pod.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: dapi-envars-fieldref +spec: + containers: + - name: test-container + image: k8s.gcr.io/busybox + command: [ "sh", "-c"] + args: + - while true; do + echo -en '\n'; + printenv MY_NODE_NAME MY_POD_NAME MY_POD_NAMESPACE; + printenv MY_POD_IP MY_POD_SERVICE_ACCOUNT; + sleep 10; + done; + env: + - name: MY_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: MY_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: MY_POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: MY_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: MY_POD_SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName + restartPolicy: Never diff --git a/content/ko/examples/pods/inject/pod-multiple-secret-env-variable.yaml b/content/ko/examples/pods/inject/pod-multiple-secret-env-variable.yaml new file mode 100644 index 0000000000..f285e41932 --- /dev/null +++ b/content/ko/examples/pods/inject/pod-multiple-secret-env-variable.yaml @@ -0,0 +1,19 @@ +apiVersion: v1 +kind: Pod +metadata: + name: envvars-multiple-secrets +spec: + containers: + - name: envars-test-container + image: nginx + env: + - name: BACKEND_USERNAME + valueFrom: + secretKeyRef: + name: backend-user + key: backend-username + - name: DB_USERNAME + valueFrom: + secretKeyRef: + name: db-user + key: db-username diff --git a/content/ko/examples/pods/inject/pod-secret-envFrom.yaml b/content/ko/examples/pods/inject/pod-secret-envFrom.yaml new file mode 100644 index 0000000000..eb1d3213ef --- /dev/null +++ b/content/ko/examples/pods/inject/pod-secret-envFrom.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Pod +metadata: + name: envfrom-secret +spec: + containers: + - name: envars-test-container + image: nginx + envFrom: + - secretRef: + name: test-secret diff --git a/content/ko/examples/pods/inject/pod-single-secret-env-variable.yaml b/content/ko/examples/pods/inject/pod-single-secret-env-variable.yaml new file mode 100644 index 0000000000..af4cf8732f --- /dev/null +++ b/content/ko/examples/pods/inject/pod-single-secret-env-variable.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Pod +metadata: + name: env-single-secret +spec: + containers: + - name: envars-test-container + image: nginx + env: + - name: SECRET_USERNAME + valueFrom: + secretKeyRef: + name: backend-user + key: backend-username diff --git a/content/ko/examples/pods/inject/secret-pod.yaml b/content/ko/examples/pods/inject/secret-pod.yaml new file mode 100644 index 0000000000..8be694cdde --- /dev/null +++ b/content/ko/examples/pods/inject/secret-pod.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Pod +metadata: + name: secret-test-pod +spec: + containers: + - name: test-container + image: nginx + volumeMounts: + # name must match the volume name below + - name: secret-volume + mountPath: /etc/secret-volume + # The secret data is exposed to Containers in the Pod through a Volume. + volumes: + - name: secret-volume + secret: + secretName: test-secret diff --git a/content/ko/examples/pods/inject/secret.yaml b/content/ko/examples/pods/inject/secret.yaml new file mode 100644 index 0000000000..706ca8670f --- /dev/null +++ b/content/ko/examples/pods/inject/secret.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Secret +metadata: + name: test-secret +data: + username: bXktYXBw + password: Mzk1MjgkdmRnN0pi From 5d0c9eae0e4103e88cc550a2c29c5652a3dc524c Mon Sep 17 00:00:00 2001 From: radva <45208722+radva@users.noreply.github.com> Date: Tue, 21 Sep 2021 17:53:57 +0200 Subject: [PATCH 043/249] Update translate-compose-kubernetes.md update to latest release version since version 1.22 doesn't support 3.8 --- .../configure-pod-container/translate-compose-kubernetes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md b/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md index 384b709720..17fc9e1421 100644 --- a/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md +++ b/content/en/docs/tasks/configure-pod-container/translate-compose-kubernetes.md @@ -29,13 +29,13 @@ Kompose is released via GitHub on a three-week cycle, you can see all current re ```sh # Linux -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-amd64 -o kompose +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-linux-amd64 -o kompose # macOS -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-darwin-amd64 -o kompose +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-darwin-amd64 -o kompose # Windows -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-windows-amd64.exe -o kompose.exe +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-windows-amd64.exe -o kompose.exe chmod +x kompose sudo mv ./kompose /usr/local/bin/kompose From 93c957e9a60b59556715a09744d719717aa9698b Mon Sep 17 00:00:00 2001 From: Mengjiao Liu Date: Wed, 22 Sep 2021 17:37:19 +0800 Subject: [PATCH 044/249] [zh] fix expiration of bound SA tokens --- .../docs/reference/access-authn-authz/service-accounts-admin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/reference/access-authn-authz/service-accounts-admin.md b/content/zh/docs/reference/access-authn-authz/service-accounts-admin.md index 451c8d4b3b..d3faacf58f 100644 --- a/content/zh/docs/reference/access-authn-authz/service-accounts-admin.md +++ b/content/zh/docs/reference/access-authn-authz/service-accounts-admin.md @@ -143,7 +143,7 @@ ServiceAccount 准入控制器将添加如下投射卷，而不是为令牌控 defaultMode: 420 # 0644 sources: - serviceAccountToken: - expirationSeconds: 3600 + expirationSeconds: 3607 path: token - configMap: items: From 2cd822f120b7b2ea7bf2d22557946160669bc7e8 Mon Sep 17 00:00:00 2001 From: fabriziopandini Date: Wed, 22 Sep 2021 11:48:18 +0200 Subject: [PATCH 045/249] ClusterClass and managed topologies blog --- ...-04-clusterclass-and-managed-topologies.md | 143 ++++++++++++++++++ .../clusterclass.svg | 4 + .../create-cluster.svg | 4 + 3 files changed, 151 insertions(+) create mode 100644 content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md create mode 100644 static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg create mode 100644 static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg diff --git a/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md b/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md new file mode 100644 index 0000000000..b576b345e9 --- /dev/null +++ b/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md @@ -0,0 +1,143 @@ +--- +layout: blog +title: "Introducing ClusterClass and Managed Topologies in Cluster API" +date: 2021-10-04 +slug: capi-clusterclass-and-managed-topologies +--- + +**Author:** Fabrizio Pandini (VMware) + +The [Cluster API community](https://cluster-api.sigs.k8s.io/) is happy to announce the implementation of *ClusterClass and Managed Topologies*, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way. + +## A little bit of context… + +Before getting into the details, let's take a step back and look at the history of Cluster API. + +The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting. + +With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more. + +Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs and documentation, and on extensive test signals which inform Kubernetes releases. + +With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users. + +Enter ClusterClass and manage topologies, tada! + +## ClusterClass + +As the name suggests, ClusterClass and managed topologies are built in two parts. + +The idea behind ClusterClass is simple: let’s define the shape of your cluster once, and then reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. + +![Defining a ClusterClass](/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg) + +ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape. + +```yaml +--- +apiVersion: cluster.x-k8s.io/v1beta1 + kind: ClusterClass + metadata: + name: my-amazing-cluster-class + namespace: bar + spec: + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlaneTemplate + name: high-availability-kcp + machineInfrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 + kind: VSphereMachineTemplate + name: controlplane-vsphere-machinetemplate + workers: + deployments: + - class: linux-worker + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: linux-bootstrap + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 + kind: VSphereMachineTemplate + name: linux-vsphere-template + - class: windows-worker + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: windows-bootstrap + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 + kind: VSphereMachineTemplate + name: windows-vsphere-template + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 + kind: VSphereClusterTemplate + name: vsphere-cluster +``` + +The possibilities are endless; you can get a default ClusterClass from the community, “off-the shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios. + +## Managed Topologies + +Managed topologies let you put the power that ClusterClass provides into action. + +Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster. + +![Create a Cluster with ClusterClass](/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg) + +Here is an example: + +```yaml +--- +apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + metadata: + name: my-amazing-cluster + namespace: bar + spec: + topology: # define a managed topology + class: my-amazing-cluster-class # use the ClusterClass mentioned earlier + version: v1.21.2 + controlPlane: + replicas: 3 + workers: + machineDeployments: + - class: linux-worker + name: big-pool-of-linux-machines-1 + replicas: 5 + - class: linux-worker + name: small-pool-of-linux-machines-1 + replicas: 1 + - class: windows-worker + name: pool-of-windows-machines + replicas: 3 +``` + +But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology. + +All the power of Cluster API, extensibility, lifecycle automation, stability, all the features required for managing an enterprise grade Kubernetes cluster on the infrastructure provider of your choice are now at your fingertips: you can create your Cluster, add new machines, upgrade to the next Kubernetes version, and all from a single place. + +It is just as simple as it looks! + +## What’s next + +While the amazing Cluster API community is working hard to deliver the first version of ClusterClass and managed topologies later this year, we are already looking forward to what comes next for the project and its ecosystem. + +There are a lot of great ideas and opportunities ahead! + +We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on. + +Stay tuned for what comes next, and if you have any question, comment or suggestion: + +* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T) +* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents +* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time diff --git a/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg b/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg new file mode 100644 index 0000000000..c22b4b9ff0 --- /dev/null +++ b/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg @@ -0,0 +1,4 @@ + + + +
    ClusterClass
    ClusterClass
    KubeadmConfigTemplate
    KubeadmConfigTemplate
    VSphereMachineTemplate
    VSphereMachineTemplate
    VSphereClusterTemplate
    VSphereClusterTemplate
    MachineDeploymentClass
    MachineDeploymentClass
    VSphereMachineTemplate
    VSphereMachineTemplate
    KubeadmControlPlaneTemplate
    KubeadmControlPlaneTemplate
    Cluster
    Cluster
    Cluster
    Cluster
    Cluster
    Cluster    Viewer does not support full SVG 1.1     \ No newline at end of file diff --git a/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg b/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg new file mode 100644 index 0000000000..d26d49d7bb --- /dev/null +++ b/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg @@ -0,0 +1,4 @@ + + + +
    ClusterClass
    ClusterClass
    Templates
    Templates
    Cluster
    Cluster
    Managed
    Topology
    Managed...
    Cluster
    Cluster
    Managed
    Topology
    Managed...
    Cluster
    Cluster
    Managed
    Topology
    Managed...    Viewer does not support full SVG 1.1     \ No newline at end of file From b4d1304c2ace4d9f9962eb1c578f2ab589a9ba8b Mon Sep 17 00:00:00 2001 From: Mengjiao Liu Date: Wed, 22 Sep 2021 18:25:20 +0800 Subject: [PATCH 046/249] [zh] Sync administer-cluster kubeadm-upgrade.md --- .../kubeadm/kubeadm-upgrade.md | 88 +++++++++---------- 1 file changed, 43 insertions(+), 45 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 81623a6b4c..25f10b0fed 100644 --- a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -16,13 +16,13 @@ min-kubernetes-server-version: 1.18 -本页介绍如何将 `kubeadm` 创建的 Kubernetes 集群从 {{< skew latestVersionAddMinor -1 >}}.x 版本 -升级到 {{< skew latestVersion >}}.x 版本以及从 {{< skew latestVersion >}}.x -升级到 {{< skew latestVersion >}}.y（其中 `y > x`）。略过次版本号的升级是 +本页介绍如何将 `kubeadm` 创建的 Kubernetes 集群从 {{< skew currentVersionAddMinor -1 >}}.x 版本 +升级到 {{< skew currentVersion >}}.x 版本以及从 {{< skew currentVersion >}}.x +升级到 {{< skew currentVersion >}}.y（其中 `y > x`）。略过次版本号的升级是 不被支持的。 -- [将 kubeadm 集群从 1.17 升级到 1.18](https://v1-18.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) -- [将 kubeadm 集群从 1.16 升级到 1.17](https://v1-17.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) -- [将 kubeadm 集群从 1.15 升级到 1.16](https://v1-16.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) -- [将 kubeadm 集群从 1.14 升级到 1.15](https://v1-15.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/) -- [将 kubeadm 集群从 1.13 升级到 1.14](https://v1-15.docs.kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-14/) +- [将 kubeadm 集群从 {{< skew currentVersionAddMinor -2 >}} 升级到 {{< skew currentVersionAddMinor -1 >}}](https://v{{< skew currentVersionAddMinor -1 "-" >}}.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) +- [将 kubeadm 集群从 {{< skew currentVersionAddMinor -3 >}} 升级到 {{< skew currentVersionAddMinor -2 >}}](https://v{{< skew currentVersionAddMinor -2 "-" >}}.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) +- [将 kubeadm 集群从 {{< skew currentVersionAddMinor -4 >}} 升级到 {{< skew currentVersionAddMinor -3 >}}](https://v{{< skew currentVersionAddMinor -3 "-" >}}.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) +- [将 kubeadm 集群从 {{< skew currentVersionAddMinor -5 >}} 升级到 {{< skew currentVersionAddMinor -4 >}}](https://v{{< skew currentVersionAddMinor -4 "-" >}}.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) ## 确定要升级到哪个版本 -使用操作系统的包管理器找到最新的稳定 {{< skew latestVersion >}}： +使用操作系统的包管理器找到最新的补丁版本 Kubernetes {{< skew currentVersion >}}： {{< tabs name="k8s_install_versions" >}} {{% tab name="Ubuntu、Debian 或 HypriotOS" %}} ``` apt update -apt-cache policy kubeadm -# 在列表中查找最新的 {{< skew latestVersion >}} 版本 -# 它看起来应该是 {{< skew latestVersion >}}.x-00，其中 x 是最新的补丁版本 +apt-cache madison kubeadm +# 在列表中查找最新的 {{< skew currentVersion >}} 版本 +# 它看起来应该是 {{< skew currentVersion >}}.x-00，其中 x 是最新的补丁版本 ``` {{% /tab %}} {{% tab name="CentOS、RHEL 或 Fedora" %}} ``` yum list --showduplicates kubeadm --disableexcludes=kubernetes -# 在列表中查找最新的 {{< skew latestVersion >}} 版本 -# 它看起来应该是 {{< skew latestVersion >}}.x-0，其中 x 是最新的补丁版本 +# 在列表中查找最新的 {{< skew currentVersion >}} 版本 +# 它看起来应该是 {{< skew currentVersion >}}.x-0，其中 x 是最新的补丁版本 ``` {{% /tab %}} {{< /tabs >}} @@ -143,20 +141,20 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc {{< tabs name="k8s_install_kubeadm_first_cp" >}} {{% tab name="Ubuntu、Debian 或 HypriotOS" %}} ```shell -# 用最新的补丁版本号替换 {{< skew latestVersion >}}.x-00 中的 x +# 用最新的补丁版本号替换 {{< skew currentVersion >}}.x-00 中的 x apt-mark unhold kubeadm && \ -apt-get update && apt-get install -y kubeadm={{< skew latestVersion >}}.x-00 && \ +apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \ apt-mark hold kubeadm - # 从 apt-get 1.1 版本起，你也可以使用下面的方法 apt-get update && \ -apt-get install -y --allow-change-held-packages kubeadm={{< skew latestVersion >}}.x-00 +apt-get install -y --allow-change-held-packages kubeadm={{< skew currentVersion >}}.x-00 ``` {{% /tab %}} {{% tab name="CentOS、RHEL 或 Fedora" %}} ```shell -# 用最新的补丁版本号替换 {{< skew latestVersion >}}.x-0 中的 x -yum install -y kubeadm-{{< skew latestVersion >}}.x-0 --disableexcludes=kubernetes +# 用最新的补丁版本号替换 {{< skew currentVersion >}}.x-0 中的 x +yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes ``` {{% /tab %}} {{< /tabs >}} @@ -213,14 +211,14 @@ yum install -y kubeadm-{{< skew latestVersion >}}.x-0 --disableexcludes=kubernet ```shell # replace x with the patch version you picked for this upgrade - sudo kubeadm upgrade apply v{{< skew latestVersion >}}.x + sudo kubeadm upgrade apply v{{< skew currentVersion >}}.x ``` --> 选择要升级到的目标版本，运行合适的命令。例如： ```shell # 将 x 替换为你为此次升级所选择的补丁版本号 - sudo kubeadm upgrade apply v{{< skew latestVersion >}}.x + sudo kubeadm upgrade apply v{{< skew currentVersion >}}.x ``` This page covers how to customize the components that kubeadm deploys. For control plane components -you can use flags in the `ClusteConfiguration` structure or patches per-node. For the kubelet +you can use flags in the `ClusterConfiguration` structure or patches per-node. For the kubelet and kube-proxy you can use `KubeletConfiguration` and `KubeProxyConfiguration`, accordingly. All of these options are possible via the kubeadm configuration API. From 4a275f1426edcc1a6041484a611457e8cdb32da4 Mon Sep 17 00:00:00 2001 From: aldlfkahs Date: Thu, 23 Sep 2021 13:31:45 +0900 Subject: [PATCH 052/249] [ko] Fix branch of link from 'param githubbranch' to 'master' --- .../cluster-administration/logging.md | 2 +- .../manage-deployment.md | 2 +- .../docs/concepts/configuration/overview.md | 4 ++-- .../containers/container-environment.md | 2 +- .../connect-applications-service.md | 6 ++--- .../concepts/services-networking/service.md | 2 +- content/ko/docs/concepts/storage/volumes.md | 24 +++++++++---------- .../concepts/workloads/controllers/job.md | 2 +- .../workloads/controllers/statefulset.md | 2 +- .../administer-cluster/access-cluster-api.md | 2 +- .../dns-custom-nameservers.md | 2 +- .../basic-stateful-set.md | 2 +- .../stateful-application/zookeeper.md | 2 +- 13 files changed, 27 insertions(+), 27 deletions(-) diff --git a/content/ko/docs/concepts/cluster-administration/logging.md b/content/ko/docs/concepts/cluster-administration/logging.md index d4e0119c41..5ae2aafffd 100644 --- a/content/ko/docs/concepts/cluster-administration/logging.md +++ b/content/ko/docs/concepts/cluster-administration/logging.md @@ -80,7 +80,7 @@ kubectl logs counter 로테이션하도록 컨테이너 런타임을 설정할 수도 있다. 예를 들어, `kube-up.sh` 가 GCP의 COS 이미지 로깅을 설정하는 방법은 -[`configure-helper` 스크립트](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh)를 통해 +[`configure-helper` 스크립트](https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh)를 통해 자세히 알 수 있다. **CRI 컨테이너 런타임** 을 사용할 때, kubelet은 로그를 로테이션하고 로깅 디렉터리 구조를 관리한다. diff --git a/content/ko/docs/concepts/cluster-administration/manage-deployment.md b/content/ko/docs/concepts/cluster-administration/manage-deployment.md index 7e3093d51e..553ce005a9 100644 --- a/content/ko/docs/concepts/cluster-administration/manage-deployment.md +++ b/content/ko/docs/concepts/cluster-administration/manage-deployment.md @@ -160,7 +160,7 @@ persistentvolumeclaim/my-pvc created 지금까지 사용한 예는 모든 리소스에 최대 한 개의 레이블만 적용하는 것이었다. 세트를 서로 구별하기 위해 여러 레이블을 사용해야 하는 많은 시나리오가 있다. -예를 들어, 애플리케이션마다 `app` 레이블에 다른 값을 사용하지만, [방명록 예제](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/)와 같은 멀티-티어 애플리케이션은 각 티어를 추가로 구별해야 한다. 프론트엔드는 다음의 레이블을 가질 수 있다. +예를 들어, 애플리케이션마다 `app` 레이블에 다른 값을 사용하지만, [방명록 예제](https://github.com/kubernetes/examples/tree/master/guestbook/)와 같은 멀티-티어 애플리케이션은 각 티어를 추가로 구별해야 한다. 프론트엔드는 다음의 레이블을 가질 수 있다. ```yaml labels: diff --git a/content/ko/docs/concepts/configuration/overview.md b/content/ko/docs/concepts/configuration/overview.md index 17fd61c6a6..f89d873a37 100644 --- a/content/ko/docs/concepts/configuration/overview.md +++ b/content/ko/docs/concepts/configuration/overview.md @@ -21,7 +21,7 @@ weight: 10 - JSON보다는 YAML을 사용해 구성 파일을 작성한다. 비록 이러한 포맷들은 대부분의 모든 상황에서 통용되어 사용될 수 있지만, YAML이 좀 더 사용자 친화적인 성향을 가진다. -- 의미상 맞다면 가능한 연관된 오브젝트들을 하나의 파일에 모아 놓는다. 때로는 여러 개의 파일보다 하나의 파일이 더 관리하기 쉽다. 이 문법의 예시로서 [guestbook-all-in-one.yaml](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/all-in-one/guestbook-all-in-one.yaml) 파일을 참고한다. +- 의미상 맞다면 가능한 연관된 오브젝트들을 하나의 파일에 모아 놓는다. 때로는 여러 개의 파일보다 하나의 파일이 더 관리하기 쉽다. 이 문법의 예시로서 [guestbook-all-in-one.yaml](https://github.com/kubernetes/examples/tree/master/guestbook/all-in-one/guestbook-all-in-one.yaml) 파일을 참고한다. - 많은 `kubectl` 커맨드들은 디렉터리에 대해 호출될 수 있다. 예를 들어, 구성 파일들의 디렉터리에 대해 `kubectl apply`를 호출할 수 있다. @@ -63,7 +63,7 @@ DNS 서버는 새로운 `서비스`를 위한 쿠버네티스 API를 Watch하며 ## 레이블 사용하기 -- `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`처럼 애플리케이션이나 디플로이먼트의 __속성에 대한 의미__ 를 식별하는 [레이블](/ko/docs/concepts/overview/working-with-objects/labels/)을 정의해 사용한다. 다른 리소스를 위해 적절한 파드를 선택하는 용도로 이러한 레이블을 이용할 수 있다. 예를 들어, 모든 `tier: frontend` 파드를 선택하거나, `app: myapp`의 모든 `phase: test` 컴포넌트를 선택하는 서비스를 생각해 볼 수 있다. 이 접근 방법의 예시는 [방명록](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/) 앱을 참고한다. +- `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`처럼 애플리케이션이나 디플로이먼트의 __속성에 대한 의미__ 를 식별하는 [레이블](/ko/docs/concepts/overview/working-with-objects/labels/)을 정의해 사용한다. 다른 리소스를 위해 적절한 파드를 선택하는 용도로 이러한 레이블을 이용할 수 있다. 예를 들어, 모든 `tier: frontend` 파드를 선택하거나, `app: myapp`의 모든 `phase: test` 컴포넌트를 선택하는 서비스를 생각해 볼 수 있다. 이 접근 방법의 예시는 [방명록](https://github.com/kubernetes/examples/tree/master/guestbook/) 앱을 참고한다. 릴리스에 특정되는 레이블을 서비스의 셀렉터에서 생략함으로써 여러 개의 디플로이먼트에 걸치는 서비스를 생성할 수 있다. 동작 중인 서비스를 다운타임 없이 갱신하려면, [디플로이먼트](/ko/docs/concepts/workloads/controllers/deployment/)를 사용한다. diff --git a/content/ko/docs/concepts/containers/container-environment.md b/content/ko/docs/concepts/containers/container-environment.md index 58c106fdce..749eb6fbb8 100644 --- a/content/ko/docs/concepts/containers/container-environment.md +++ b/content/ko/docs/concepts/containers/container-environment.md @@ -52,7 +52,7 @@ FOO_SERVICE_HOST=<서비스가 동작 중인 호스트> FOO_SERVICE_PORT=<서비스가 동작 중인 포트> ``` -서비스에 지정된 IP 주소가 있고 [DNS 애드온](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/)이 활성화된 경우, DNS를 통해서 컨테이너가 서비스를 사용할 수 있다. +서비스에 지정된 IP 주소가 있고 [DNS 애드온](https://releases.k8s.io/master/cluster/addons/dns/)이 활성화된 경우, DNS를 통해서 컨테이너가 서비스를 사용할 수 있다. diff --git a/content/ko/docs/concepts/services-networking/connect-applications-service.md b/content/ko/docs/concepts/services-networking/connect-applications-service.md index 0848c35772..2f5a9064da 100644 --- a/content/ko/docs/concepts/services-networking/connect-applications-service.md +++ b/content/ko/docs/concepts/services-networking/connect-applications-service.md @@ -129,7 +129,7 @@ curl을 할 수 있을 것이다. 서비스 IP는 완전히 가상이므로 외 쿠버네티스는 서비스를 찾는 두 가지 기본 모드인 환경 변수와 DNS를 지원한다. 전자는 기본적으로 작동하지만 후자는 -[CoreDNS 클러스터 애드온](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/coredns)이 필요하다. +[CoreDNS 클러스터 애드온](https://releases.k8s.io/master/cluster/addons/dns/coredns)이 필요하다. {{< note >}} 만약 서비스 환경 변수가 필요하지 않은 경우(소유한 프로그램과의 예상되는 충돌 가능성, 처리할 변수가 너무 많은 경우, DNS만 사용하는 경우 등) [파드 사양](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core)에서 @@ -227,7 +227,7 @@ Address 1: 10.0.162.149 * 인증서를 사용하도록 구성된 nginx 서버 * 파드에 접근할 수 있는 인증서를 만드는 [시크릿](/ko/docs/concepts/configuration/secret/) -[nginx https 예제](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/https-nginx/)에서 이 모든 것을 얻을 수 있다. 이를 위해서는 도구를 설치해야 한다. 만약 설치하지 않으려면 나중에 수동으로 단계를 수행한다. 한마디로: +[nginx https 예제](https://github.com/kubernetes/examples/tree/master/staging/https-nginx/)에서 이 모든 것을 얻을 수 있다. 이를 위해서는 도구를 설치해야 한다. 만약 설치하지 않으려면 나중에 수동으로 단계를 수행한다. 한마디로: ```shell make keys KEY=/tmp/nginx.key CERT=/tmp/nginx.crt @@ -299,7 +299,7 @@ nginxsecret kubernetes.io/tls 2 1m nginx-secure-app의 매니페스트에 대한 주목할만한 점: - 이것은 동일한 파일에 디플로이먼트와 서비스의 사양을 모두 포함하고 있다. -- [nginx 서버](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/https-nginx/default.conf) +- [nginx 서버](https://github.com/kubernetes/examples/tree/master/staging/https-nginx/default.conf) 는 포트 80에서 HTTP 트래픽을 443에서 HTTPS 트래픽 서비스를 제공하고, nginx 서비스는 두 포트를 모두 노출한다. - 각 컨테이너는 `/etc/nginx/ssl` 에 마운트된 볼륨을 통해 키에 접근할 수 있다. diff --git a/content/ko/docs/concepts/services-networking/service.md b/content/ko/docs/concepts/services-networking/service.md index 5c4b9edeee..710e462a67 100644 --- a/content/ko/docs/concepts/services-networking/service.md +++ b/content/ko/docs/concepts/services-networking/service.md @@ -394,7 +394,7 @@ CIDR 범위 내의 유효한 IPv4 또는 IPv6 주소여야 한다. 파드가 노드에서 실행될 때, kubelet은 각 활성화된 서비스에 대해 환경 변수 세트를 추가한다. [도커 링크 호환](https://docs.docker.com/userguide/dockerlinks/) 변수 -([makeLinkVariables](https://releases.k8s.io/{{< param "githubbranch" >}}/pkg/kubelet/envvars/envvars.go#L49) 참조)와 +([makeLinkVariables](https://releases.k8s.io/master/pkg/kubelet/envvars/envvars.go#L49) 참조)와 보다 간단한 `{SVCNAME}_SERVICE_HOST` 및 `{SVCNAME}_SERVICE_PORT` 변수를 지원하고, 이때 서비스 이름은 대문자이고 대시는 밑줄로 변환된다. diff --git a/content/ko/docs/concepts/storage/volumes.md b/content/ko/docs/concepts/storage/volumes.md index 29f4755172..050566388e 100644 --- a/content/ko/docs/concepts/storage/volumes.md +++ b/content/ko/docs/concepts/storage/volumes.md @@ -130,7 +130,7 @@ EBS 볼륨이 파티션된 경우, 선택적 필드인 `partition: "}}/staging/volumes/azure_disk/README.md)을 참고한다. +더 자세한 내용은 [`azureDisk` 볼륨 플러그인](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_disk/README.md)을 참고한다. #### azureDisk CSI 마이그레이션 @@ -148,7 +148,7 @@ EBS 볼륨이 파티션된 경우, 선택적 필드인 `partition: "}}/staging/volumes/azure_file/README.md)을 참고한다. +더 자세한 내용은 [`azureFile` 볼륨 플러그인](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_file/README.md)을 참고한다. #### azureFile CSI 마이그레이션 @@ -176,7 +176,7 @@ Azure File CSI 드라이버는 동일한 볼륨을 다른 fsgroup에서 사용 CephFS를 사용하기 위해선 먼저 Ceph 서버를 실행하고 공유를 내보내야 한다. {{< /note >}} -더 자세한 내용은 [CephFS 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/cephfs/)를 참조한다. +더 자세한 내용은 [CephFS 예시](https://github.com/kubernetes/examples/tree/master/volumes/cephfs/)를 참조한다. ### cinder @@ -347,7 +347,7 @@ targetWWN은 해당 WWN이 다중 경로 연결에서 온 것으로 예상한다 쿠버네티스 호스트가 해당 LUN에 접근할 수 있다. {{< /note >}} -더 자세한 내용은 [파이버 채널 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/fibre_channel)를 참고한다. +더 자세한 내용은 [파이버 채널 예시](https://github.com/kubernetes/examples/tree/master/staging/volumes/fibre_channel)를 참고한다. ### flocker (사용 중단됨(deprecated)){#flocker} @@ -365,7 +365,7 @@ Flocker는 파드가 스케줄 되어있는 노드에 다시 연결한다. 이 `flocker` 볼륨을 사용하기 위해서는 먼저 Flocker를 설치하고 실행한다. {{< /note >}} -더 자세한 내용은 [Flocker 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/flocker)를 참조한다. +더 자세한 내용은 [Flocker 예시](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker)를 참조한다. ### gcePersistentDisk @@ -525,7 +525,7 @@ glusterfs 볼륨에 데이터를 미리 채울 수 있으며, 파드 간에 데 사용하려면 먼저 GlusterFS를 설치하고 실행해야 한다. {{< /note >}} -더 자세한 내용은 [GlusterFS 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/glusterfs)를 본다. +더 자세한 내용은 [GlusterFS 예시](https://github.com/kubernetes/examples/tree/master/volumes/glusterfs)를 본다. ### hostPath {#hostpath} @@ -653,7 +653,7 @@ iSCSI 특징은 여러 고객이 읽기 전용으로 마운트할 수 iSCSI 볼륨은 읽기-쓰기 모드에서는 단일 고객만 마운트할 수 있다. 동시 쓰기는 허용되지 않는다. -더 자세한 내용은 [iSCSI 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/iscsi)를 본다. +더 자세한 내용은 [iSCSI 예시](https://github.com/kubernetes/examples/tree/master/volumes/iscsi)를 본다. ### local @@ -741,7 +741,7 @@ local [스토리지클래스(StorageClas)](/ko/docs/concepts/storage/storage-cla 사용하려면 먼저 NFS 서버를 실행하고 공유를 내보내야 한다. {{< /note >}} -더 자세한 내용은 [NFS 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/nfs)를 본다. +더 자세한 내용은 [NFS 예시](https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs)를 본다. ### persistentVolumeClaim {#persistentvolumeclaim} @@ -789,7 +789,7 @@ spec: 있는지 확인한다. {{< /note >}} -자세한 내용은 [Portworx 볼륨](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/portworx/README.md) 예제를 참고한다. +자세한 내용은 [Portworx 볼륨](https://github.com/kubernetes/examples/tree/master/staging/volumes/portworx/README.md) 예제를 참고한다. ### projected @@ -803,7 +803,7 @@ spec: * `serviceAccountToken` 모든 소스는 파드와 동일한 네임스페이스에 있어야 한다. 더 자세한 내용은 -[올인원 볼륨 디자인 문서](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/node/all-in-one-volume.md)를 본다. +[올인원 볼륨 디자인 문서](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/all-in-one-volume.md)를 본다. #### 시크릿, 다운워드 API 그리고 컨피그맵이 있는 구성 예시 {#example-configuration-secret-downwardapi-configmap} @@ -964,7 +964,7 @@ RBD의 특징은 여러 고객이 동시에 읽기 전용으로 마운트할 수 RBD는 읽기-쓰기 모드에서 단일 고객만 마운트할 수 있다. 동시 쓰기는 허용되지 않는다. -더 자세한 내용은 [RBD 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/rbd)를 +더 자세한 내용은 [RBD 예시](https://github.com/kubernetes/examples/tree/master/volumes/rbd)를 참고한다. ### scaleIO (사용 중단됨) {#scaleio} @@ -1008,7 +1008,7 @@ spec: fsType: xfs ``` -더 자세한 내용은 [ScaleIO](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/scaleio) 예제를 참고한다. +더 자세한 내용은 [ScaleIO](https://github.com/kubernetes/examples/tree/master/staging/volumes/scaleio) 예제를 참고한다. ### secret diff --git a/content/ko/docs/concepts/workloads/controllers/job.md b/content/ko/docs/concepts/workloads/controllers/job.md index c24beb0fca..4972df45cd 100644 --- a/content/ko/docs/concepts/workloads/controllers/job.md +++ b/content/ko/docs/concepts/workloads/controllers/job.md @@ -594,7 +594,7 @@ spec: 시작하기에는 다소 복잡할 수 있으며 쿠버네티스와의 통합성이 낮아진다. 이 패턴의 한 예시는 파드를 시작하는 잡이다. 파드는 스크립트를 실행해서 -스파크(Spark) 마스터 컨트롤러 ([스파크 예시](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/spark/README.md)를 본다)를 시작하고, +스파크(Spark) 마스터 컨트롤러 ([스파크 예시](https://github.com/kubernetes/examples/tree/master/staging/spark/README.md)를 본다)를 시작하고, 스파크 드라이버를 실행한 다음, 정리한다. 이 접근 방식의 장점은 전체 프로세스가 잡 오브젝트의 완료를 보장하면서도, diff --git a/content/ko/docs/concepts/workloads/controllers/statefulset.md b/content/ko/docs/concepts/workloads/controllers/statefulset.md index 3a1f784259..dbb30150e3 100644 --- a/content/ko/docs/concepts/workloads/controllers/statefulset.md +++ b/content/ko/docs/concepts/workloads/controllers/statefulset.md @@ -32,7 +32,7 @@ weight: 30 ## 제한사항 -* 파드에 지정된 스토리지는 관리자에 의해 [퍼시스턴트 볼륨 프로비저너](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/persistent-volume-provisioning/README.md)를 기반으로 하는 `storage class` 를 요청해서 프로비전하거나 사전에 프로비전이 되어야 한다. +* 파드에 지정된 스토리지는 관리자에 의해 [퍼시스턴트 볼륨 프로비저너](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/README.md)를 기반으로 하는 `storage class` 를 요청해서 프로비전하거나 사전에 프로비전이 되어야 한다. * 스테이트풀셋을 삭제 또는 스케일 다운해도 스테이트풀셋과 연관된 볼륨이 *삭제되지 않는다*. 이는 일반적으로 스테이트풀셋과 연관된 모든 리소스를 자동으로 제거하는 것보다 더 중요한 데이터의 안전을 보장하기 위함이다. * 스테이트풀셋은 현재 파드의 네트워크 신원을 책임지고 있는 [헤드리스 서비스](/ko/docs/concepts/services-networking/service/#헤드리스-headless-서비스)가 필요하다. 사용자가 이 서비스를 생성할 책임이 있다. * 스테이트풀셋은 스테이트풀셋의 삭제 시 파드의 종료에 대해 어떠한 보증을 제공하지 않는다. 스테이트풀셋에서는 파드가 순차적이고 정상적으로 종료(graceful termination)되도록 하려면, 삭제 전 스테이트풀셋의 스케일을 0으로 축소할 수 있다. diff --git a/content/ko/docs/tasks/administer-cluster/access-cluster-api.md b/content/ko/docs/tasks/administer-cluster/access-cluster-api.md index 2d44a51b0e..41b2225549 100644 --- a/content/ko/docs/tasks/administer-cluster/access-cluster-api.md +++ b/content/ko/docs/tasks/administer-cluster/access-cluster-api.md @@ -30,7 +30,7 @@ content_type: task kubectl config view ``` -많은 [예제](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/)는 kubectl 사용에 대한 소개를 +많은 [예제](https://github.com/kubernetes/examples/tree/master/)는 kubectl 사용에 대한 소개를 제공한다. 전체 문서는 [kubectl 매뉴얼](/ko/docs/reference/kubectl/overview/)에 있다. ### REST API에 직접 접근 diff --git a/content/ko/docs/tasks/administer-cluster/dns-custom-nameservers.md b/content/ko/docs/tasks/administer-cluster/dns-custom-nameservers.md index f681bc8778..8d67bea2db 100644 --- a/content/ko/docs/tasks/administer-cluster/dns-custom-nameservers.md +++ b/content/ko/docs/tasks/administer-cluster/dns-custom-nameservers.md @@ -23,7 +23,7 @@ DNS 변환(DNS resolution) 절차를 사용자 정의하는 방법을 설명한 ## 소개 -DNS는 _애드온 관리자_ 인 [클러스터 애드온](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/README.md)을 +DNS는 _애드온 관리자_ 인 [클러스터 애드온](https://releases.k8s.io/master/cluster/addons/README.md)을 사용하여 자동으로 시작되는 쿠버네티스 내장 서비스이다. diff --git a/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md b/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md index ee7cccb70d..b879049c55 100644 --- a/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/content/ko/docs/tutorials/stateful-application/basic-stateful-set.md @@ -20,7 +20,7 @@ weight: 10 * [클러스터 DNS(Cluster DNS)](/ko/docs/concepts/services-networking/dns-pod-service/) * [헤드리스 서비스(Headless Services)](/ko/docs/concepts/services-networking/service/#헤드리스-headless-서비스) * [퍼시스턴트볼륨(PersistentVolumes)](/ko/docs/concepts/storage/persistent-volumes/) -* [퍼시턴트볼륨 프로비저닝](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/persistent-volume-provisioning/) +* [퍼시턴트볼륨 프로비저닝](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/) * [스테이트풀셋](/ko/docs/concepts/workloads/controllers/statefulset/) * [kubectl](/ko/docs/reference/kubectl/kubectl/) 커맨드 라인 도구 diff --git a/content/ko/docs/tutorials/stateful-application/zookeeper.md b/content/ko/docs/tutorials/stateful-application/zookeeper.md index 3fca0a6749..33627ac2c4 100644 --- a/content/ko/docs/tutorials/stateful-application/zookeeper.md +++ b/content/ko/docs/tutorials/stateful-application/zookeeper.md @@ -19,7 +19,7 @@ weight: 40 - [클러스터 DNS](/ko/docs/concepts/services-networking/dns-pod-service/) - [헤드리스 서비스](/ko/docs/concepts/services-networking/service/#헤드리스-headless-서비스) - [퍼시스턴트볼륨](/ko/docs/concepts/storage/persistent-volumes/) -- [퍼시스턴트볼륨 프로비저닝](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/persistent-volume-provisioning/) +- [퍼시스턴트볼륨 프로비저닝](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/) - [스테이트풀셋](/ko/docs/concepts/workloads/controllers/statefulset/) - [PodDisruptionBudget](/ko/docs/concepts/workloads/pods/disruptions/#파드-disruption-budgets) - [파드안티어피니티](/ko/docs/concepts/scheduling-eviction/assign-pod-node/#어피니티-affinity-와-안티-어피니티-anti-affinity) From a6ed49bca93a3a8cfc5b9b360706fe9c917cf96b Mon Sep 17 00:00:00 2001 From: S Nitesh Singh Date: Thu, 23 Sep 2021 10:59:48 +0530 Subject: [PATCH 053/249] remove 'Tools Included' from sidebar of translated page --- layouts/partials/sidebar-tree.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/partials/sidebar-tree.html b/layouts/partials/sidebar-tree.html index f1a3c9926a..3b246d86f1 100644 --- a/layouts/partials/sidebar-tree.html +++ b/layouts/partials/sidebar-tree.html @@ -69,7 +69,7 @@ {{ range . }} {{ with (where $.section.Translations ".Lang" . ) -}} {{ $p := index . 0 -}} - {{ $pages = $pages | lang.Merge (union $p.Pages $p.Sections) -}} + {{ $pages = where ( $pages | lang.Merge (union $p.Pages $p.Sections)) ".Params.toc_hide" "!=" true -}} {{ end -}} {{ end -}} {{ end -}} From 02809ef886d87f0ead78e03c518471345fb100f2 Mon Sep 17 00:00:00 2001 From: fabriziopandini Date: Thu, 23 Sep 2021 16:01:08 +0200 Subject: [PATCH 054/249] address comments --- ...2021-10-04-clusterclass-and-managed-topologies.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md b/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md index b576b345e9..ec49d73e7d 100644 --- a/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md +++ b/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md @@ -17,17 +17,17 @@ The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) start With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more. -Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs and documentation, and on extensive test signals which inform Kubernetes releases. +Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs, documentation, and on extensive test signals which inform Kubernetes releases. With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users. -Enter ClusterClass and manage topologies, tada! +Enter ClusterClass and Managed Topologies, tada! ## ClusterClass As the name suggests, ClusterClass and managed topologies are built in two parts. -The idea behind ClusterClass is simple: let’s define the shape of your cluster once, and then reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. +The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. ![Defining a ClusterClass](/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg) @@ -84,11 +84,11 @@ apiVersion: cluster.x-k8s.io/v1beta1 name: vsphere-cluster ``` -The possibilities are endless; you can get a default ClusterClass from the community, “off-the shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios. +The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios. ## Managed Topologies -Managed topologies let you put the power that ClusterClass provides into action. +Managed Topologies let you put the power of ClusterClass into action. Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster. @@ -136,7 +136,7 @@ There are a lot of great ideas and opportunities ahead! We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on. -Stay tuned for what comes next, and if you have any question, comment or suggestion: +Stay tuned for what comes next, and if you have any questions, comments or suggestions: * Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T) * Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents From 09206415705983b30b19ee8795a816c30ecadb0b Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Thu, 23 Sep 2021 11:53:58 -0300 Subject: [PATCH 055/249] Update to the introduction --- content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md index 6d27839d18..88c9fc890b 100644 --- a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md +++ b/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md @@ -9,8 +9,7 @@ slug: sig-node-spotlight-2021 ## Introduction -[SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) is responsible for the components that support the controlled interactions between pods and host resources. The work of this group manages the lifecycle of pods that are scheduled to a node, and focuses on enabling a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved. - +In Kubernetes, a _Node_ is a representation of a single machine in your cluster. [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved. ## A summary of our conversation From 577e0c7ca53e282b4b3e655fddd5f379296e091f Mon Sep 17 00:00:00 2001 From: Dewan Ahmed Date: Thu, 23 Sep 2021 11:55:16 -0300 Subject: [PATCH 056/249] Match folder name with publication date --- .../index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename content/en/blog/_posts/{2021-08-13-SIG-Node-Spotlight => 2021-09-27-SIG-Node-Spotlight}/index.md (100%) diff --git a/content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md b/content/en/blog/_posts/2021-09-27-SIG-Node-Spotlight/index.md similarity index 100% rename from content/en/blog/_posts/2021-08-13-SIG-Node-Spotlight/index.md rename to content/en/blog/_posts/2021-09-27-SIG-Node-Spotlight/index.md From 4ba25b088566c2f219f1211424ef015c3745c125 Mon Sep 17 00:00:00 2001 From: Arhell Date: Fri, 24 Sep 2021 00:23:51 +0300 Subject: [PATCH 057/249] [id] Add seccomp tutorial to index --- content/id/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/id/docs/tutorials/_index.md b/content/id/docs/tutorials/_index.md index 9f3ec5afb3..041b7b62d2 100644 --- a/content/id/docs/tutorials/_index.md +++ b/content/id/docs/tutorials/_index.md @@ -50,6 +50,8 @@ Sebelum melangkah lebih lanjut ke tutorial, sebaiknya tandai dulu halaman [Kamus * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Servis * [Menggunakan Source IP](/docs/tutorials/services/source-ip/) From d8245b95be0a4b8c560977cc4f12e4aea6f14338 Mon Sep 17 00:00:00 2001 From: Mengjiao Liu Date: Fri, 24 Sep 2021 11:18:40 +0800 Subject: [PATCH 058/249] [zh] Update kep link and release version for dockershim removal in faq --- content/zh/blog/_posts/2020-12-02-dockershim-faq.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/content/zh/blog/_posts/2020-12-02-dockershim-faq.md b/content/zh/blog/_posts/2020-12-02-dockershim-faq.md index 65292d90ac..d873fab049 100644 --- a/content/zh/blog/_posts/2020-12-02-dockershim-faq.md +++ b/content/zh/blog/_posts/2020-12-02-dockershim-faq.md @@ -45,7 +45,7 @@ You can read more about the community discussion and planning in the --> Dockershim 向来都是一个临时解决方案（因此得名：shim）。 你可以进一步阅读 -[移除 Kubernetes 增强方案 Dockershim](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim) +[移除 Kubernetes 增强方案 Dockershim][drkep] 以了解相关的社区讨论和计划。 @@ -78,12 +79,16 @@ startup if using Docker as the runtime. 考虑到此改变带来的影响，我们使用了一个加长的废弃时间表。 在 Kubernetes 1.22 版之前，它不会被彻底移除；换句话说，dockershim 被移除的最早版本会是 2021 年底发布的 1.23 版。 +_更新_：dockershim 计划在 Kubernetes 1.24 版被移除， +请参阅[移除 Kubernetes 增强方案 Dockershim][drkep]。 我们将与供应商以及其他生态团队紧密合作，确保顺利过渡，并将依据事态的发展评估后续事项。 * 提供程序对双协议栈网络的支持 (云供应商或其他方式必须能够为 Kubernetes 节点 提供可路由的 IPv4/IPv6 网络接口) -* 一个能够支持双协议栈的 +* 一个能够支持[双协议栈](/zh/docs/concepts/services-networking/dual-stack/)的 [网络插件](/zh/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)， - （如 kubenet 或 Calico）。 + （如 Calico，Cilium 或 Kubenet）。 * [启用双协议栈](/zh/docs/concepts/services-networking/dual-stack/) 集群 From 40d2fdbe65055ca2b78fe805849306b83d97ea04 Mon Sep 17 00:00:00 2001 From: superleo Date: Thu, 23 Sep 2021 23:37:00 +0800 Subject: [PATCH 060/249] Update content/zh/docs/tasks/configure-pod-container/configure-gmsa.md --- .../configure-pod-container/configure-gmsa.md | 146 +++++++++++++----- 1 file changed, 107 insertions(+), 39 deletions(-) diff --git a/content/zh/docs/tasks/configure-pod-container/configure-gmsa.md b/content/zh/docs/tasks/configure-pod-container/configure-gmsa.md index 3687f9b918..8e183745c6 100644 --- a/content/zh/docs/tasks/configure-pod-container/configure-gmsa.md +++ b/content/zh/docs/tasks/configure-pod-container/configure-gmsa.md @@ -381,6 +381,85 @@ As Pod specs with GMSA fields populated (as described above) are applied in a cl 1. 容器运行时为每个 Windows 容器配置所指定的 GMSA 凭据规约，这样容器就可以以 活动目录中该 GMSA 所代表的身份来执行操作，使用该身份来访问域中的服务。 + +## Containerd +在 Windows Server 2019 上对 containerd 使用 GMSA，需要使用 Build 17763.1817（或更新的版本）， +你可以安装补丁 [KB5000822](https://support.microsoft.com/en-us/topic/march-9-2021-kb5000822-os-build-17763-1817-2eb6197f-e3b1-4f42-ab51-84345e063564)。 + +containerd 场景从 Pod 连接 SMB 共享的时候有一个已知问题： +配置了 GMSA 以后，无法通过主机名或者 FQDN 访问 SMB共享，但是通过 IP 地址访问没有问题。 + +```PowerShell +ping adserver.ad.local +``` + + + +主机名可以被解析为 IPv4 地址，输出类似如下所示： + +``` +Pinging adserver.ad.local [192.168.111.18] with 32 bytes of data: +Reply from 192.168.111.18: bytes=32 time=6ms TTL=124 +Reply from 192.168.111.18: bytes=32 time=5ms TTL=124 +Reply from 192.168.111.18: bytes=32 time=5ms TTL=124 +Reply from 192.168.111.18: bytes=32 time=5ms TTL=124 +``` + + +但是，当尝试使用主机名浏览目录时: + +```PowerShell +cd \\adserver.ad.local\test +``` + + +你会看到一个错误，提示目标共享不存在: + +``` +cd : Cannot find path '\\adserver.ad.local\test' because it does not exist. +At line:1 char:1 ++ cd \\adserver.ad.local\test ++ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + CategoryInfo : ObjectNotFound: (\\adserver.ad.local\test:String) [Set-Location], ItemNotFoundException + + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand +``` + + +但是你会注意到，如果你改为使用其 IPv4 地址浏览共享，错误就会消失；例如： + +```PowerShell +cd \\192.168.111.18\test +``` + + +切换到共享中的目录后，你会看到类似于以下内容的提示： + +``` +Microsoft.PowerShell.Core\FileSystem::\\192.168.111.18\test> +``` + + +要解决问题，你需要在节点上运行以下命令以添加所需的注册表项 +`reg add "HKLM\SYSTEM\CurrentControlSet\Services\hns\State" /v EnableCompartmentNamespace /t REG_DWORD /d 1`。 +此更改只会在新创建的 Pod 中生效，这意味着你必须重新创建任何需要访问 SMB 共享的正在运行的 Pod。 ## 故障排查 -如果在你的环境中配置 GMSA 时遇到了困难，你可以采取若干步骤来排查可能 -的故障。 +如果在你的环境中配置 GMSA 时遇到了困难，你可以采取若干步骤来排查可能的故障。 -首先，确保凭据规约已经被传递到 Pod。要实现这点，你需要先通过 `exec` 进入到 -你的 Pod 之一，检查 `nltest.exe /parentdomain` 命令的输出。 +首先，确保 credspec 已传递给 Pod。为此，你需要先运行 `exec` 进入到你的一个 Pod 中并检查 `nltest.exe /parentdomain` 命令的输出。 在下面的例子中，Pod 未能正确地获得凭据规约： -```shell +```PowerShell kubectl exec -it iis-auth-7776966999-n5nzr powershell.exe +``` -Windows PowerShell -Copyright (C) Microsoft Corporation. All rights reserved. - -PS C:\> nltest.exe /parentdomain + +`nltest.exe /parentdomain` 导致以下错误： +``` Getting parent domain failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE -PS C:\> ``` -这一输出告诉我们，由于某些原因，Pod 无法使用凭据规约中的账号登录到域。 -你可以通过运行 `nltest.exe /sc_reset:domain.example` 命令尝试修复安全通道。 +这告诉我们，由于某种原因，Pod 无法使用 credspec 中指定的帐户登录到域。 +你可以尝试通过运行以下命令来修复安全通道： -```shell -PS C:\> nltest /sc_reset:domain.example +```PowerShell +nltest /sc_reset:domain.example +``` + + +如果命令成功，你将看到类似以下内容的输出： + +``` Flags: 30 HAS_IP HAS_TIMESERV Trusted DC Name \\dc10.domain.example Trusted DC Connection Status Status = 0 0x0 NERR_Success The command completed successfully -PS C:\> ``` -如果上述命令修复了错误，你就可以通过向你的 Pod 规约添加生命周期回调来将此操作 -自动化。如果上述命令未能奏效，你就需要再次检查凭据规约，以确保其数据时正确的 -而且是完整的。 +如果以上命令修复了错误，你可以通过将以下生命周期回调添加到你的 Pod 规约中来自动执行该步骤。 +如果这些操作没有修复错误，你将需要再次检查你的 credspec 并确认它是正确和完整的。 ```yaml image: registry.domain.example/iis-auth:1809v1 @@ -477,18 +560,3 @@ If you add the `lifecycle` section show above to your Pod spec, the Pod will exe 列举的命令来重启 `netlogon` 服务，直到 `nltest.exe /query` 命令返回时没有错误信息。 - -## GMSA 的局限 {#gmsa-limitations} - -在使用 [Windows 版本的 ContainerD 运行时](/zh/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#cri-containerd) -时，通过 GMSA 域身份标识访问受限制的网络共享资源时会出错。 -容器会收到身份标识且 `nltest.exe /query` 调用也能正常工作。 -当需要访问网络共享资源时，建议使用 -[Docker EE 运行时](/zh/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#docker-ee)。 -Windows Server 团队正在 Windows 内核中解决这一问题，并在将来发布解决此问题的补丁。 -你可以在 [Microsoft Windows Containers 问题跟踪列表](https://github.com/microsoft/Windows-Containers/issues/44) -中查找这类更新。 - From 548dee7af3b50994b6903546009d40a074829b1b Mon Sep 17 00:00:00 2001 From: Jumping Qu Date: Fri, 24 Sep 2021 16:28:56 +0800 Subject: [PATCH 061/249] Update debug-cluster.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit remove the ":" after "具体情况: " and "缓解措施：" --- .../zh/docs/tasks/debug-application-cluster/debug-cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/tasks/debug-application-cluster/debug-cluster.md b/content/zh/docs/tasks/debug-application-cluster/debug-cluster.md index f97364dd4c..7f7ffff4e8 100644 --- a/content/zh/docs/tasks/debug-application-cluster/debug-cluster.md +++ b/content/zh/docs/tasks/debug-application-cluster/debug-cluster.md @@ -131,7 +131,7 @@ This is an incomplete list of things that could go wrong, and how to adjust your - kubelets will not be able to reach it but will continue to run the same pods and provide the same service proxying - manual recovery or recreation of apiserver state necessary before apiserver is restarted --> -### 具体情况: +### 具体情况 - API 服务器所在的 VM 关机或者 API 服务器崩溃 - 结果 @@ -208,7 +208,7 @@ This is an incomplete list of things that could go wrong, and how to adjust your - Mitigates: API server backing storage (i.e., etcd's data directory) lost - Assumes HA (highly-available) etcd configuration --> -### 缓解措施： +### 缓解措施 - 措施：对于 IaaS 上的 VMs，使用 IaaS 的自动 VM 重启功能 - 缓解：API 服务器 VM 关机或 API 服务器崩溃 From 6ce11bb1f869c63fae768019331baa05a8c75fa0 Mon Sep 17 00:00:00 2001 From: zhiguo-lu Date: Tue, 21 Sep 2021 21:45:01 +0800 Subject: [PATCH 062/249] [zh] translate concepts/Traces For Kubernetes System Components --- .../cluster-administration/system-traces.md | 156 ++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 content/zh/docs/concepts/cluster-administration/system-traces.md diff --git a/content/zh/docs/concepts/cluster-administration/system-traces.md b/content/zh/docs/concepts/cluster-administration/system-traces.md new file mode 100644 index 0000000000..c829341afd --- /dev/null +++ b/content/zh/docs/concepts/cluster-administration/system-traces.md @@ -0,0 +1,156 @@ +--- +title: 追踪 Kubernetes 系统组件 +content_type: concept +weight: 60 +--- + + + + +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + + +系统组件追踪功能记录各个集群操作的时延信息和这些操作之间的关系。 + + +Kubernetes 组件基于 gRPC 导出器的 +[OpenTelemetry 协议](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#opentelemetry-protocol-specification) +发送追踪信息，并用 +[OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector#-opentelemetry-collector) +收集追踪信息，再将其转交给追踪系统的后台。 + + + + +## 追踪信息的收集 {#trace-collection} + + +关于收集追踪信息、以及使用收集器的完整指南，可参见 +[Getting Started with the OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/)。 +不过，还有一些特定于 Kubernetes 组件的事项值得注意。 + + +默认情况下，Kubernetes 组件使用 gRPC 的 OTLP 导出器来导出追踪信息，将信息写到 +[IANA OpenTelemetry 端口](https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=opentelemetry)。 +举例来说，如果收集器以 Kubernetes 组件的边车模式运行，以下接收器配置会收集 spans 信息，并将它们写入到标准输出。 + + +```yaml +receivers: + otlp: + protocols: + grpc: +exporters: + # 用适合你后端环境的导出器替换此处的导出器 + logging: + logLevel: debug +service: + pipelines: + traces: + receivers: [otlp] + exporters: [logging] +``` + + +## 组件追踪 {#component-traces} + +### kube-apiserver 追踪 {#kube-apiserver-traces} + + +kube-apiserver 为传入的 HTTP 请求、传出到 webhook 和 etcd 的请求以及重入的请求生成 spans。 +由于 kube-apiserver 通常是一个公开的端点，所以它通过出站的请求传播 +[W3C 追踪上下文](https://www.w3.org/TR/trace-context/)， +但不使用入站请求的追踪上下文。 + + +#### 在 kube-apiserver 中启用追踪 {#enabling-tracing-in-the-kube-apiserver} + + +要启用追踪特性，需要启用 kube-apiserver 上的 `APIServerTracing` +[特性门控](/zh/docs/reference/command-line-tools-reference/feature-gates/)。 +然后，使用 `--tracing-config-file=<<配置文件路径>` 为 kube-apiserver 提供追踪配置文件。 +下面是一个示例配置，它为万分之一的请求记录 spans，并使用了默认的 OpenTelemetry 端口。 + +```yaml +apiVersion: apiserver.config.k8s.io/v1alpha1 +kind: TracingConfiguration +# default value +#endpoint: localhost:4317 +samplingRatePerMillion: 100 +``` + + +有关 TracingConfiguration 结构体的更多信息，请参阅 +[API 服务器配置 API (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/#apiserver-k8s-io-v1alpha1-TracingConfiguration)。 + + +## 稳定性 {#stability} + + +追踪工具仍在积极开发中，未来它会以多种方式发生变化。 +这些变化包括：span 名称、附加属性、检测端点等等。 +此类特性在达到稳定版本之前，不能保证追踪工具的向后兼容性。 + +## {{% heading "whatsnext" %}} + + +* 阅读[Getting Started with the OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/) \ No newline at end of file From e2448063df1b653aa9f4a061e5a985c2b0e6a24b Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Fri, 24 Sep 2021 06:42:16 -0700 Subject: [PATCH 063/249] use k8s-staging-test-infra/gcb-docker-gcloud --- cloudbuild.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cloudbuild.yaml b/cloudbuild.yaml index 61b5adc5f4..5039818482 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -8,7 +8,7 @@ options: substitution_option: ALLOW_LOOSE steps: # It's fine to bump the tag to a recent version, as needed - - name: "gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4" + - name: "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55" entrypoint: make env: - DOCKER_CLI_EXPERIMENTAL=enabled From eb65e4d9f56ed834909bf984c28a158dd789ae03 Mon Sep 17 00:00:00 2001 From: Carlos Panato Date: Sat, 18 Sep 2021 15:31:10 +0200 Subject: [PATCH 064/249] patches: update releases for October cycle Signed-off-by: Carlos Panato --- content/en/releases/patch-releases.md | 5 ++++- data/releases/schedule.yaml | 27 ++++++++++++++++++--------- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/content/en/releases/patch-releases.md b/content/en/releases/patch-releases.md index c7597e332f..91bd47bcd0 100644 --- a/content/en/releases/patch-releases.md +++ b/content/en/releases/patch-releases.md @@ -78,10 +78,10 @@ releases may also occur in between these. | Monthly Patch Release | Cherry Pick Deadline | Target date | | --------------------- | -------------------- | ----------- | -| September 2021 | 2021-09-10 | 2021-09-15 | | October 2021 | 2021-10-15 | 2021-10-20 | | November 2021 | 2021-11-12 | 2021-11-17 | | December 2021 | 2021-12-10 | 2021-12-15 | +| January 2022 | 2021-01-14 | 2021-01-19 | ## Detailed Release History for Active Branches @@ -93,6 +93,7 @@ End of Life for **1.22** is **2022-10-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | |---------------|----------------------|-------------|------| +| 1.22.3 | 2021-10-15 | 2021-10-20 | | | 1.22.2 | 2021-09-10 | 2021-09-15 | | | 1.22.1 | 2021-08-16 | 2021-08-19 | | @@ -104,6 +105,7 @@ End of Life for **1.21** is **2022-06-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | | ------------- | -------------------- | ----------- | ---------------------------------------------------------------------- | +| 1.21.6 | 2021-10-15 | 2021-10-20 | | | 1.21.5 | 2021-09-10 | 2021-09-15 | | | 1.21.4 | 2021-08-07 | 2021-08-11 | | | 1.21.3 | 2021-07-10 | 2021-07-14 | | @@ -118,6 +120,7 @@ End of Life for **1.20** is **2022-02-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | | ------------- | -------------------- | ----------- | ----------------------------------------------------------------------------------- | +| 1.20.12 | 2021-10-15 | 2021-10-20 | | | 1.20.11 | 2021-09-10 | 2021-09-15 | | | 1.20.10 | 2021-08-07 | 2021-08-11 | | | 1.20.9 | 2021-07-10 | 2021-07-14 | | diff --git a/data/releases/schedule.yaml b/data/releases/schedule.yaml index 99abec370e..55d5b12633 100644 --- a/data/releases/schedule.yaml +++ b/data/releases/schedule.yaml @@ -1,19 +1,25 @@ schedules: - release: 1.22 - next: 1.22.2 - cherryPickDeadline: 2021-09-10 - targetDate: 2021-09-15 + next: 1.22.3 + cherryPickDeadline: 2021-10-15 + targetDate: 2021-10-20 endOfLifeDate: 2022-10-28 previousPatches: + - release: 1.22.2 + cherryPickDeadline: 2021-09-10 + targetDate: 2021-09-15 - release: 1.22.1 cherryPickDeadline: 2021-08-16 targetDate: 2021-08-19 - release: 1.21 - next: 1.21.5 - cherryPickDeadline: 2021-09-10 - targetDate: 2021-09-15 + next: 1.21.6 + cherryPickDeadline: 2021-10-15 + targetDate: 2021-10-20 endOfLifeDate: 2022-06-28 previousPatches: + - release: 1.21.5 + cherryPickDeadline: 2021-09-10 + targetDate: 2021-09-15 - release: 1.21.4 cherryPickDeadline: 2021-08-07 targetDate: 2021-08-11 @@ -28,11 +34,14 @@ schedules: targetDate: 2021-05-12 note: Regression https://groups.google.com/g/kubernetes-dev/c/KuF8s2zueFs - release: 1.20 - next: 1.20.11 - cherryPickDeadline: 2021-09-10 - targetDate: 2021-09-15 + next: 1.20.12 + cherryPickDeadline: 2021-10-15 + targetDate: 2021-10-20 endOfLifeDate: 2022-02-28 previousPatches: + - release: 1.20.11 + cherryPickDeadline: 2021-09-10 + targetDate: 2021-09-15 - release: 1.20.10 cherryPickDeadline: 2021-08-07 targetDate: 2021-08-11 From 391148fcd5180d624588a858716ec36b54300ab5 Mon Sep 17 00:00:00 2001 From: Pushkar Joglekar <3390906+PushkarJ@users.noreply.github.com> Date: Fri, 24 Sep 2021 15:02:56 -0700 Subject: [PATCH 065/249] Update content/en/releases/release-managers.md Co-authored-by: Bob Killen --- content/en/releases/release-managers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/releases/release-managers.md b/content/en/releases/release-managers.md index e8b75492bc..a25619c8c9 100644 --- a/content/en/releases/release-managers.md +++ b/content/en/releases/release-managers.md @@ -31,7 +31,7 @@ The responsibilities of each role are described below. ### Security Embargo Policy -Some information about releases is subject to embargo and we have defined policy about how those embargoes are set. Please refer to the [Security Embargo Policy](https://github.com/kubernetes/committee-security-response/blob/master/private-distributors-list.md#embargo-policy) for more information. +Some information about releases is subject to embargo and we have defined policy about how those embargoes are set. Please refer to the [Security Embargo Policy](https://github.com/kubernetes/committee-security-response/blob/main/private-distributors-list.md#embargo-policy) for more information. ## Handbooks From 780dae2785d7553cf43a3d95c41f35dd1f8a6491 Mon Sep 17 00:00:00 2001 From: Richard Tweed Date: Sat, 25 Sep 2021 00:02:21 +0100 Subject: [PATCH 066/249] Clarified scenarios that could lead to privilege escalation (#29378) * Clarified scenarios that could lead to privilege escalation Made it clearer that it's not just creating pods which enables the privilege escalation. It's all workloads, all reconfiguration of workloads, and conceptually the creation and reconfiguration of custom resources which create workloads. * Allowing link to priv escalation heading if required * Update content/en/docs/reference/access-authn-authz/authorization.md Co-authored-by: Tim Bannister * Adding further clarifications * Retitled escalation section * Apply suggestions from vjftw Co-authored-by: VJ Patel * Clarified CRDs and reduced duplication * Updating caution based on Geoffrey's comments * Updating controller comment and linking out to reference docs Co-authored-by: Tim Bannister Co-authored-by: VJ Patel --- .../access-authn-authz/authorization.md | 33 ++++++++++++------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/content/en/docs/reference/access-authn-authz/authorization.md b/content/en/docs/reference/access-authn-authz/authorization.md index af73a23350..0292ad8130 100644 --- a/content/en/docs/reference/access-authn-authz/authorization.md +++ b/content/en/docs/reference/access-authn-authz/authorization.md @@ -192,22 +192,33 @@ The following flags can be used: You can choose more than one authorization module. Modules are checked in order so an earlier module has higher priority to allow or deny a request. -## Privilege escalation via pod creation +## Privilege escalation via workload creation or edits {#privilege-escalation-via-pod-creation} -Users who have the ability to create pods in a namespace can potentially -escalate their privileges within that namespace. They can create pods that -access their privileges within that namespace. They can create pods that access -secrets the user cannot themselves read, or that run under a service account -with different/greater permissions. +Users who can create/edit pods in a namespace, either directly or through a [controller](/docs/concepts/architecture/controller/) +such as an operator, could escalate their privileges in that namespace. {{< caution >}} -System administrators, use care when granting access to pod creation. A user -granted permission to create pods (or controllers that create pods) in the -namespace can: read all secrets in the namespace; read all config maps in the -namespace; and impersonate any service account in the namespace and take any -action the account could take. This applies regardless of authorization mode. +System administrators, use care when granting access to create or edit workloads. +Details of how these can be misused are documented in [escalation paths](/docs/reference/access-authn-authz/authorization/#escalation-paths) {{< /caution >}} +### Escalation paths {#escalation-paths} +- Mounting arbitrary secrets in that namespace + - Can be used to access secrets meant for other workloads + - Can be used to obtain a more privileged service account's service account token +- Using arbitrary Service Accounts in that namespace + - Can perform Kubernetes API actions as another workload (impersonation) + - Can perform any privileged actions that Service Account has +- Mounting configmaps meant for other workloads in that namespace + - Can be used to obtain information meant for other workloads, such as DB host names. +- Mounting volumes meant for other workloads in that namespace + - Can be used to obtain information meant for other workloads, and change it. + +{{< caution >}} +System administrators should be cautious when deploying CRDs that +change the above areas. These may open privilege escalations paths. +This should be considered when deciding on your RBAC controls. +{{< /caution >}} ## {{% heading "whatsnext" %}} From 9bcd6a35793fca04e10e200d0b46370b38aa4eea Mon Sep 17 00:00:00 2001 From: Arhell Date: Sat, 25 Sep 2021 04:41:55 +0300 Subject: [PATCH 067/249] [it] Add seccomp tutorial to index --- content/it/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/it/docs/tutorials/_index.md b/content/it/docs/tutorials/_index.md index 04069ff5eb..240fce078c 100644 --- a/content/it/docs/tutorials/_index.md +++ b/content/it/docs/tutorials/_index.md @@ -50,6 +50,8 @@ Prima di procedere con vari tutorial, raccomandiamo di aggiungere il * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Servizi * [Utilizzare Source IP](/docs/tutorials/services/source-ip/) From 461a0b4a6d3d4ec9ddcab8039928934c067e2d56 Mon Sep 17 00:00:00 2001 From: Anubhav Vardhan Date: Sat, 25 Sep 2021 15:29:46 +0530 Subject: [PATCH 068/249] Update OWNERS_ALIASES --- OWNERS_ALIASES | 1 + 1 file changed, 1 insertion(+) diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES index ce45f0e787..7764d5c04e 100644 --- a/OWNERS_ALIASES +++ b/OWNERS_ALIASES @@ -79,6 +79,7 @@ aliases: - anubha-v-ardhan - divya-mohan0209 - mittalyashu + - verma-kunal sig-docs-id-owners: # Admins for Indonesian content - ariscahyadi - danninov From 24f5f7251bc69bdc34407fa00520f0e73d4314c6 Mon Sep 17 00:00:00 2001 From: Bigsmooth68 <57505439+Bigsmooth68@users.noreply.github.com> Date: Sat, 25 Sep 2021 16:19:54 +0200 Subject: [PATCH 069/249] Update nodes.md and => et --- content/fr/docs/concepts/architecture/nodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/fr/docs/concepts/architecture/nodes.md b/content/fr/docs/concepts/architecture/nodes.md index b6718c6750..f493fe5b17 100644 --- a/content/fr/docs/concepts/architecture/nodes.md +++ b/content/fr/docs/concepts/architecture/nodes.md @@ -12,7 +12,7 @@ weight: 10 Un nœud est une machine de travail dans Kubernetes, connue auparavant sous le nom de `minion`. Un nœud peut être une machine virtuelle ou une machine physique, selon le cluster. Chaque nœud contient les services nécessaires à l'exécution de [pods](/docs/concepts/workloads/pods/pod/) et est géré par les composants du master. -Les services sur un nœud incluent le [container runtime](/docs/concepts/overview/components/#node-components), kubelet and kube-proxy. +Les services sur un nœud incluent le [container runtime](/docs/concepts/overview/components/#node-components), kubelet et kube-proxy. Consultez la section [Le Nœud Kubernetes](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md#the-kubernetes-node) dans le document de conception de l'architecture pour plus de détails. From bd59a5fc2011edc9499f52d1e2ad579328bb057f Mon Sep 17 00:00:00 2001 From: Robert Martin Date: Sat, 25 Sep 2021 20:02:10 -0500 Subject: [PATCH 070/249] Update hello-minikube.md Fix minor typos --- content/en/docs/tutorials/hello-minikube.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/tutorials/hello-minikube.md b/content/en/docs/tutorials/hello-minikube.md index 3911ff2de6..e6398d7c72 100644 --- a/content/en/docs/tutorials/hello-minikube.md +++ b/content/en/docs/tutorials/hello-minikube.md @@ -69,13 +69,13 @@ By default, the dashboard is only accessible from within the internal Kubernetes The `dashboard` command creates a temporary proxy to make the dashboard accessible from outside the Kubernetes virtual network. To stop the proxy, run `Ctrl+C` to exit the process. -After the command exits, the dashboard remains running in Kubernetes cluster. +After the command exits, the dashboard remains running in the Kubernetes cluster. You can run the `dashboard` command again to create another proxy to access the dashboard. {{< /note >}} ## Open Dashboard with URL -If you don't want to open a web browser, run the dashboard command with the url flag to emit a URL: +If you don't want to open a web browser, run the dashboard command with the `--url` flag to emit a URL: ```shell minikube dashboard --url @@ -185,7 +185,7 @@ Kubernetes [*Service*](/docs/concepts/services-networking/service/). 4. Katacoda environment only: Click the plus sign, and then click **Select port to view on Host 1**. -5. Katacoda environment only: Note the 5 digit port number displayed opposite to `8080` in services output. This port number is randomly generated and it can be different for you. Type your number in the port number text box, then click Display Port. Using the example from earlier, you would type `30369`. +5. Katacoda environment only: Note the 5-digit port number displayed opposite to `8080` in services output. This port number is randomly generated and it can be different for you. Type your number in the port number text box, then click Display Port. Using the example from earlier, you would type `30369`. This opens up a browser window that serves your app and shows the app's response. From ea36ff4621bb5081b6abfb403f3384a839e648f6 Mon Sep 17 00:00:00 2001 From: Steven Yan Date: Fri, 24 Sep 2021 14:19:59 +0800 Subject: [PATCH 071/249] [zh] translation for admin 1 --- .../enabling-service-topology.md | 59 +++++++++------- .../administer-cluster/kubelet-config-file.md | 70 ++++++++----------- ...k-if-dockershim-deprecation-affects-you.md | 6 +- .../administer-cluster/safely-drain-node.md | 39 ++++++++++- 4 files changed, 104 insertions(+), 70 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/enabling-service-topology.md b/content/zh/docs/tasks/administer-cluster/enabling-service-topology.md index d97171de3e..1307065c5a 100644 --- a/content/zh/docs/tasks/administer-cluster/enabling-service-topology.md +++ b/content/zh/docs/tasks/administer-cluster/enabling-service-topology.md @@ -1,76 +1,83 @@ --- title: 开启服务拓扑 content_type: task +min-kubernetes-server-version: 1.17 --- +{{< feature-state for_k8s_version="v1.21" state="deprecated" >}} -本页面提供了在 Kubernetes 中启用服务拓扑的概述。 +这项功能，特别是 Alpha 状态的 `topologyKeys` 字段，在 kubernetes v1.21 中已经弃用。 +在 kubernetes v1.21 加入的[拓扑感知提示](/zh/docs/concepts/services-networking/topology-aware-hints/) +提供了类似的功能。 ## {{% heading "prerequisites" %}} {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} - - -## 介绍 -_服务拓扑（Service Topology）_ 使服务能够根据集群中的 Node 拓扑来路由流量。 +_服务拓扑（Service Topology）_ 使 {{< glossary_tooltip term_id="service" text="服务">}} +能够根据集群中的 Node 拓扑来路由流量。 比如，服务可以指定将流量优先路由到与客户端位于同一节点或者同一可用区域的端点上。 - -## 先决条件 - 需要下面列的先决条件，才能启用拓扑感知的服务路由： * Kubernetes 1.17 或更新版本 - * {{< glossary_tooltip text="Kube-proxy" term_id="kube-proxy" >}} 以 iptables 或者 IPVS 模式运行 - * 启用[端点切片](/zh/docs/concepts/services-networking/endpoint-slices/) + * 配置 {{< glossary_tooltip text="kube-proxy" term_id="kube-proxy" >}} 以 iptables 或者 IPVS 模式运行 + ## 启用服务拓扑 -{{< feature-state for_k8s_version="v1.17" state="alpha" >}} +{{< feature-state for_k8s_version="v1.21" state="deprecated" >}} -要启用服务拓扑功能，需要为所有 Kubernetes 组件启用 `ServiceTopology` 和 `EndpointSlice` 特性门控： +要启用服务拓扑功能，需要为所有 Kubernetes 组件启用 `ServiceTopology` +[特性门控](/zh/docs/reference/command-line-tools-reference/feature-gates/)： ``` ---feature-gates="ServiceTopology=true,EndpointSlice=true" +--feature-gates="ServiceTopology=true` ``` ## {{% heading "whatsnext" %}} -* 阅读[服务拓扑](/zh/docs/concepts/services-networking/service-topology)概念 + +* 阅读[拓扑感知提示](/zh/docs/concepts/services-networking/topology-aware-hints/)，该技术是用来替换 `topologyKeys` 字段的。 * 阅读[端点切片](/zh/docs/concepts/services-networking/endpoint-slices) +* 阅读[服务拓扑](/zh/docs/concepts/services-networking/service-topology)概念 * 阅读[通过服务来连接应用](/zh/docs/concepts/services-networking/connect-applications-service/) \ No newline at end of file diff --git a/content/zh/docs/tasks/administer-cluster/kubelet-config-file.md b/content/zh/docs/tasks/administer-cluster/kubelet-config-file.md index e41eb9e3d9..3f8ee37eb7 100644 --- a/content/zh/docs/tasks/administer-cluster/kubelet-config-file.md +++ b/content/zh/docs/tasks/administer-cluster/kubelet-config-file.md @@ -12,15 +12,11 @@ content_type: task -{{< feature-state for_k8s_version="v1.10" state="beta" >}} - 通过保存在硬盘的配置文件设置 kubelet 的部分配置参数，这可以作为命令行参数的替代。 -此功能在 v1.10 中为 beta 版。 建议通过配置文件的方式提供参数，因为这样可以简化节点部署和配置管理。 -## {{% heading "prerequisites" %}} - - -- 需要安装 1.10 或更高版本的 kubelet 可执行文件，才能使用此 beta 功能。 - ## 创建配置文件 -`KubeletConfiguration` 结构体定义了可以通过文件配置的 Kubelet 配置子集， -该结构体在 [这里（v1beta1）](https://github.com/kubernetes/kubernetes/blob/{{< param "docsbranch" >}}/staging/src/k8s.io/kubelet/config/v1beta1/types.go) -可以找到。 +[`KubeletConfiguration`](/zh/docs/reference/config-api/kubelet-config.v1beta1/) 结构体定义了可以通过文件配置的 Kubelet 配置子集， -在这个示例中, 当可用内存低于 200Mi 时, kubelet 将会开始驱逐 Pods。 +在这个示例中, Kubelet 被设置为在地址 192.168.0.8 端口 20250 上提供服务，以并行方式拖拽镜像， +当可用内存低于 200Mi 时, kubelet 将会开始驱逐 Pods。 没有声明的其余配置项都将使用默认值，除非使用命令行参数来重载。 命令行中的参数将会覆盖配置文件中的对应值。 -作为一个小技巧，你可以从活动节点生成配置文件，相关方法请查看 -[重新配置活动集群节点的 kubelet](/zh/docs/tasks/administer-cluster/reconfigure-kubelet)。 - ## 启动通过配置文件配置的 Kubelet 进程 +{{< note >}} +如果你使用 kubeadm 初始化你的集群，在使用 `kubeadmin init` 创建你的集群的时候请使用 kubelet-config。 +更多细节请阅读[使用 kubeadm 配置 kubelet](/zh/docs/setup/production-environment/tools/kubeadm/kubelet-integration/) +{{< /note >}} + 启动 Kubelet 需要将 `--config` 参数设置为 Kubelet 配置文件的路径。Kubelet 将从此文件加载其配置。 +## {{% heading "whatsnext" %}} + -## 与动态 Kubelet 配置的关系 - -如果你正在使用[动态 kubelet 配置](/zh/docs/tasks/administer-cluster/reconfigure-kubelet)特性， -那么自动回滚机制将认为通过 `--config` 提供的配置与覆盖这些值的任何参数的组合是 - "最后已知正常（last known good）" 的配置。 - - +- 参阅 [`KubeletConfiguration`](/zh/docs/reference/config-api/kubelet-config.v1beta1/) + 进一步学习 kubelet 的配置。 diff --git a/content/zh/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you.md b/content/zh/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you.md index e3d8d89d94..c1cf73c96e 100644 --- a/content/zh/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you.md +++ b/content/zh/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you.md @@ -99,14 +99,14 @@ Kubernetes 负责编排和调度 Pod；在每一个节点上， 使用抽象的容器运行时接口，所以你可以任意选用兼容的容器运行时。 -在早期版本中，Kubernetes 提供的兼容性只支持一个容器运行时：Docker。 +在早期版本中，Kubernetes 提供的兼容性支持一个容器运行时：Docker。 在 Kubernetes 发展历史中，集群运营人员希望采用更多的容器运行时。 于是 CRI 被设计出来满足这类灵活性需要 - 而 kubelet 亦开始支持 CRI。 然而，因为 Docker 在 CRI 规范创建之前就已经存在，Kubernetes 就创建了一个适配器组件：`dockershim`。 @@ -144,7 +144,7 @@ or execute something inside container using `docker exec`. {{< note >}} diff --git a/content/zh/docs/tasks/administer-cluster/safely-drain-node.md b/content/zh/docs/tasks/administer-cluster/safely-drain-node.md index 85de0215ca..c07c282b78 100644 --- a/content/zh/docs/tasks/administer-cluster/safely-drain-node.md +++ b/content/zh/docs/tasks/administer-cluster/safely-drain-node.md @@ -196,6 +196,39 @@ itself. To attempt an eviction (perhaps more REST-precisely, to attempt to Pod 的 Eviction 子资源可以看作是一种策略控制的 DELETE 操作，作用于 Pod 本身。 要尝试驱逐（更准确地说，尝试 *创建* 一个 Eviction），需要用 POST 发出所尝试的操作。这里有一个例子： +{{< tabs name="Eviction_example" >}} +{{% tab name="policy/v1" %}} + +{{< note >}} +`policy/v1` 驱逐在 v1.22+ 中可用。在之前版本中请使用 `policy/v1beta1` 。 +{{< /note >}} + + +```json +{ + "apiVersion": "policy/v1", + "kind": "Eviction", + "metadata": { + "name": "quux", + "namespace": "default" + } +} +``` +{{% /tab %}} +{{% tab name="policy/v1beta1" %}} + +{{< note >}} +在 v1.22 中已弃用，以 `policy/v1` 取代 +{{< /note >}} + ```json { "apiVersion": "policy/v1beta1", @@ -206,6 +239,8 @@ Pod 的 Eviction 子资源可以看作是一种策略控制的 DELETE 操作， } } ``` +{{% /tab %}} +{{< /tabs >}} 你首先需要有一个支持网络策略的 Kubernetes 集群。已经有许多支持 NetworkPolicy 的网络提供商，包括： +* [Antrea](/zh/docs/tasks/administer-cluster/network-policy-provider/antrea-network-policy/) * [Calico](/zh/docs/tasks/administer-cluster/network-policy-provider/calico-network-policy/) * [Cilium](/zh/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy/) * [Kube-router](/zh/docs/tasks/administer-cluster/network-policy-provider/kube-router-network-policy/) diff --git a/content/zh/docs/tasks/administer-cluster/extended-resource-node.md b/content/zh/docs/tasks/administer-cluster/extended-resource-node.md index 9eeb4e09bc..5c1d54b590 100644 --- a/content/zh/docs/tasks/administer-cluster/extended-resource-node.md +++ b/content/zh/docs/tasks/administer-cluster/extended-resource-node.md @@ -67,13 +67,13 @@ Host: k8s-master:8080 注意：Kubernetes 不需要了解 dongle 资源的含义和用途。 -前面的 PATCH 请求仅仅告诉 Kubernetes 你的节点拥有四个你称之为 dongle 的东西。 +前面的 PATCH 请求告诉 Kubernetes 你的节点拥有四个你称之为 dongle 的东西。 启动一个代理（proxy），以便你可以很容易地向 Kubernetes API server 发送请求： diff --git a/content/zh/docs/tasks/administer-cluster/namespaces-walkthrough.md b/content/zh/docs/tasks/administer-cluster/namespaces-walkthrough.md index 7fbf132dcf..39f3f01718 100644 --- a/content/zh/docs/tasks/administer-cluster/namespaces-walkthrough.md +++ b/content/zh/docs/tasks/administer-cluster/namespaces-walkthrough.md @@ -345,9 +345,9 @@ Apply the manifest to create a Deployment 应用清单文件来创建 Deployment。 -我们刚刚创建了一个副本大小为 2 的 Deployment，该 Deployment 运行名为 `snowflake` 的 Pod， +我们创建了一个副本大小为 2 的 Deployment，该 Deployment 运行名为 `snowflake` 的 Pod， 其中包含一个仅提供主机名服务的基本容器。 ```shell diff --git a/content/zh/docs/tasks/administer-cluster/network-policy-provider/antrea-network-policy.md b/content/zh/docs/tasks/administer-cluster/network-policy-provider/antrea-network-policy.md new file mode 100644 index 0000000000..d8dbecfcce --- /dev/null +++ b/content/zh/docs/tasks/administer-cluster/network-policy-provider/antrea-network-policy.md @@ -0,0 +1,49 @@ + +--- +title: 使用 Antrea 提供 NetworkPolicy +content_type: task +weight: 10 +--- + + + +本页展示了如何在 kubernetes 中安装和使用 Antrea CNI 插件。 +要了解 Antrea 项目的背景，请阅读 [Antrea 介绍](https://antrea.io/docs/)。 + +## {{% heading "prerequisites" %}} + + +你需要拥有一个 kuernetes 集群。 +遵循 [kubeadm 入门指南](/zh/docs/reference/setup-tools/kubeadm/)自行创建一个。 + + + + +## 使用 kubeadm 部署 Antrea +遵循[入门](https://github.com/vmware-tanzu/antrea/blob/main/docs/getting-started.md)指南 +为 kubeadm 部署 Antrea 。 + +## {{% heading "whatsnext" %}} + + +一旦你的集群已经运行，你可以遵循 +[声明网络策略](/zh/docs/tasks/administer-cluster/declare-network-policy/) +来尝试 Kubernetes NetworkPolicy。 \ No newline at end of file diff --git a/content/zh/docs/tasks/administer-cluster/network-policy-provider/calico-network-policy.md b/content/zh/docs/tasks/administer-cluster/network-policy-provider/calico-network-policy.md index 2b5e260c5c..9c0dee6fbb 100644 --- a/content/zh/docs/tasks/administer-cluster/network-policy-provider/calico-network-policy.md +++ b/content/zh/docs/tasks/administer-cluster/network-policy-provider/calico-network-policy.md @@ -29,9 +29,9 @@ Decide whether you want to deploy a [cloud](#creating-a-calico-cluster-with-goog **先决条件**: [gcloud](https://cloud.google.com/sdk/docs/quickstarts) -1. 启动一个带有 Calico 的 GKE 集群，只需加上参数 `--enable-network-policy`。 +1. 启动一个带有 Calico 的 GKE 集群，需要加上参数 `--enable-network-policy`。 **语法** ```shell diff --git a/content/zh/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md b/content/zh/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md index d75203b71e..697e102753 100644 --- a/content/zh/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md +++ b/content/zh/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md @@ -33,25 +33,25 @@ For background on Cilium, read the [Introduction to Cilium](https://docs.cilium. ## Deploying Cilium on Minikube for Basic Testing To get familiar with Cilium easily you can follow the -[Cilium Kubernetes Getting Started Guide](https://docs.cilium.io/en/stable/gettingstarted/minikube/) +[Cilium Kubernetes Getting Started Guide](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/) to perform a basic DaemonSet installation of Cilium in minikube. -To start minikube, minimal version required is >= v1.3.1, run the with the +To start minikube, minimal version required is >= v1.5.2, run the with the following arguments: --> ## 在 Minikube 上部署 Cilium 用于基本测试 为了轻松熟悉 Cilium 你可以根据 -[Cilium Kubernetes 入门指南](https://docs.cilium.io/en/stable/gettingstarted/minikube/) +[Cilium Kubernetes 入门指南](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/s) 在 minikube 中执行一个 cilium 的基本 DaemonSet 安装。 -要启动 minikube，需要的最低版本为 1.3.1，使用下面的参数运行： +要启动 minikube，需要的最低版本为 1.5.2，使用下面的参数运行： ```shell minikube version ``` ``` -minikube version: v1.3.1 +minikube version: v1.5.2 ``` ```shell @@ -59,36 +59,45 @@ minikube start --network-plugin=cni --memory=4096 ``` -挂载 BPF 文件系统： +对于 minikube 你可以使用 Cilium 的 CLI 工具安装它。 +Cilium 将自动检测集群配置并为成功的集群部署选择合适的组件。 ```shell -minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf +curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz +sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin +rm cilium-linux-amd64.tar.gz +cilium install ``` - - -在 minikube 环境中，你可以部署下面的"一体化" YAML 文件，其中包含 Cilium -的 DaemonSet 配置以及适当的 RBAC 配置： - - -```shell -kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.8/install/kubernetes/quick-install.yaml ``` - -``` -configmap/cilium-config created -serviceaccount/cilium created -serviceaccount/cilium-operator created -clusterrole.rbac.authorization.k8s.io/cilium created -clusterrole.rbac.authorization.k8s.io/cilium-operator created -clusterrolebinding.rbac.authorization.k8s.io/cilium created -clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created -daemonset.apps/cilium create -deployment.apps/cilium-operator created +🔮 Auto-detected Kubernetes kind: minikube +✨ Running "minikube" validation checks +✅ Detected minikube version "1.20.0" +ℹ️ Cilium version not set, using default version "v1.10.0" +🔮 Auto-detected cluster name: minikube +🔮 Auto-detected IPAM mode: cluster-pool +🔮 Auto-detected datapath mode: tunnel +🔑 Generating CA... +2021/05/27 02:54:44 [INFO] generate received request +2021/05/27 02:54:44 [INFO] received CSR +2021/05/27 02:54:44 [INFO] generating key: ecdsa-256 +2021/05/27 02:54:44 [INFO] encoded CSR +2021/05/27 02:54:44 [INFO] signed certificate with serial number 48713764918856674401136471229482703021230538642 +🔑 Generating certificates for Hubble... +2021/05/27 02:54:44 [INFO] generate received request +2021/05/27 02:54:44 [INFO] received CSR +2021/05/27 02:54:44 [INFO] generating key: ecdsa-256 +2021/05/27 02:54:44 [INFO] encoded CSR +2021/05/27 02:54:44 [INFO] signed certificate with serial number 3514109734025784310086389188421560613333279574 +🚀 Creating Service accounts... +🚀 Creating Cluster roles... +🚀 Creating ConfigMap... +🚀 Creating Agent DaemonSet... +🚀 Creating Operator Deployment... +⌛ Waiting for Cilium to be installed... ``` 你将看到像这样的 Pods 列表： ```console -NAME READY STATUS RESTARTS AGE -cilium-6rxbd 1/1 Running 0 1m +NAME READY STATUS RESTARTS AGE +cilium-kkdhz 1/1 Running 0 3m23s ... ``` diff --git a/content/zh/docs/tasks/administer-cluster/reserve-compute-resources.md b/content/zh/docs/tasks/administer-cluster/reserve-compute-resources.md index f03e2bde30..e0b2d8a6e8 100644 --- a/content/zh/docs/tasks/administer-cluster/reserve-compute-resources.md +++ b/content/zh/docs/tasks/administer-cluster/reserve-compute-resources.md @@ -26,7 +26,7 @@ itself. Unless resources are set aside for these system daemons, pods and system daemons compete for resources and lead to resource starvation issues on the node. -The `kubelet` exposes a feature named `Node Allocatable` that helps to reserve +The `kubelet` exposes a feature named 'Node Allocatable' that helps to reserve compute resources for system daemons. Kubernetes recommends cluster administrators to configure `Node Allocatable` based on their workload density on each node. @@ -35,12 +35,11 @@ Kubernetes 的节点可以按照 `Capacity` 调度。默认情况下 pod 能够 这是个问题，因为节点自己通常运行了不少驱动 OS 和 Kubernetes 的系统守护进程。 除非为这些系统守护进程留出资源，否则它们将与 pod 争夺资源并导致节点资源短缺问题。 -`kubelet` 公开了一个名为 `Node Allocatable` 的特性，有助于为系统守护进程预留计算资源。 +`kubelet` 公开了一个名为 'Node Allocatable' 的特性，有助于为系统守护进程预留计算资源。 Kubernetes 推荐集群管理员按照每个节点上的工作负载密度配置 `Node Allocatable`。 ## {{% heading "prerequisites" %}} - {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} 要想为系统守护进程上可选地实施 `system-reserved` 约束，请指定 kubelet 的 `--system-reserved-cgroup` 标志值为 OS 系统守护进程的父级控制组。 @@ -234,19 +232,19 @@ exist. Kubelet will fail if an invalid cgroup is specified. 推荐将 OS 系统守护进程放在一个顶级控制组之下（例如 systemd 机器上的 `system.slice`）。 -请注意，如果 `--system-reserved-cgroup` 不存在，Kubelet **不会** 创建它。 -如果指定了无效的 cgroup，Kubelet 将会失败。 +请注意，如果 `--system-reserved-cgroup` 不存在，`kubelet` **不会** 创建它。 +如果指定了无效的 cgroup，`kubelet` 将会失败。 ### 显式保留的 CPU 列表 {#explicitly-reserved-cpu-list} {{< feature-state for_k8s_version="v1.17" state="stable" >}} -- **Kubelet 标志**: `--reserved-cpus=0-3` +-**Kubelet 标志**: `--reserved-cpus=0-3` ### 实施节点可分配约束 {#enforcing-node-allocatable} -- **Kubelet 标志**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]` +-**Kubelet 标志**: `--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]` -调度器将 `Allocatable` 视为 Pod 可用的 `capacity`（资源容量）。 +调度器将 'Allocatable' 视为 Pod 可用的 `capacity`（资源容量）。 -`kubelet` 默认对 Pod 执行 `Allocatable` 约束。 -无论何时，如果所有 Pod 的总用量超过了 `Allocatable`，驱逐 Pod 的措施将被执行。 +`kubelet` 默认对 Pod 执行 'Allocatable' 约束。 +无论何时，如果所有 Pod 的总用量超过了 'Allocatable'，驱逐 Pod 的措施将被执行。 有关驱逐策略的更多细节可以在 -[这里](/zh/docs/tasks/administer-cluster/out-of-resource/#eviction-policy)找到。 +[节点压力驱逐](/zh/docs/concepts/scheduling-eviction/node-pressure-eviction/)页找到。 可通过设置 kubelet `--enforce-node-allocatable` 标志值为 `pods` 控制这个措施。 可选地，通过在同一标志中同时指定 `kube-reserved` 和 `system-reserved` 值， @@ -351,7 +351,7 @@ respectively. ## 一般原则 {#general-guidelines} -系统守护进程一般会被按照类似 `Guaranteed` Pod 一样对待。 +系统守护进程一般会被按照类似 'Guaranteed' Pod 一样对待。 系统守护进程可以在与其对应的控制组中出现突发资源用量，这一行为要作为 kubernetes 部署的一部分进行管理。 例如，`kubelet` 应该有它自己的控制组并和容器运行时共享 `Kube-reserved` 资源。 @@ -373,9 +373,9 @@ to critical system services being CPU starved, OOM killed, or unable to fork on the node. The recommendation is to enforce `system-reserved` only if a user has profiled their nodes exhaustively to come up with precise estimates and is confident in their -ability to recover if any process in that group is oom_killed. +ability to recover if any process in that group is oom-killed. -* To begin with enforce `Allocatable` on `pods`. +* To begin with enforce 'Allocatable' on `pods`. * Once adequate monitoring and alerting is in place to track kube system daemons, attempt to enforce `kube-reserved` based on usage heuristics. * If absolutely necessary, enforce `system-reserved` over time. @@ -386,7 +386,7 @@ ability to recover if any process in that group is oom_killed. 并且对该组中进程因内存不足而被杀死时，有足够的信心将其恢复时， 才可以强制执行 `system-reserved` 策略。 -* 作为起步，可以先针对 `pods` 上执行 `Allocatable` 约束。 +* 作为起步，可以先针对 `pods` 上执行 'Allocatable' 约束。 * 一旦用于追踪系统守护进程的监控和告警的机制到位，可尝试基于用量估计的 方式执行 `kube-reserved`策略。 * 随着时间推进，在绝对必要的时候可以执行 `system-reserved` 策略。 @@ -424,27 +424,27 @@ Here is an example to illustrate Node Allocatable computation: * `--eviction-hard` 被设置为 `memory.available<500Mi,nodefs.available<10%` -在这个场景下，`Allocatable` 将会是 `14.5 CPUs`、`28.5Gi` 内存以及 `88Gi` 本地存储。 -调度器保证这个节点上的所有 Pod 的内存 `requests` 总量不超过 `28.5Gi`， -存储不超过 `88Gi`。 -当 Pod 的内存使用总量超过 `28.5Gi` 或者磁盘使用总量超过 `88Gi` 时， +在这个场景下，'Allocatable' 将会是 14.5 CPUs、28.5Gi 内存以及 `88Gi` 本地存储。 +调度器保证这个节点上的所有 Pod 的内存 `requests` 总量不超过 28.5Gi， +存储不超过 '88Gi'。 +当 Pod 的内存使用总量超过 28.5Gi 或者磁盘使用总量超过 88Gi 时， kubelet 将会驱逐它们。 如果节点上的所有进程都尽可能多地使用 CPU，则 Pod 加起来不能使用超过 -`14.5 CPUs` 的资源。 +14.5 CPUs 的资源。 当没有执行 `kube-reserved` 和/或 `system-reserved` 策略且系统守护进程 -使用量超过其预留时，如果节点内存用量高于 `31.5Gi` 或`存储`大于 `90Gi`， +使用量超过其预留时，如果节点内存用量高于 31.5Gi 或`存储`大于 90Gi， kubelet 将会驱逐 Pod。 From 6aaf609db6526741d9c4bd933c5233309aa5e30e Mon Sep 17 00:00:00 2001 From: howieyuen Date: Mon, 27 Sep 2021 11:48:07 +0800 Subject: [PATCH 074/249] [zh]sync content/zh/docs/concepts/configuration/secret.md --- .../zh/docs/concepts/configuration/secret.md | 110 ++++++++++-------- 1 file changed, 64 insertions(+), 46 deletions(-) diff --git a/content/zh/docs/concepts/configuration/secret.md b/content/zh/docs/concepts/configuration/secret.md index 53a301f03e..0d018141a7 100644 --- a/content/zh/docs/concepts/configuration/secret.md +++ b/content/zh/docs/concepts/configuration/secret.md @@ -22,47 +22,58 @@ weight: 30 - - -`Secret` 对象类型用来保存敏感信息，例如密码、OAuth 令牌和 SSH 密钥。 -将这些信息放在 `secret` 中比放在 {{< glossary_tooltip term_id="pod" >}} 的定义或者 {{< glossary_tooltip text="容器镜像" term_id="image" >}} 中来说更加安全和灵活。 -参阅 [Secret 设计文档](https://git.k8s.io/community/contributors/design-proposals/auth/secrets.md) 获取更多详细信息。 - Secret 是一种包含少量敏感信息例如密码、令牌或密钥的对象。 -这样的信息可能会被放在 Pod 规约中或者镜像中。 -用户可以创建 Secret，同时系统也创建了一些 Secret。 +这样的信息可能会被放在 {{< glossary_tooltip term_id="pod" >}} 规约中或者镜像中。 +使用 Secret 意味着你不需要在应用程序代码中包含机密数据。 + + +由于创建 Secret 可以独立于使用它们的 Pod， +因此在创建、查看和编辑 Pod 的工作流程中暴露 Secret（及其数据）的风险较小。 +Kubernetes 和在集群中运行的应用程序也可以对 Secret 采取额外的预防措施， +例如避免将机密数据写入非易失性存储。 + +Secret 类似于 {{}} +但专门用于保存机密数据。 {{< caution >}} -Kubernetes Secret 默认情况下存储为 base64-编码的、非加密的字符串。 -默认情况下，能够访问 API 的任何人，或者能够访问 Kubernetes 下层数据存储（etcd） -的任何人都可以以明文形式读取这些数据。 -为了能够安全地使用 Secret，我们建议你（至少）： +默认情况下，Kubernetes Secret 未加密地存储在 API 服务器的底层数据存储（etcd）中。 +任何拥有 API 访问权限的人都可以检索或修改 Secret，任何有权访问 etcd 的人也可以。 +此外，任何有权限在命名空间中创建 Pod 的人都可以使用该访问权限读取该命名空间中的任何 Secret； +这包括间接访问，例如创建 Deployment 的能力。 + +为了安全地使用 Secret，请至少执行以下步骤： 1. 为 Secret [启用静态加密](/zh/docs/tasks/administer-cluster/encrypt-data/)； -2. [启用 或配置 RBAC 规则](/zh/docs/reference/access-authn-authz/authorization/)来限制对 Secret 的读写操作。 - 要注意，任何被允许创建 Pod 的人都默认地具有读取 Secret 的权限。 +2. 启用或配置 [RBAC 规则](/zh/docs/reference/access-authn-authz/authorization/)来限制读取 Secret 的数据（包括通过间接方式）。 +3. 在适当的情况下，还可以使用 RBAC 等机制来限制允许哪些主体创建新 Secret 或替换现有 Secret。 {{< /caution >}} @@ -89,6 +100,15 @@ Pod 可以用三种方式之一来使用 Secret： - 作为[容器的环境变量](#using-secrets-as-environment-variables) - 由 [kubelet 在为 Pod 拉取镜像时使用](#using-imagepullsecrets) + +Kubernetes 控制平面也使用 Secret； +例如，[引导令牌 Secret](#bootstrap-token-secrets) +是一种帮助自动化节点注册的机制。 + ## Secret 的类型 {#secret-types} -在创建 Secret 对象时，你可以使用 -[`Secret`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#secret-v1-core) -资源的 `type` 字段，或者与其等价的 `kubectl` 命令行参数（如果有的话）为其设置类型。 -Secret 的类型用来帮助编写程序处理 Secret 数据。 +创建 Secret 时，你可以使用 Secret 资源的 `type` 字段， +或者与其等价的 `kubectl` 命令行参数（如果有的话）为其设置类型。 +Secret 的 `type` 有助于对不同类型机密数据的编程处理。 Kubernetes 提供若干种内置的类型，用于一些常见的使用场景。 针对这些类型，Kubernetes 所执行的合法性检查操作以及对其所实施的限制各不相同。 @@ -553,13 +572,13 @@ kubectl create secret tls my-tls-secret \ ``` 这里的公钥/私钥对都必须事先已存在。用于 `--cert` 的公钥证书必须是 .PEM 编码的 （Base64 编码的 DER 格式），且与 `--key` 所给定的私钥匹配。 @@ -1334,6 +1353,8 @@ This includes any pods created using kubectl, or indirectly via a replication controller. It does not include pods created via the kubelets `--manifest-url` flag, its `--config` flag, or its REST API (these are not common ways to create pods.) +The `spec` of a {{< glossary_tooltip text="static Pod" term_id="static-pod" >}} cannot refer to a Secret +or any other API objects. --> 每个 Secret 的大小限制为 1MB。这是为了防止创建非常大的 Secret 导致 API 服务器 和 kubelet 的内存耗尽。然而，创建过多较小的 Secret 也可能耗尽内存。 @@ -1343,6 +1364,8 @@ kubelet 仅支持从 API 服务器获得的 Pod 使用 Secret。 这包括使用 `kubectl` 创建的所有 Pod，以及间接通过副本控制器创建的 Pod。 它不包括通过 kubelet `--manifest-url` 标志，`--config` 标志或其 REST API 创建的 Pod（这些不是创建 Pod 的常用方法）。 +{{}} +的 `spec` 不能引用 Secret 或任何其他 API 对象。 ### 风险 @@ -1968,9 +1987,6 @@ for secret data, so that the secrets are not stored in the clear into {{< glossa - 如果您将 Secret 数据编码为 base64 的清单（JSON 或 YAML）文件，共享该文件或将其检入代码库，该密码将会被泄露。 Base64 编码不是一种加密方式，应该视同纯文本。 - 应用程序在从卷中读取 Secret 后仍然需要保护 Secret 的值，例如不会意外将其写入日志或发送给不信任方。 - 可以创建使用 Secret 的 Pod 的用户也可以看到该 Secret 的值。即使 API 服务器策略不允许用户读取 Secret 对象，用户也可以运行 Pod 导致 Secret 暴露。 -- 目前，任何节点的 root 用户都可以通过模拟 kubelet 来读取 API 服务器中的任何 Secret。 - 仅向实际需要 Secret 的节点发送 Secret 数据才能限制节点的 root 账号漏洞的影响， - 该功能还在计划中。 ## {{% heading "whatsnext" %}} @@ -1978,8 +1994,10 @@ for secret data, so that the secrets are not stored in the clear into {{< glossa - Learn how to [manage Secret using `kubectl`](/docs/tasks/configmap-secret/managing-secret-using-kubectl/) - Learn how to [manage Secret using config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/) - Learn how to [manage Secret using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/) +- Read the [API reference](/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/) for `Secret` --> - 学习如何[使用 `kubectl` 管理 Secret](/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl/) - 学习如何[使用配置文件管理 Secret](/zh/docs/tasks/configmap-secret/managing-secret-using-config-file/) - 学习如何[使用 kustomize 管理 Secret](/zh/docs/tasks/configmap-secret/managing-secret-using-kustomize/) +- 阅读 [API 参考](/zh/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/)了解 `Secret` From 649cc1d25ad16a3476e9ccd0cb5d57a512413d8b Mon Sep 17 00:00:00 2001 From: chenxuc Date: Sat, 18 Sep 2021 10:20:11 +0800 Subject: [PATCH 075/249] [zh] sync admin cluster docs --- .../access-cluster-services.md | 6 ++-- .../tasks/administer-cluster/certificates.md | 7 ++++ ...aranteed-scheduling-critical-addon-pods.md | 24 +++++++------ .../administer-cluster/securing-a-cluster.md | 35 ++++++++++--------- .../administer-cluster/sysctl-cluster.md | 12 +++++-- .../administer-cluster/topology-manager.md | 16 +++++---- 6 files changed, 61 insertions(+), 39 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/access-cluster-services.md b/content/zh/docs/tasks/administer-cluster/access-cluster-services.md index ee817c141f..d87fb93758 100644 --- a/content/zh/docs/tasks/administer-cluster/access-cluster-services.md +++ b/content/zh/docs/tasks/administer-cluster/access-cluster-services.md @@ -46,7 +46,7 @@ You have several options for connecting to nodes, pods and services from outside - Use a service with type `NodePort` or `LoadBalancer` to make the service reachable outside the cluster. See the [services](/docs/concepts/services-networking/service/) and [kubectl expose](/docs/reference/generated/kubectl/kubectl-commands/#expose) documentation. - - Depending on your cluster environment, this may just expose the service to your corporate network, + - Depending on your cluster environment, this may only expose the service to your corporate network, or it may expose it to the internet. Think about whether the service being exposed is secure. Does it do its own authentication? - Place pods behind services. To access one specific pod from a set of replicas, such as for debugging, @@ -148,7 +148,7 @@ See [Access Clusters Using the Kubernetes API](/docs/tasks/administer-cluster/ac +1. 查看证书签名请求： + + openssl req -noout -text -in ./server.csr + diff --git a/content/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md b/content/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md index 0f6d8d1312..111d30d32d 100644 --- a/content/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md +++ b/content/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods.md @@ -6,20 +6,25 @@ content_type: concept -除了在主机上运行的 Kubernetes 核心组件（如 api-server 、scheduler 、controller-manager）之外，还有许多插件，由于各种原因， -必须在常规集群节点（而不是 Kubernetes 主节点）上运行。 +Kubernetes 核心组件（如 API 服务器、调度器、控制器管理器）在控制平面节点上运行。 +但是插件必须在常规集群节点上运行。 其中一些插件对于功能完备的群集至关重要，例如 Heapster、DNS 和 UI。 如果关键插件被逐出（手动或作为升级等其他操作的副作用）或者变成挂起状态，群集可能会停止正常工作。 关键插件进入挂起状态的例子有：集群利用率过高；被逐出的关键插件 Pod 释放了空间，但该空间被之前悬决的 Pod 占用；由于其它原因导致节点上可用资源的总量发生变化。 - + +注意，把某个 Pod 标记为关键 Pod 并不意味着完全避免该 Pod 被逐出；它只能防止该 Pod 变成永久不可用。 +被标记为关键性的静态 Pod 不会被逐出。但是，被标记为关键性的非静态 Pod 总是会被重新调度。 @@ -29,12 +34,9 @@ vacated by the evicted critical add-on pod or the amount of resources available ### 标记关键 Pod -要将 pod 标记为关键性（critical），pod 必须在 kube-system 命名空间中运行（可通过参数配置）。 -同时，需要将 `priorityClassName` 设置为 `system-cluster-critical` 或 `system-node-critical` ，后者是整个群集的最高级别。 -或者，也可以为 Pod 添加名为 `scheduler.alpha.kubernetes.io/critical-pod`、值为空字符串的注解。 -不过，这一注解从 1.13 版本开始不再推荐使用，并将在 1.14 中删除。 +要将 Pod 标记为关键性（critical），设置 Pod 的 priorityClassName 为 `system-cluster-critical` 或者 `system-node-critical`。 +`system-node-critical` 是最高级别的可用性优先级，甚至比 `system-cluster-critical` 更高。 diff --git a/content/zh/docs/tasks/administer-cluster/securing-a-cluster.md b/content/zh/docs/tasks/administer-cluster/securing-a-cluster.md index ee3a9cf5fa..5731d530b3 100644 --- a/content/zh/docs/tasks/administer-cluster/securing-a-cluster.md +++ b/content/zh/docs/tasks/administer-cluster/securing-a-cluster.md @@ -29,7 +29,7 @@ and provides recommendations on overall security. ## 控制对 Kubernetes API 的访问 @@ -53,7 +53,7 @@ Kubernetes 期望集群中所有的 API 通信在默认情况下都使用 TLS ### API Authentication Choose an authentication mechanism for the API servers to use that matches the common access patterns -when you install a cluster. For instance, small single user clusters may wish to use a simple certificate +when you install a cluster. For instance, small single-user clusters may wish to use a simple certificate or static Bearer token approach. Larger clusters may wish to integrate an existing OIDC or LDAP server that allow users to be subdivided into groups. @@ -80,7 +80,7 @@ Consult the [authentication reference document](/docs/reference/access-authn-aut Once authenticated, every API call is also expected to pass an authorization check. Kubernetes ships an integrated [Role-Based Access Control (RBAC)](/docs/reference/access-authn-authz/rbac/) component that matches an incoming user or group to a set of permissions bundled into roles. These permissions combine verbs (get, create, delete) with -resources (pods, services, nodes) and can be namespace or cluster scoped. A set of out of the box +resources (pods, services, nodes) and can be namespace-scoped or cluster-scoped. A set of out-of-the-box roles are provided that offer reasonable default separation of responsibility depending on what actions a client might want to perform. It is recommended that you use the [Node](/docs/reference/access-authn-authz/node/) and [RBAC](/docs/reference/access-authn-authz/rbac/) authorizers together, in combination with the [NodeRestriction](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) admission plugin. @@ -110,8 +110,8 @@ With authorization, it is important to understand how updates on one object may other places. For instance, a user may not be able to create pods directly, but allowing them to create a deployment, which creates pods on their behalf, will let them create those pods indirectly. Likewise, deleting a node from the API will result in the pods scheduled to that node -being terminated and recreated on other nodes. The out of the box roles represent a balance -between flexibility and the common use cases, but more limited roles should be carefully reviewed +being terminated and recreated on other nodes. The out-of-the-box roles represent a balance +between flexibility and common use cases, but more limited roles should be carefully reviewed to prevent accidental escalation. You can make roles specific to your use case if the out-of-box ones don't meet your needs. Consult the [authorization reference section](/docs/reference/access-authn-authz/authorization/) for more information. @@ -183,7 +183,7 @@ reserved resources like memory, or to provide default limits when none are speci ### Controlling what privileges containers run with A pod definition contains a [security context](/docs/tasks/configure-pod-container/security-context/) -that allows it to request access to running as a specific Linux user on a node (like root), +that allows it to request access to run as a specific Linux user on a node (like root), access to run privileged or access the host network, and other controls that would otherwise allow it to run unfettered on a hosting node. [Pod security policies](/docs/concepts/policy/pod-security-policy/) can limit which users or service accounts can provide dangerous security context settings. For example, pod security policies can limit volume mounts, especially `hostPath`, which are aspects of a pod that should be controlled. @@ -227,11 +227,11 @@ now respect network policy. 对于可以控制用户的应用程序是否在集群之外可见的许多集群，配额和限制范围也可用于 @@ -248,7 +248,7 @@ By default these APIs are accessible by pods running on an instance and can cont credentials for that node, or provisioning data such as kubelet credentials. These credentials can be used to escalate within the cluster or to other cloud services under the same account. -When running Kubernetes on a cloud platform limit permissions given to instance credentials, use +When running Kubernetes on a cloud platform, limit permissions given to instance credentials, use [network policies](/docs/tasks/administer-cluster/declare-network-policy/) to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets. --> @@ -268,7 +268,7 @@ to the metadata API, and avoid using provisioning data to deliver secrets. By default, there are no restrictions on which nodes may run a pod. Kubernetes offers a [rich set of policies for controlling placement of pods onto nodes](/docs/concepts/configuration/assign-pod-node/) -and the [taint based pod placement and eviction](/docs/concepts/configuration/taint-and-toleration/) +and the [taint-based pod placement and eviction](/docs/concepts/configuration/taint-and-toleration/) that are available to end users. For many clusters use of these policies to separate workloads can be a convention that authors adopt or enforce via tooling. @@ -360,7 +360,7 @@ Kubernetes 的 alpha 和 beta 特性还在努力开发中，可能存在导致 The shorter the lifetime of a secret or credential the harder it is for an attacker to make use of that credential. Set short lifetimes on certificates and automate their rotation. Use an authentication provider that can control how long issued tokens are available and use short -lifetimes where possible. If you use service account tokens in external integrations, plan to +lifetimes where possible. If you use service-account tokens in external integrations, plan to rotate those tokens frequently. For example, once the bootstrap phase is complete, a bootstrap token used for setting up nodes should be revoked or its authorization removed. --> ### 频繁回收基础设施证书 @@ -406,9 +406,10 @@ and may grant an attacker significant visibility into the state of your cluster. your backups using a well reviewed backup and encryption solution, and consider using full disk encryption where possible. -Kubernetes 1.7 contains [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/), an alpha feature that will encrypt `Secret` resources in etcd, preventing +Kubernetes supports [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/), a feature +introduced in 1.7, and beta since 1.13. This will encrypt `Secret` resources in etcd, preventing parties that gain access to your etcd backups from viewing the content of those secrets. While -this feature is currently experimental, it may offer an additional level of defense when backups +this feature is currently beta, it offers an additional level of defense when backups are not encrypted or an attacker gains read access to etcd. --> ### 对 Secret 进行静态加密 @@ -417,9 +418,9 @@ are not encrypted or an attacker gains read access to etcd. 并且可以授予攻击者对集群状态的可见性。 始终使用经过良好审查的备份和加密解决方案来加密备份，并考虑在可能的情况下使用全磁盘加密。 -Kubernetes 1.7 包含了[静态数据加密](/zh/docs/tasks/administer-cluster/encrypt-data/)， -它是一个 alpha 特性，会加密 etcd 里面的 `Secret` 资源，以防止某一方通过查看 -etcd 的备份文件查看到这些 Secret 的内容。虽然目前这还只是实验性的功能， +Kubernetes 支持 [静态数据加密](/zh/docs/tasks/administer-cluster/encrypt-data/)， +该功能在 1.7 版本引入，并在 1.13 版本成为 Beta。它会加密 etcd 里面的 `Secret` 资源，以防止某一方通过查看 +etcd 的备份文件查看到这些 Secret 的内容。虽然目前这还只是 Beta 阶段的功能， 但是在备份没有加密或者攻击者获取到 etcd 的读访问权限的时候，它能提供额外的防御层级。 - +{{< feature-state for_k8s_version="v1.21" state="stable" >}} +对一些步骤，你需要能够重新配置在你的集群里运行的 kubelet 命令行的选项。 @@ -272,6 +278,8 @@ to schedule those pods onto the right nodes. ## PodSecurityPolicy +{{< feature-state for_k8s_version="v1.21" state="deprecated" >}} + +{{< note >}} +为了将 Pod 规约中的 memory（和 hugepages）资源与所请求的其他资源对齐，需要启用内存管理器， +并且在节点配置适当的内存管理器策略。查看[内存管理器](/zh/docs/tasks/administer-cluster/memory-manager/) +文档。 +{{< /note >}} + ### 已知的局限性 1. 拓扑管理器所能处理的最大 NUMA 节点个数是 8。若 NUMA 节点数超过 8， 枚举可能的 NUMA 亲和性并为之生成提示时会发生状态爆炸。 -2. 调度器不支持拓扑功能，因此可能会由于拓扑管理器的原因而在节点上进行调度，然后在该节点上调度失败。 -3. 设备管理器和 CPU 管理器时能够采纳拓扑管理器 HintProvider 接口的唯一两个组件。 - 这意味着 NUMA 对齐只能针对 CPU 管理器和设备管理器所管理的资源实现。 - 内存和大页面在拓扑管理器决定 NUMA 对齐时都还不会被考虑在内。 +2. 调度器不是拓扑感知的，所以有可能一个 Pod 被调度到一个节点之后，会因为拓扑管理器的缘故在该节点上启动失败。 From baa9233f770618793d6ef36d278157d2a9106235 Mon Sep 17 00:00:00 2001 From: Juhee Kang Date: Sun, 26 Sep 2021 11:10:54 +0900 Subject: [PATCH 076/249] [ko] Update outdated files in dev-1.22-ko.1 (p2) --- .../container-runtimes.md | 34 +++++++++- .../tools/kubeadm/install-kubeadm.md | 9 ++- .../windows/intro-windows-in-kubernetes.md | 61 ++++++++++-------- .../windows/user-guide-windows-containers.md | 16 ++--- ...port-forward-access-application-cluster.md | 4 +- .../kubeadm/adding-windows-nodes.md | 10 +-- .../kubeadm/kubeadm-certs.md | 18 ++++-- .../cilium-network-policy.md | 63 ++++++++++++------- .../managing-secret-using-kubectl.md | 2 +- 9 files changed, 144 insertions(+), 73 deletions(-) diff --git a/content/ko/docs/setup/production-environment/container-runtimes.md b/content/ko/docs/setup/production-environment/container-runtimes.md index fb749a2346..18c8a9cbb1 100644 --- a/content/ko/docs/setup/production-environment/container-runtimes.md +++ b/content/ko/docs/setup/production-environment/container-runtimes.md @@ -26,7 +26,7 @@ weight: 20 다른 운영 체제의 경우, 해당 플랫폼과 관련된 문서를 찾아보자. {{< /note >}} -## Cgroup 드라이버 +## cgroup 드라이버 Control group은 프로세스에 할당된 리소스를 제한하는데 사용된다. @@ -57,6 +57,38 @@ kubelet을 재시작하는 것은 에러를 해결할 수 없을 것이다. 교체하거나, 자동화를 사용하여 다시 설치한다. {{< /caution >}} +## cgroup v2 + +cgroup v2는 cgroup Linux API의 다음 버전이다. +cgroup v1과는 다르게 각 컨트롤러마다 다른 계층 대신 단일 계층이 있다. + +새 버전은 cgroup v1에 비해 몇 가지 향상된 기능을 제공하며, 개선 사항 중 일부는 다음과 같다. + +- API를 더 쉽고 깔끔하게 사용할 수 있음 +- 컨테이너로의 안전한 하위 트리 위임 +- 압력 중지 정보와 같은 새로운 기능 + +일부 컨트롤러는 cgroup v1에 의해 관리되고 다른 컨트롤러는 cgroup v2에 의해 관리되는 하이브리드 구성을 지원하더라도, +쿠버네티스는 모든 컨트롤러를 관리하기 위해 +동일한 cgroup 버전만 지원한다. + +systemd가 기본적으로 cgroup v2를 사용하지 않는 경우, 커널 명령줄에 `systemd.unified_cgroup_hierarchy=1`을 +추가하여 cgroup v2를 사용하도록 시스템을 구성할 수 있다. + +```shell +# dnf install -y grubby && \ + sudo grubby \ + --update-kernel=ALL \ + --args=”systemd.unified_cgroup_hierarchy=1" +``` + +구성을 적용하려면 노드를 재부팅해야 한다. + +cgroup v2로 전환할 때 사용자가 노드 또는 컨테이너 내에서 +cgroup 파일 시스템에 직접 접근하지 않는 한 사용자 경험에 현저한 차이가 없어야 한다. + +cgroup v2를 사용하려면 CRI 런타임에서도 cgroup v2를 지원해야 한다. + ### kubeadm으로 생성한 클러스터의 드라이버를 `systemd`로 변경하기 kubeadm으로 생성한 클러스터의 cgroup 드라이버를 `systemd`로 변경하려면 diff --git a/content/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md b/content/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md index 6f50124f8d..3138aad1f8 100644 --- a/content/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md +++ b/content/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md @@ -239,8 +239,9 @@ CNI 플러그인 설치(대부분의 파드 네트워크에 필요) ```bash CNI_VERSION="v0.8.2" +ARCH="amd64" sudo mkdir -p /opt/cni/bin -curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz +curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz ``` 명령어 파일을 다운로드할 디렉터리 정의 @@ -259,15 +260,17 @@ crictl 설치(kubeadm / Kubelet 컨테이너 런타임 인터페이스(CRI)에 ```bash CRICTL_VERSION="v1.17.0" -curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz +ARCH="amd64" +curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz ``` `kubeadm`, `kubelet`, `kubectl` 설치 및 `kubelet` systemd 서비스 추가 ```bash RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)" +ARCH="amd64" cd $DOWNLOAD_DIR -sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl} +sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl} sudo chmod +x {kubeadm,kubelet,kubectl} RELEASE_VERSION="v0.4.0" diff --git a/content/ko/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md b/content/ko/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md index 67db901778..c4cc773ad9 100644 --- a/content/ko/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md +++ b/content/ko/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md @@ -67,10 +67,9 @@ weight: 65 | 쿠버네티스 버전 | 윈도우 서버 LTSC 릴리스 | 윈도우 서버 SAC 릴리스 | | --- | --- | --- | --- | -| *Kubernetes v1.19* | Windows Server 2019 | Windows Server ver 1909, Windows Server ver 2004 | | *Kubernetes v1.20* | Windows Server 2019 | Windows Server ver 1909, Windows Server ver 2004 | | *Kubernetes v1.21* | Windows Server 2019 | Windows Server ver 2004, Windows Server ver 20H2 | - +| *Kubernetes v1.22* | Windows Server 2019 | Windows Server ver 2004, Windows Server ver 20H2 | 지원 모델을 포함한 다양한 윈도우 서버 서비스 채널에 대한 정보는 @@ -98,12 +97,16 @@ weight: 65 일단 쿠버네티스에서 Hyper-V 격리가 포함된 윈도우 컨테이너를 지원하면, 제한 및 호환성 규칙이 변경될 것이다. -#### 퍼즈(Pause) 이미지 +#### 퍼즈(Pause) 이미지 {#pause-image} -Microsoft는 `mcr.microsoft.com/oss/kubernetes/pause:3.4.1`에서 -윈도우 퍼즈 인프라 컨테이너를 유지한다. -이외에도 `k8s.gcr.io/pause:3.5`를 통해 쿠버네티스에서 관리하는 다중 아키텍처 이미지를 -사용할 수도 있는데, 이 이미지는 리눅스와 윈도우를 모두 지원한다. +쿠버네티스는 윈도우 지원을 포함하는 다중 아키텍처 이미지를 유지보수한다. +쿠버네티스 v1.22의 경우 권장 퍼즈 이미지는 `k8s.gcr.io/pause:3.5`이다. +[소스 코드](https://github.com/kubernetes/kubernetes/tree/master/build/pause)는 +GitHub에서 찾을 수 있다. + +Microsoft는 리눅스, 윈도우 amd64를 지원하는 다중 아키텍처 이미지를 `mcr.microsoft.com/oss/kubernetes/pause:3.5`에서 유지보수하고 있다. +이 이미지는 쿠버네티스가 유지 관리하는 이미지와 동일한 소스코드에서 생성되었지만, 모든 윈도우 바이너리는 Microsoft에 의해 서명된 [인증 코드](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode)이다. +프로덕션 환경에서 서명된 바이너리가 필요한 경우, Microsoft가 유지 관리하는 이미지를 사용하는 것을 권장한다. #### 컴퓨트 @@ -237,7 +240,7 @@ powershell 스크립트로 배포된 다음의 FlexVolume ##### CSI 플러그인 -{{< feature-state for_k8s_version="v1.19" state="beta" >}} +{{< feature-state for_k8s_version="v1.22" state="stable" >}} {{< glossary_tooltip text="CSI" term_id="csi" >}} 플러그인과 관련된 코드는 일반적으로 컨테이너 이미지로 배포되고 데몬셋(DaemonSets) @@ -246,9 +249,7 @@ powershell 스크립트로 배포된 다음의 FlexVolume 바이너리로 제공된다. CSI 플러그인은 쿠버네티스에서 볼륨 프로비저닝/디-프로비저닝, 볼륨 크기 조정, 쿠버네티스 노드에 볼륨 연결/분리, 파드의 개별 컨테이너에 볼륨 마운트/분리, 스냅샷 및 복제를 사용하여 퍼시스턴트 데이터 백업/복원과 같은 -다양한 볼륨 관리 작업을 처리한다. CSI 플러그인은 -일반적으로 (각 노드에서 데몬셋으로 실행되는) 노드 플러그인과 컨트롤러 -플러그인으로 구성된다. +다양한 볼륨 관리 작업을 처리한다. CSI 노드 플러그인(특히 블록 디바이스 또는 공유 파일시스템으로 노출된 퍼시스턴트 볼륨과 관련된 플러그인)은 디스크 장치 스캔, 파일 시스템 마운트 등과 같은 @@ -262,6 +263,13 @@ CSI 노드 플러그인에 대한 특권이 필요한 작업은 커뮤니티에 내용은 배포하려는 CSI 플러그인의 배포 가이드를 참조한다. +윈도우 노드에서 CSI 노드 플러그인은 일반적으로 로컬 스토리지 작업을 처리하는 +커뮤니티에서 관리하는 [csi-proxy](https://github.com/kubernetes-csi/csi-proxy)에 의해 노출된 API를 호출한다. + +설치에 대한 자세한 내용은 윈도우 CSI 플러그인을 +배포할 환경의 배포 가이드를 참고한다. +또한 다음 [설치 단계](https://github.com/kubernetes-csi/csi-proxy#installation)를 참고할 수도 있다. + #### 네트워킹 윈도우 컨테이너용 네트워킹은 @@ -1068,9 +1076,8 @@ PodSecurityContext 필드는 윈도우에서 작동하지 않는다. 참조를 kubelet.exe를 등록한다. ```powershell - # Microsoft는 mcr.microsoft.com/oss/kubernetes/pause:1.4.1에서 pause 인프라 컨테이너를 릴리스했다. nssm install kubelet C:\k\kubelet.exe - nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.4.1 --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config + nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=k8s.gcr.io/pause:3.5 --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config nssm set kubelet AppDirectory C:\k nssm start kubelet ``` @@ -1224,13 +1231,13 @@ PodSecurityContext 필드는 윈도우에서 작동하지 않는다. 참조를 * 내 파드가 "Container Creating"에서 멈췄거나 계속해서 다시 시작된다. - pause 이미지가 OS 버전과 호환되는지 확인한다. + 퍼즈 이미지가 OS 버전과 호환되는지 확인한다. [지침](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources)에서는 OS와 컨테이너가 모두 버전 1803이라고 가정한다. 이후 버전의 윈도우가 있는 경우, Insider 빌드와 같이 그에 따라 이미지를 조정해야 한다. 이미지는 Microsoft의 [도커 리포지터리](https://hub.docker.com/u/microsoft/)를 참조한다. - 그럼에도 불구하고, pause 이미지 Dockerfile과 샘플 서비스는 이미지가 + 그럼에도 불구하고, 퍼즈 이미지 Dockerfile과 샘플 서비스는 이미지가 :latest로 태그될 것으로 예상한다. * DNS 확인(resolution)이 제대로 작동하지 않는다. @@ -1239,12 +1246,18 @@ PodSecurityContext 필드는 윈도우에서 작동하지 않는다. 참조를 * `kubectl port-forward`가 "unable to do port forwarding: wincat not found"로 실패한다. - 이는 쿠버네티스 1.15 및 pause 인프라 컨테이너 + 이는 쿠버네티스 1.15 및 퍼즈 인프라 컨테이너 `mcr.microsoft.com/oss/kubernetes/pause:1.4.1`에서 구현되었다. - 해당 버전 또는 최신 버전을 사용해야 한다. 자체 pause + 해당 버전 또는 최신 버전을 사용해야 한다. 자체 퍼즈 인프라 컨테이너를 빌드하려면 [wincat](https://github.com/kubernetes-sigs/sig-windows-tools/tree/master/cmd/wincat)을 포함해야 한다. + 윈도우에서 포트 포워딩을 지원하려면 [퍼즈 인프라 컨테이너](#pause-image)를 이용하기 위해서 + wincat.exe가 필요하다. + 윈도우 OS 버전과 호환되는 지원되는 이미지를 사용하고 있는지 확인해야 한다. + 자신만의 퍼즈 인프라 컨테이너를 구축하려면 + [wincat](https://github.com/kubernetes/kubernetes/tree/master/build/pause/windows/wincat)을 포함해야 한다. + * 내 윈도우 서버 노드가 프록시 뒤에 있기 때문에 내 쿠버네티스 설치가 실패한다. @@ -1256,20 +1269,18 @@ PodSecurityContext 필드는 윈도우에서 작동하지 않는다. 참조를 [Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.example.com:443/", [EnvironmentVariableTarget]::Machine) ``` -* `pause` 컨테이너란 무엇인가? +* 퍼즈(`pause`) 컨테이너란 무엇인가? 쿠버네티스 파드에서는 컨테이너 엔드포인트를 호스팅하기 위해 - 먼저 인프라 또는 "pause" 컨테이너가 생성된다. 인프라 및 워커 컨테이너를 포함하여 + 먼저 인프라 또는 "퍼즈" 컨테이너가 생성된다. 인프라 및 워커 컨테이너를 포함하여 동일한 파드에 속하는 컨테이너는 공통 네트워크 네임스페이스 및 엔드포인트(동일한 IP 및 포트 공간)를 공유한다. 네트워크 구성을 잃지 않고 - 워커 컨테이너가 충돌하거나 다시 시작되도록 하려면 pause 컨테이너가 + 워커 컨테이너가 충돌하거나 다시 시작되도록 하려면 퍼즈 컨테이너가 필요하다. - "pause" (인프라) 이미지는 Microsoft Container Registry(MCR)에서 - 호스팅된다. `mcr.microsoft.com/oss/kubernetes/pause:1.4.1`을 사용하여 접근할 수 있다. - 자세한 내용은 - [DOCKERFILE](https://github.com/kubernetes-sigs/windows-testing/blob/master/images/pause/Dockerfile)을 참고한다. - + 퍼즈 이미지 추천 버전을 찾기 위해서는 + [퍼즈 이미지](#pause-image)를 참고한다. + ### 추가 조사 이러한 단계로 문제가 해결되지 않으면, 다음을 통해 쿠버네티스의 윈도우 노드에서 diff --git a/content/ko/docs/setup/production-environment/windows/user-guide-windows-containers.md b/content/ko/docs/setup/production-environment/windows/user-guide-windows-containers.md index 5c3d52e475..2e1694834e 100644 --- a/content/ko/docs/setup/production-environment/windows/user-guide-windows-containers.md +++ b/content/ko/docs/setup/production-environment/windows/user-guide-windows-containers.md @@ -20,8 +20,8 @@ weight: 75 ## 시작하기 전에 -* [윈도우 서버에서 운영하는 마스터와 워커 노드](/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes)를 -포함한 쿠버네티스 클러스터를 생성한다. +* 컨트롤 플레인과 [윈도우 서버로 운영되는 워커 노드](/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)를 +포함하는 쿠버네티스 클러스터를 생성한다. * 쿠버네티스에서 서비스와 워크로드를 생성하고 배포하는 것은 리눅스나 윈도우 컨테이너 모두 비슷한 방식이라는 것이 중요하다. [Kubectl 커맨드](/ko/docs/reference/kubectl/overview/)로 클러스터에 접속하는 것은 동일하다. @@ -100,15 +100,15 @@ spec: 1. 이 디플로이먼트가 성공적인지 확인한다. 다음을 검토하자. * 윈도우 노드에 파드당 두 컨테이너가 존재하는지 확인하려면, `docker ps`를 사용한다. - * 리눅스 마스터에서 나열된 두 파드가 존재하는지 확인하려면, `kubectl get pods`를 사용한다. - * 네트워크를 통한 노드에서 파드로의 통신이 되는지 확인하려면, 리눅스 마스터에서 `curl`을 + * 리눅스 컨트롤 플레인 노드에서 나열된 두 파드가 존재하는지 확인하려면, `kubectl get pods`를 사용한다. + * 네트워크를 통한 노드에서 파드로의 통신이 되는지 확인하려면, 리눅스 컨트롤 플레인 노드에서 `curl`을 파드 IP 주소의 80 포트로 실행하여 웹 서버 응답을 확인한다. * 파드 간 통신이 되는지 확인하려면, `docker exec` 나 `kubectl exec`를 이용해 파드 간에 핑(ping)한다(윈도우 노드가 2대 이상이라면, 서로 다른 노드에 있는 파드 간 통신도 확인할 수 있다). - * 서비스에서 파드로의 통신이 되는지 확인하려면, 리눅스 마스터와 독립 파드에서 `curl`을 가상 서비스 + * 서비스에서 파드로의 통신이 되는지 확인하려면, 리눅스 컨트롤 플레인 노드와 독립 파드에서 `curl`을 가상 서비스 IP 주소(`kubectl get services`로 볼 수 있는)로 실행한다. * 서비스 검색(discovery)이 되는지 확인하려면, 쿠버네티스 [기본 DNS 접미사](/ko/docs/concepts/services-networking/dns-pod-service/#서비스)와 서비스 이름으로 `curl`을 실행한다. - * 인바운드 연결이 되는지 확인하려면, 클러스터 외부 장비나 리눅스 마스터에서 NodePort로 `curl`을 실행한다. + * 인바운드 연결이 되는지 확인하려면, 클러스터 외부 장비나 리눅스 컨트롤 플레인 노드에서 NodePort로 `curl`을 실행한다. * 아웃바운드 연결이 되는지 확인하려면, `kubectl exec`를 이용해서 파드에서 외부 IP 주소로 `curl`을 실행한다. {{< note >}} @@ -178,8 +178,8 @@ GMSA로 구성한 컨테이너는 GMSA로 구성된 신원을 들고 있는 동 예를 들면, `--register-with-taints='os=windows:NoSchedule'` 모든 윈도우 노드에 테인트를 추가하여 아무 것도 거기에 스케줄링하지 않게 될 것이다(존재하는 리눅스 파드를 포함하여). -윈도우 파드가 윈도우 노드에 스케줄링되려면, -윈도우를 선택하기 위한 노드 셀렉터 및 적합하게 일치하는 톨러레이션이 모두 필요하다. +윈도우 파드가 윈도우 노드에 스케줄링되도록 하려면, +윈도우 노드가 선택되도록 하기 위한 노드 셀렉터 및 적합하게 일치하는 톨러레이션이 모두 필요하다. ```yaml nodeSelector: diff --git a/content/ko/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md b/content/ko/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md index 333001af81..6aa257dca4 100644 --- a/content/ko/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md +++ b/content/ko/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md @@ -31,7 +31,7 @@ min-kubernetes-server-version: v1.10 1. MongoDB를 실행하기 위해 디플로이먼트를 생성한다. ```shell - kubectl apply -f https://k8s.io/examples/application/guestbook/mongo-deployment.yaml + kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-deployment.yaml ``` 성공적인 명령어의 출력은 디플로이먼트가 생성됐다는 것을 확인해준다. @@ -84,7 +84,7 @@ min-kubernetes-server-version: v1.10 2. MongoDB를 네트워크에 노출시키기 위해 서비스를 생성한다. ```shell - kubectl apply -f https://k8s.io/examples/application/guestbook/mongo-service.yaml + kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-service.yaml ``` 성공적인 커맨드의 출력은 서비스가 생성되었다는 것을 확인해준다. diff --git a/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md b/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md index e67a08a74e..d70669a93e 100644 --- a/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md +++ b/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md @@ -157,10 +157,10 @@ Install-WindowsFeature -Name containers #### wins, kubelet 및 kubeadm 설치 - ```PowerShell - curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1 - .\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} - ``` +```PowerShell +curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1 +.\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} +``` #### `kubeadm` 실행하여 노드에 조인 @@ -201,7 +201,7 @@ curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/lates #### wins, kubelet 및 kubeadm 설치 ```PowerShell -curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1 +curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1 .\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} -ContainerRuntime containerD ``` diff --git a/content/ko/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/ko/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index de6feb480d..fe34d7a7f7 100644 --- a/content/ko/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/ko/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -126,7 +126,17 @@ kubeadm 1.17 이전 버전에는 `kubeadm upgrade node` 명령에서 `kubeadm certs renew` 명령을 사용하여 언제든지 인증서를 수동으로 갱신할 수 있다. -이 명령은 `/etc/kubernetes/pki` 에 저장된 CA(또는 프론트 프록시 CA) 인증서와 키를 사용하여 갱신을 수행한다. +이 명령은 `/etc/kubernetes/pki` 에 저장된 CA(또는 프론트 프록시 CA) 인증서와 키를 사용하여 갱신을 수행한다. + +명령을 실행한 후에는 컨트롤 플레인 파드를 재시작해야 한다. +이는 현재 일부 구성 요소 및 인증서에 대해 인증서를 동적으로 다시 로드하는 것이 지원되지 않기 때문이다. +[스태틱(static) 파드](/ko/docs/tasks/configure-pod-container/static-pod/)는 API 서버가 아닌 로컬 kubelet에서 관리되므로 +kubectl을 사용하여 삭제 및 재시작할 수 없다. +스태틱 파드를 다시 시작하려면 `/etc/kubernetes/manifests/`에서 매니페스트 파일을 일시적으로 제거하고 +20초를 기다리면 된다 ([KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/)의 `fileCheckFrequency` 값을 참고한다). +파드가 매니페스트 디렉터리에 더 이상 없는 경우 kubelet은 파드를 종료한다. +그런 다음 파일을 다시 이동할 수 있으며 또 다른 `fileCheckFrequency` 기간이 지나면, +kubelet은 파드를 생성하고 구성 요소에 대한 인증서 갱신을 완료할 수 있다. {{< warning >}} HA 클러스터를 실행 중인 경우, 모든 컨트롤 플레인 노드에서 이 명령을 실행해야 한다. @@ -161,10 +171,10 @@ HA 클러스터를 실행 중인 경우, 모든 컨트롤 플레인 노드에서 빌트인 서명자를 활성화하려면, `--cluster-signing-cert-file` 와 `--cluster-signing-key-file` 플래그를 전달해야 한다. -새 클러스터를 생성하는 경우, kubeadm [구성 파일](/docs/reference/config-api/kubeadm-config.v1beta2/)을 사용할 수 있다. +새 클러스터를 생성하는 경우, kubeadm [구성 파일](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3)을 사용할 수 있다. ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration controllerManager: extraArgs: @@ -223,7 +233,7 @@ TLS로 보안되지 않음을 의미한다. 다음의 최소 구성을 `kubeadm init` 에 전달해야 한다. ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration --- apiVersion: kubelet.config.k8s.io/v1beta1 diff --git a/content/ko/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md b/content/ko/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md index eb953143be..4a6a769556 100644 --- a/content/ko/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md +++ b/content/ko/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy.md @@ -22,44 +22,59 @@ weight: 20 실리움에 쉽게 친숙해지기 위해 Minikube에 실리움을 기본적인 데몬셋으로 설치를 수행하는 -[실리움 쿠버네티스 시작하기 안내](https://docs.cilium.io/en/stable/gettingstarted/minikube/)를 따라 해볼 수 있다. +[실리움 쿠버네티스 시작하기 안내](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/)를 따라 해볼 수 있다. -Minikube를 시작하려면 최소 버전으로 >= v1.3.1 이 필요하고, +Minikube를 시작하려면 최소 버전으로 >= v1.5.2 이 필요하고, 다음의 실행 파라미터로 실행한다. ```shell minikube version ``` ``` -minikube version: v1.3.1 +minikube version: v1.5.2 ``` ```shell -minikube start --network-plugin=cni --memory=4096 +minikube start --network-plugin=cni ``` -BPF 파일시스템을 마운트한다 +minikube의 경우 CLI 도구를 사용하여 실리움을 설치할 수 있다. +실리움은 클러스터 구성을 자동으로 감지하고 +성공적인 설치를 위해 적절한 구성 요소를 설치한다. ```shell -minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf +curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz +sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin +rm cilium-linux-amd64.tar.gz +cilium install ``` -Minikube에서 실리움의 데몬셋 구성과 적절한 RBAC 설정을 포함하는 필요한 구성을 -간단한 ``올인원`` YAML 파일로 배포할 수 있다. - ```shell -kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.8/install/kubernetes/quick-install.yaml -``` -``` -configmap/cilium-config created -serviceaccount/cilium created -serviceaccount/cilium-operator created -clusterrole.rbac.authorization.k8s.io/cilium created -clusterrole.rbac.authorization.k8s.io/cilium-operator created -clusterrolebinding.rbac.authorization.k8s.io/cilium created -clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created -daemonset.apps/cilium create -deployment.apps/cilium-operator created +🔮 Auto-detected Kubernetes kind: minikube +✨ Running "minikube" validation checks +✅ Detected minikube version "1.20.0" +ℹ️ Cilium version not set, using default version "v1.10.0" +🔮 Auto-detected cluster name: minikube +🔮 Auto-detected IPAM mode: cluster-pool +🔮 Auto-detected datapath mode: tunnel +🔑 Generating CA... +2021/05/27 02:54:44 [INFO] generate received request +2021/05/27 02:54:44 [INFO] received CSR +2021/05/27 02:54:44 [INFO] generating key: ecdsa-256 +2021/05/27 02:54:44 [INFO] encoded CSR +2021/05/27 02:54:44 [INFO] signed certificate with serial number 48713764918856674401136471229482703021230538642 +🔑 Generating certificates for Hubble... +2021/05/27 02:54:44 [INFO] generate received request +2021/05/27 02:54:44 [INFO] received CSR +2021/05/27 02:54:44 [INFO] generating key: ecdsa-256 +2021/05/27 02:54:44 [INFO] encoded CSR +2021/05/27 02:54:44 [INFO] signed certificate with serial number 3514109734025784310086389188421560613333279574 +🚀 Creating Service accounts... +🚀 Creating Cluster roles... +🚀 Creating ConfigMap... +🚀 Creating Agent DaemonSet... +🚀 Creating Operator Deployment... +⌛ Waiting for Cilium to be installed... ``` 시작하기 안내서의 나머지 부분은 예제 애플리케이션을 이용하여 @@ -82,14 +97,14 @@ L3/L4(예, IP 주소 + 포트) 모두의 보안 정책뿐만 아니라 L7(예, H 파드의 목록을 보려면 다음을 실행한다. ```shell -kubectl get pods --namespace=kube-system +kubectl get pods --namespace=kube-system -l k8s-app=cilium ``` 다음과 유사한 파드의 목록을 볼 것이다. ```console -NAME READY STATUS RESTARTS AGE -cilium-6rxbd 1/1 Running 0 1m +NAME READY STATUS RESTARTS AGE +cilium-kkdhz 1/1 Running 0 3m23s ... ``` diff --git a/content/ko/docs/tasks/configmap-secret/managing-secret-using-kubectl.md b/content/ko/docs/tasks/configmap-secret/managing-secret-using-kubectl.md index 8b3f62217e..7a5e502c47 100644 --- a/content/ko/docs/tasks/configmap-secret/managing-secret-using-kubectl.md +++ b/content/ko/docs/tasks/configmap-secret/managing-secret-using-kubectl.md @@ -67,7 +67,7 @@ kubectl create secret generic db-user-pass \ 다음 커맨드를 실행한다. ```shell -kubectl create secret generic dev-db-secret \ +kubectl create secret generic db-user-pass \ --from-literal=username=devuser \ --from-literal=password='S!B\*d$zDsb=' ``` From 6d61d43ab084f1b14a5060575b77f07106c649e8 Mon Sep 17 00:00:00 2001 From: zhiguo-lu Date: Sun, 26 Sep 2021 17:56:02 +0800 Subject: [PATCH 077/249] [zh] translate glossary/Garbage Collection --- .../reference/glossary/garbage-collection.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 content/zh/docs/reference/glossary/garbage-collection.md diff --git a/content/zh/docs/reference/glossary/garbage-collection.md b/content/zh/docs/reference/glossary/garbage-collection.md new file mode 100644 index 0000000000..effc51483c --- /dev/null +++ b/content/zh/docs/reference/glossary/garbage-collection.md @@ -0,0 +1,49 @@ +--- +title: 垃圾收集 +id: garbage-collection +date: 2021-07-07 +full_link: /zh/docs/concepts/workloads/controllers/garbage-collection/ +short_description: > + Kubernetes 用于清理集群资源的各种机制的统称。 + +aka: +tags: +- fundamental +- operation +--- + + + +垃圾收集是 Kubernetes 用于清理集群资源的各种机制的统称。 + + + + +Kubernetes 使用垃圾收集机制来清理资源，例如： +[未使用的容器和镜像](/zh/docs/concepts/workloads/controllers/garbage-collection/#containers-images)、 +[失败的 Pod](/zh/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection)、 +[目标资源拥有的对象](/zh/docs/concepts/overview/working-with-objects/owners-dependents/)、 +[已完成的 Job](/zh/docs/concepts/workloads/controllers/ttlafterfinished/)、 +过期或出错的资源。 \ No newline at end of file From 6cdf4202f036ea51d185f3f9703cbc7be8c4c158 Mon Sep 17 00:00:00 2001 From: Juhee Kang Date: Mon, 27 Sep 2021 15:51:06 +0900 Subject: [PATCH 078/249] [ko] Update outdated files in dev-1.22-ko.1 (p3) --- .../docs/reference/glossary/api-eviction.md | 2 +- content/ko/docs/reference/kubectl/kubectl.md | 24 ++++ content/ko/docs/reference/kubectl/overview.md | 2 +- .../ko/docs/reference/scheduling/config.md | 120 ++++++++++-------- .../ko/docs/reference/scheduling/policies.md | 2 +- content/ko/docs/reference/tools/_index.md | 3 +- .../docs/reference/using-api/health-checks.md | 2 +- .../setup/best-practices/cluster-large.md | 3 + 8 files changed, 97 insertions(+), 61 deletions(-) diff --git a/content/ko/docs/reference/glossary/api-eviction.md b/content/ko/docs/reference/glossary/api-eviction.md index f8a65c606e..7d5ee60667 100644 --- a/content/ko/docs/reference/glossary/api-eviction.md +++ b/content/ko/docs/reference/glossary/api-eviction.md @@ -2,7 +2,7 @@ title: API를 이용한 축출(Eviction) id: api-eviction date: 2021-04-27 -full_link: /docs/concepts/scheduling-eviction/pod-eviction/#api-eviction +full_link: /ko/docs/concepts/scheduling-eviction/api-eviction/ short_description: > API를 이용한 축출은 축출 API를 사용하여 파드의 정상 종료를 트리거하는 축출 오브젝트를 만드는 프로세스이다 diff --git a/content/ko/docs/reference/kubectl/kubectl.md b/content/ko/docs/reference/kubectl/kubectl.md index 81e4d3fa74..b2a16bf452 100644 --- a/content/ko/docs/reference/kubectl/kubectl.md +++ b/content/ko/docs/reference/kubectl/kubectl.md @@ -328,7 +328,31 @@ kubectl [flags]

    +## {{% heading "envvars" %}} + ++++ + + + + + + + + + + + + + + + + + +
    KUBECONFIG
    kubectl 구성 ("kubeconfig") 파일 경로. 기본: "$HOME/.kube/config"
    KUBECTL_COMMAND_HEADERS
    false로 설정하면, 호출된 kubectl 명령(쿠버네티스 버전 v1.22 이상)을 자세히 설명하는 추가 HTTP 헤더를 해제
    ## {{% heading "seealso" %}} diff --git a/content/ko/docs/reference/kubectl/overview.md b/content/ko/docs/reference/kubectl/overview.md index 3d8179a08a..247489a6ee 100644 --- a/content/ko/docs/reference/kubectl/overview.md +++ b/content/ko/docs/reference/kubectl/overview.md @@ -87,7 +87,7 @@ kubectl [command] [TYPE] [NAME] [flags] `cluster-info` | `kubectl cluster-info [flags]` | 클러스터의 마스터와 서비스에 대한 엔드포인트 정보를 표시한다. `completion` | `kubectl completion SHELL [options]` | 지정된 셸(bash 또는 zsh)에 대한 셸 완성 코드를 출력한다. `config` | `kubectl config SUBCOMMAND [flags]` | kubeconfig 파일을 수정한다. 세부 사항은 개별 하위 명령을 참고한다. -`convert` | `kubectl convert -f FILENAME [options]` | 다른 API 버전 간에 구성 파일을 변환한다. YAML 및 JSON 형식이 모두 허용된다. +`convert` | `kubectl convert -f FILENAME [options]` | 다른 API 버전 간에 구성 파일을 변환한다. YAML 및 JSON 형식이 모두 허용된다. 참고 - `kubectl-convert` 플러그인을 설치해야 한다. `cordon` | `kubectl cordon NODE [options]` | 노드를 스케줄 불가능(unschedulable)으로 표시한다. `cp` | `kubectl cp [options]` | 컨테이너에서 그리고 컨테이너로 파일 및 디렉터리를 복사한다. `create` | `kubectl create -f FILENAME [flags]` | 파일이나 표준입력에서 하나 이상의 리소스를 생성한다. diff --git a/content/ko/docs/reference/scheduling/config.md b/content/ko/docs/reference/scheduling/config.md index 2f46c78d8b..07f06863d4 100644 --- a/content/ko/docs/reference/scheduling/config.md +++ b/content/ko/docs/reference/scheduling/config.md @@ -18,7 +18,8 @@ weight: 20 각 단계는 익스텐션 포인트(extension point)를 통해 노출된다. 플러그인은 이러한 익스텐션 포인트 중 하나 이상을 구현하여 스케줄링 동작을 제공한다. -[KubeSchedulerConfiguration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) +KubeSchedulerConfiguration ([v1beta1](/docs/reference/config-api/kube-scheduler-config.v1beta1/) +또는 [v1beta2](/docs/reference/config-api/kube-scheduler-config.v1beta2/)) 구조에 맞게 파일을 작성하고, `kube-scheduler --config `을 실행하여 스케줄링 프로파일을 지정할 수 있다. @@ -26,7 +27,7 @@ weight: 20 최소 구성은 다음과 같다. ```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 +apiVersion: kubescheduler.config.k8s.io/v1beta2 kind: KubeSchedulerConfiguration clientConnection: kubeconfig: /etc/srv/kubernetes/kube-scheduler/kubeconfig @@ -48,38 +49,41 @@ clientConnection: 스케줄링은 다음 익스텐션 포인트를 통해 노출되는 일련의 단계에서 발생한다. -1. `QueueSort`: 이 플러그인은 스케줄링 대기열에서 보류 중인 파드를 +1. `queueSort`: 이 플러그인은 스케줄링 대기열에서 보류 중인 파드를 정렬하는 데 사용되는 정렬 기능을 제공한다. 대기열 정렬 플러그인은 한 번에 단 하나만 활성화될 수 있다. 사용할 수 있다. -1. `PreFilter`: 이 플러그인은 필터링하기 전에 파드 또는 클러스터에 대한 정보를 +1. `preFilter`: 이 플러그인은 필터링하기 전에 파드 또는 클러스터에 대한 정보를 사전 처리하거나 확인하는 데 사용된다. 이 플러그인은 파드를 unschedulable로 표시할 수 있다. -1. `Filter`: 이 플러그인은 스케줄링 정책의 단정(Predicates)과 동일하며 +1. `filter`: 이 플러그인은 스케줄링 정책의 단정(Predicates)과 동일하며 파드를 실행할 수 없는 노드를 필터링하는 데 사용된다. 필터는 구성된 순서대로 호출된다. 노드가 모든 필터를 통과하지 않으면 파드는 unschedulable로 표시된다. -1. `PreScore`: 이것은 사전 스코어링 작업을 수행하는 데 사용할 수 있는 +1. `postFilter`: 이 플러그인은 파드의 실행 가능한 노드를 찾을 수 없을 때, + 구성된 순서대로 호출된다. `postFilter` 플러그인이 파드 _schedulable_ 을 표시하는 경우, + 나머지 플러그인은 호출 되지 않는다. +1. `preScore`: 이것은 사전 스코어링 작업을 수행하는 데 사용할 수 있는 정보성 익스텐션 포인트이다. -1. `Score`: 이 플러그인은 필터링 단계를 통과한 각 노드에 점수를 +1. `score`: 이 플러그인은 필터링 단계를 통과한 각 노드에 점수를 제공한다. 그런 다음 스케줄러는 가중치 합계가 가장 높은 노드를 선택한다. -1. `Reserve`: 지정된 파드에 리소스가 예약된 경우 플러그인에 +1. `reserve`: 지정된 파드에 리소스가 예약된 경우 플러그인에 알리는 정보성 익스텐션 포인트이다. 플러그인은 또한 `Reserve` 도중 또는 이후에 실패한 경우 호출 되는 `Unreserve` 호출을 구현한다. -1. `Permit`: 이 플러그인은 파드 바인딩을 방지하거나 지연시킬 수 있다. -1. `PreBind`: 이 플러그인은 파드가 바인딩되기 전에 필요한 모든 작업을 수행한다. -1. `Bind`: 플러그인은 파드를 노드에 바인딩한다. Bind 플러그인은 순서대로 호출되며 - 일단 바인딩이 완료되면 나머지 플러그인은 건너뛴다. Bind +1. `permit`: 이 플러그인은 파드 바인딩을 방지하거나 지연시킬 수 있다. +1. `preBind`: 이 플러그인은 파드가 바인딩되기 전에 필요한 모든 작업을 수행한다. +1. `bind`: 플러그인은 파드를 노드에 바인딩한다. `bind` 플러그인은 순서대로 호출되며 + 일단 바인딩이 완료되면 나머지 플러그인은 건너뛴다. bind 플러그인은 적어도 하나 이상 필요하다. -1. `PostBind`: 파드가 바인드된 후 호출되는 +1. `postBind`: 파드가 바인드된 후 호출되는 정보성 익스텐션 포인트이다. 각 익스텐션 포인트에 대해 특정 [기본 플러그인](#스케줄링-플러그인)을 비활성화하거나 자체 플러그인을 활성화할 수 있다. 예를 들면, 다음과 같다. ```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 +apiVersion: kubescheduler.config.k8s.io/v1beta2 kind: KubeSchedulerConfiguration profiles: - plugins: @@ -99,98 +103,100 @@ profiles: ### 스케줄링 플러그인 -1. `UnReserve`: 파드가 예약된 후 거부되고 `Permit` 플러그인에 의해 보류된 경우 - 호출되는 정보성 익스텐션 포인트이다. - -## 스케줄링 플러그인 - 기본적으로 활성화된 다음의 플러그인은 이들 익스텐션 포인트 중 하나 이상을 구현한다. -- `SelectorSpread`: {{< glossary_tooltip text="서비스" term_id="service" >}}, - {{< glossary_tooltip text="레플리카셋(ReplicaSets)" term_id="replica-set" >}} 및 - {{< glossary_tooltip text="스테이트풀셋(StatefulSets)" term_id="statefulset" >}}에 - 속하는 파드에 대해 노드 간 분산을 선호한다. - 익스텐션 포인트: `PreScore`, `Score`. - `ImageLocality`: 파드가 실행하는 컨테이너 이미지가 이미 있는 노드를 선호한다. - 익스텐션 포인트: `Score`. + 익스텐션 포인트: `score`. - `TaintToleration`: [테인트(taint)와 톨러레이션(toleration)](/ko/docs/concepts/scheduling-eviction/taint-and-toleration/)을 구현한다. - 익스텐션 포인트 구현: `Filter`, `Prescore`, `Score`. + 익스텐션 포인트 구현: `filter`, `preScore`, `score`. - `NodeName`: 파드 명세 노드 이름이 현재 노드와 일치하는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `NodePorts`: 노드에 요청된 파드 포트에 대해 사용 가능한 포트가 있는지 확인한다. - 익스텐션 포인트: `PreFilter`, `Filter`. + 익스텐션 포인트: `preFilter`, `filter`. - `NodePreferAvoidPods`: 노드 {{< glossary_tooltip text="어노테이션" term_id="annotation" >}} `scheduler.alpha.kubernetes.io/preferAvoidPods` 에 따라 노드 점수를 매긴다. - 익스텐션 포인트: `Score`. + 익스텐션 포인트: `score`. - `NodeAffinity`: [노드 셀렉터](/ko/docs/concepts/scheduling-eviction/assign-pod-node/#노드-셀렉터-nodeselector)와 [노드 어피니티](/ko/docs/concepts/scheduling-eviction/assign-pod-node/#노드-어피니티)를 구현한다. - 익스텐션 포인트: `Filter`, `Score`. + 익스텐션 포인트: `filter`, `score`. - `PodTopologySpread`: [파드 토폴로지 분배](/ko/docs/concepts/workloads/pods/pod-topology-spread-constraints/)를 구현한다. - 익스텐션 포인트: `PreFilter`, `Filter`, `PreScore`, `Score`. + 익스텐션 포인트: `preFilter`, `filter`, `preScore`, `score`. - `NodeUnschedulable`: `.spec.unschedulable` 이 true로 설정된 노드를 필터링한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `NodeResourcesFit`: 노드에 파드가 요청하는 모든 리소스가 있는지 - 확인한다. - 익스텐션 포인트: `PreFilter`, `Filter`. + 확인한다. 점수는 `LeastAllocated`(기본값), `MostAllocated`, `RequestedToCapacityRatio` 등 3가지 전략 중 하나를 사용할 수 있다. + 익스텐션 포인트: `preFilter`, `filter`, `score`. - `NodeResourcesBalancedAllocation`: 파드가 스케줄된 경우, 보다 균형잡힌 리소스 사용량을 얻을 수 있는 노드를 선호한다. - 익스텐션 포인트: `Score`. + 익스텐션 포인트: `score`. - `NodeResourcesLeastAllocated`: 리소스 할당이 적은 노드를 선호한다. 익스텐션 포인트: `Score`. - `VolumeBinding`: 노드에 요청된 {{< glossary_tooltip text="볼륨" term_id="volume" >}}이 있는지 또는 바인딩할 수 있는지 확인한다. - 익스텐션 포인트: `PreFilter`, `Filter`, `Reserve`, `PreBind`, `Score`. + 익스텐션 포인트: `preFilter`, `filter`, `reserve`, `preBind`, `score`. {{< note >}} - `Score` 익스텐션 포인트는 `VolumeCapacityPriority` 기능이 + `score` 익스텐션 포인트는 `VolumeCapacityPriority` 기능이 활성화되어 있어야 활성화되며, 요청된 볼륨 사이즈를 만족하는 가장 작은 PV들을 우선순위 매긴다. {{< /note >}} - `VolumeRestrictions`: 노드에 마운트된 볼륨이 볼륨 제공자에 특정한 제한 사항을 충족하는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `VolumeZone`: 요청된 볼륨이 가질 수 있는 영역 요구 사항을 충족하는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `NodeVolumeLimits`: 노드에 대해 CSI 볼륨 제한을 충족할 수 있는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `EBSLimits`: 노드에 대해 AWS EBS 볼륨 제한을 충족할 수 있는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `GCEPDLimits`: 노드에 대해 GCP-PD 볼륨 제한을 충족할 수 있는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `AzureDiskLimits`: 노드에 대해 Azure 디스크 볼륨 제한을 충족할 수 있는지 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `filter`. - `InterPodAffinity`: [파드 간 어피니티 및 안티-어피니티](/ko/docs/concepts/scheduling-eviction/assign-pod-node/#파드간-어피니티와-안티-어피니티)를 구현한다. - 익스텐션 포인트: `PreFilter`, `Filter`, `PreScore`, `Score`. + 익스텐션 포인트: `preFilter`, `filter`, `preScore`, `score`. - `PrioritySort`: 기본 우선 순위 기반 정렬을 제공한다. - 익스텐션 포인트: `QueueSort`. + 익스텐션 포인트: `queueSort`. - `DefaultBinder`: 기본 바인딩 메커니즘을 제공한다. - 익스텐션 포인트: `Bind`. + 익스텐션 포인트: `bind`. - `DefaultPreemption`: 기본 선점 메커니즘을 제공한다. - 익스텐션 포인트: `PostFilter`. + 익스텐션 포인트: `postFilter`. 기본으로 활성화되지 않는 다음의 플러그인을 컴포넌트 구성 API를 통해 활성화할 수도 있다. +- `SelectorSpread`: {{< glossary_tooltip text="Services" term_id="service" >}}, + {{< glossary_tooltip text="ReplicaSets" term_id="replica-set" >}}와 + {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}}에 속하는 파드의 경우, + 노드간에 퍼지는 것을 선호한다. + 익스텐션 포인트: `preScore`, `score`. +- `CinderLimits`: 노드에 대해 [OpenStack Cinder](https://docs.openstack.org/cinder/) + 볼륨 제한이 충족될 수 있는지 확인한다. + 익스텐션 포인트: `filter`. + +다음 플러그인은 더 이상 사용되지 않으며 `v1beta1`에서만 +사용할 수 있다. + +- `NodeResourcesLeastAllocated`: 리소스 할당이 낮은 노드를 + 선호한다. + Extension points: `score`. - `NodeResourcesMostAllocated`: 리소스 할당이 많은 노드를 선호한다. - 익스텐션 포인트: `Score`. + 익스텐션 포인트: `score`. - `RequestedToCapacityRatio`: 할당된 리소스의 구성된 기능에 따라 노드를 선호한다. - 익스텐션 포인트: `Score`. -- `CinderVolume`: 노드에 대해 OpenStack Cinder 볼륨 제한을 충족할 수 있는지 - 확인한다. - 익스텐션 포인트: `Filter`. + 익스텐션 포인트: `score`. - `NodeLabel`: Filters and / or scores a node according to configured {{< glossary_tooltip text="label(s)" term_id="label" >}}. 익스텐션 포인트: `Filter`, `Score`. @@ -198,7 +204,10 @@ profiles: 속한 파드가 구성된 레이블로 정의된 노드 집합에 맞는지 확인한다. 이 플러그인은 또한 서비스에 속한 파드를 노드 간에 분산하는 것을 선호한다. - 익스텐션 포인트: `PreFilter`, `Filter`, `Score`. + 익스텐션 포인트: `preFilter`, `filter`, `score`. +- `NodePreferAvoidPods`: 노드 주석 `scheduler.alpha.kubernetes.io/preferAvoidPods`에 따라 + 노드의 우선 순위를 지정한다. + 익스텐션 포인트: `score`. ### 여러 프로파일 @@ -211,7 +220,7 @@ profiles: 실행된다. ```yaml -apiVersion: kubescheduler.config.k8s.io/v1beta1 +apiVersion: kubescheduler.config.k8s.io/v1beta2 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler @@ -243,7 +252,7 @@ profiles: {{< /note >}} {{< note >}} -모든 프로파일은 QueueSort 익스텐션 포인트에서 동일한 플러그인을 사용해야 하며 +모든 프로파일은 `queueSort` 익스텐션 포인트에서 동일한 플러그인을 사용해야 하며 동일한 구성 파라미터(해당하는 경우)를 가져야 한다. 그 이유는 스케줄러가 보류 중 상태인 파드 대기열을 단 하나만 가질 수 있기 때문이다. {{< /note >}} @@ -253,3 +262,4 @@ profiles: * [kube-scheduler 레퍼런스](/docs/reference/command-line-tools-reference/kube-scheduler/) 읽어보기 * [스케줄링](/ko/docs/concepts/scheduling-eviction/kube-scheduler/)에 대해 알아보기 * [kube-scheduler configuration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) 레퍼런스 읽어보기 +* [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) 레퍼런스 읽어보기 \ No newline at end of file diff --git a/content/ko/docs/reference/scheduling/policies.md b/content/ko/docs/reference/scheduling/policies.md index c2b9cbbdff..1146ba033a 100644 --- a/content/ko/docs/reference/scheduling/policies.md +++ b/content/ko/docs/reference/scheduling/policies.md @@ -98,5 +98,5 @@ weight: 10 * [스케줄링](/ko/docs/concepts/scheduling-eviction/kube-scheduler/)에 대해 배우기 * [kube-scheduler 프로파일](/docs/reference/scheduling/profiles/)에 대해 배우기 -* [kube-scheduler configuration 레퍼런스 (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1) 읽어보기 +* [kube-scheduler configuration 레퍼런스 (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2) 읽어보기 * [kube-scheduler Policy 레퍼런스 (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) 읽어보기 diff --git a/content/ko/docs/reference/tools/_index.md b/content/ko/docs/reference/tools/_index.md index 6ac3b1dc82..b854bc3bd5 100644 --- a/content/ko/docs/reference/tools/_index.md +++ b/content/ko/docs/reference/tools/_index.md @@ -8,8 +8,7 @@ no_list: true --- -쿠버네티스는 쿠버네티스 시스템으로 작업하는 데 도움이되는 몇 가지 기본 제공 도구를 포함한다. - +쿠버네티스는 쿠버네티스 시스템으로 작업하는 데 필요한 공통적으로 사용되거나 관련성 있는 여러 내장 도구와 외부 도구를 포함한다. diff --git a/content/ko/docs/reference/using-api/health-checks.md b/content/ko/docs/reference/using-api/health-checks.md index 07857ea5d2..2e00d338a7 100644 --- a/content/ko/docs/reference/using-api/health-checks.md +++ b/content/ko/docs/reference/using-api/health-checks.md @@ -91,7 +91,7 @@ curl -k 'https://localhost:6443/readyz?verbose&exclude=etcd' {{< feature-state state="alpha" >}} -각 개별 헬스 체크는 http 엔드포인트를 노출하고 개별적으로 체크가 가능하다. +각 개별 헬스 체크는 HTTP 엔드포인트를 노출하고 개별적으로 체크가 가능하다. 개별 체크를 위한 스키마는 `/livez/` 이고, 여기서 `livez` 와 `readyz` 는 API 서버의 활성 상태 또는 준비 상태인지를 확인할 때 사용한다. `` 경로 위에서 설명한 `verbose` 플래그를 사용해서 찾을 수 있고, `[+]` 와 `ok` 사이의 경로를 사용한다. 이러한 개별 헬스 체크는 머신에서 사용되서는 안되며, 운영자가 시스템의 현재 상태를 디버깅하는데 유용하다. diff --git a/content/ko/docs/setup/best-practices/cluster-large.md b/content/ko/docs/setup/best-practices/cluster-large.md index 899c63f6b7..5cf90e840e 100644 --- a/content/ko/docs/setup/best-practices/cluster-large.md +++ b/content/ko/docs/setup/best-practices/cluster-large.md @@ -121,3 +121,6 @@ _A_ 영역에 있는 컨트롤 플레인 호스트로만 전달한다. 단일 [클러스터 오토스케일러](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#readme)는 여러 클라우드 프로바이더와 통합되어 클러스터의 리소스 요구 수준에 맞는 노드 수를 실행할 수 있도록 도와준다. + +[addon resizer](https://github.com/kubernetes/autoscaler/tree/master/addon-resizer#readme)는 +클러스터 스케일이 변경될 때 자동으로 애드온 크기를 조정할 수 있도록 도와준다. \ No newline at end of file From 740c8762e2c4d9ba934ef743c1f87a6b49a342cb Mon Sep 17 00:00:00 2001 From: Gilson Melo Date: Mon, 27 Sep 2021 11:08:02 -0500 Subject: [PATCH 079/249] Include Oracle Cloud Infrastructure Including Oracle Cloud Infrastructure Security page. --- content/en/docs/concepts/security/overview.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/docs/concepts/security/overview.md b/content/en/docs/concepts/security/overview.md index ce75653dfd..0506fbbab0 100644 --- a/content/en/docs/concepts/security/overview.md +++ b/content/en/docs/concepts/security/overview.md @@ -60,6 +60,7 @@ Amazon Web Services | https://aws.amazon.com/security/ | Google Cloud Platform | https://cloud.google.com/security/ | IBM Cloud | https://www.ibm.com/cloud/security | Microsoft Azure | https://docs.microsoft.com/en-us/azure/security/azure-security | +Oracle Cloud Infrastructure | https://www.oracle.com/security/ | VMWare VSphere | https://www.vmware.com/security/hardening-guides.html | {{< /table >}} From 225f93c49e48780a2f498e36c49ccce5f43eef01 Mon Sep 17 00:00:00 2001 From: superleo Date: Sun, 26 Sep 2021 22:33:20 +0800 Subject: [PATCH 080/249] Sync 6 files under configure-pod-container/ with 1.22 version --- ...igure-liveness-readiness-startup-probes.md | 65 ++++++++++++------- .../configure-runasusername.md | 23 ++++--- .../configure-service-account.md | 23 ++++--- .../pull-image-private-registry.md | 4 +- .../quality-service-pod.md | 16 +++-- .../security-context.md | 39 ++++++++++- 6 files changed, 120 insertions(+), 50 deletions(-) diff --git a/content/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index 7355e5c11e..e180bbaea7 100644 --- a/content/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -217,14 +217,14 @@ Any code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure. You can see the source code for the server in -[server.go](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/test/images/agnhost/liveness/server.go). +[server.go](https://github.com/kubernetes/kubernetes/blob/master/test/images/agnhost/liveness/server.go). For the first 10 seconds that the container is alive, the `/healthz` handler returns a status of 200. After that, the handler returns a status of 500. --> 任何大于或等于 200 并且小于 400 的返回代码标示成功，其它返回代码都标示失败。 -可以在这里看服务的源码 [server.go](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/test/images/agnhost/liveness/server.go)。 +可以在这里看服务的源码 [server.go](https://github.com/kubernetes/kubernetes/blob/master/test/images/agnhost/liveness/server.go)。 容器存活的最开始 10 秒中，`/healthz` 处理程序返回一个 200 的状态码。之后处理程序返回 500 的状态码。 @@ -672,35 +672,52 @@ to resolve it. --> ### 探测器级别 `terminationGracePeriodSeconds` -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} -在 1.21 版之前，pod 级别的 `terminationGracePeriodSeconds` 被用来终止 -未能成功处理活跃性探测或启动探测的容器。 -这种耦合是意料之外的，可能会导致在设置了 pod 级别的 `terminationGracePeriodSeconds` 后， -需要很长的时间来重新启动失败的容器。 +在 1.21 及更高版本中，当特性门控 `ProbeTerminationGracePeriod` 为 +启用状态时，用户可以指定一个探测级别的 `terminationGracePeriodSeconds` 作为 +探针规格的一部分。当特性门控被启用时，并且 +Pod 级和探针级的 `terminationGracePeriodSeconds` 都已设置，kubelet 将 +使用探针级设置的值。 + +{{< note >}} + +从 Kubernetes 1.22 开始，`ProbeTerminationGracePeriod` 特性门控只 +在 API 服务器上可用。 kubelet 始终遵守探针级别 +`terminationGracePeriodSeconds` 字段（如果它存在于 Pod 上）。 -在1.21中，启用特性标志 `ProbeTerminationGracePeriod` 后， -用户可以指定一个探测器级别的 `terminationGracePeriodSeconds` 作为探测器规格的一部分。 -当该特性标志被启用时，若同时设置了 Pod 级别和探测器级别的 `terminationGracePeriodSeconds`， -kubelet 将使用探测器级的值。 +如果你已经为现有 Pod 设置了 “terminationGracePeriodSeconds” 字段并且 +不再希望使用针对每个探针的终止宽限期，则必须删除那些现有的 Pod。 -例如， + +当你（或控制平面或某些其他组件）创建替换 +Pods，并且特性门控 “ProbeTerminationGracePeriod” 被禁用，那么 +API 服务器会忽略 Pod 级别的 `terminationGracePeriodSeconds` 字段，即使 +Pod 或 Pod 模板指定了它。 +{{< /note >}} + +例如: ```yaml spec: diff --git a/content/zh/docs/tasks/configure-pod-container/configure-runasusername.md b/content/zh/docs/tasks/configure-pod-container/configure-runasusername.md index 19a9e4e71e..4b4e05dd4b 100644 --- a/content/zh/docs/tasks/configure-pod-container/configure-runasusername.md +++ b/content/zh/docs/tasks/configure-pod-container/configure-runasusername.md @@ -31,14 +31,17 @@ You need to have a Kubernetes cluster and the kubectl command-line tool must be ## 为 Pod 设置 Username -要指定运行 Pod 容器时所使用的用户名，请在 Pod 声明中包含 `securityContext` -（[PodSecurityContext](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritycontext-v1-core)）字段， -并在其内部包含 `windowsOptions` -（[WindowsSecurityContextOptions](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core)） +要指定运行 Pod 容器时所使用的用户名，请在 Pod 声明中包含 `securityContext` +([PodSecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritycontext-v1-core)) 字段， +并在其内部包含 `windowsOptions` +([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core)) 字段的 `runAsUserName` 字段。 ## 为容器设置 Username 要指定运行容器时所使用的用户名，请在容器清单中包含 `securityContext` -（[SecurityContext](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#securitycontext-v1-core)） +([SecurityContext](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#securitycontext-v1-core)) 字段，并在其内部包含 `windowsOptions` -（[WindowsSecurityContextOptions](/zh/docs/reference/generated/kubernetes-api/{{< param +（[WindowsSecurityContextOptions](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core)） 字段的 `runAsUserName` 字段。 diff --git a/content/zh/docs/tasks/configure-pod-container/configure-service-account.md b/content/zh/docs/tasks/configure-pod-container/configure-service-account.md index aba366f47e..e7bba36668 100644 --- a/content/zh/docs/tasks/configure-pod-container/configure-service-account.md +++ b/content/zh/docs/tasks/configure-pod-container/configure-service-account.md @@ -262,8 +262,8 @@ The output is similar to this: Name: build-robot-secret Namespace: default Labels: -Annotations: kubernetes.io/service-account.name=build-robot - kubernetes.io/service-account.uid=da68f9c6-9d26-11e7-b84e-002dc52800da +Annotations: kubernetes.io/service-account.name: build-robot + kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da Type: kubernetes.io/service-account-token @@ -540,15 +540,18 @@ JSON Web Key Set（JWKS）。 -集群包括一个默认的 RBAC ClusterRole， -名为 `system:service-account-issuer-discovery`。 -默认情况下不提供角色绑定对象。 -举例而言，管理员可以根据其安全性需要以及期望集成的外部系统选择是否将该角色绑定到 +集群包括一个的默认 RBAC ClusterRole, 名为 `system:service-account-issuer-discovery`。 +默认的 RBAC ClusterRoleBinding 将此角色分配给 `system:serviceaccounts` 组， +所有服务帐户隐式属于该组。这使得集群上运行的 Pod 能够通过它们所挂载的服务帐户令牌访问服务帐户发现文档。 +此外，管理员可以根据其安全性需要以及期望集成的外部系统选择是否将该角色绑定到 `system:authenticated` 或 `system:unauthenticated`。 另请参见： - [服务账号的集群管理员指南](/zh/docs/reference/access-authn-authz/service-accounts-admin/) -- [服务账号签署密钥检索 KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190730-oidc-discovery.md) +- [服务账号签署密钥检索 KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1393-oidc-discovery) - [OIDC 发现规范](https://openid.net/specs/openid-connect-discovery-1_0.html) diff --git a/content/zh/docs/tasks/configure-pod-container/pull-image-private-registry.md b/content/zh/docs/tasks/configure-pod-container/pull-image-private-registry.md index 6ddd85537b..c3b896ec35 100644 --- a/content/zh/docs/tasks/configure-pod-container/pull-image-private-registry.md +++ b/content/zh/docs/tasks/configure-pod-container/pull-image-private-registry.md @@ -110,7 +110,7 @@ kubectl create secret docker-registry regcred \ where: * `` is your Private Docker Registry FQDN. - Use `https://index.docker.io/v2/` for DockerHub. + Use `https://index.docker.io/v1/` for DockerHub. * `` is your Docker username. * `` is your Docker password. * `` is your Docker email. @@ -120,7 +120,7 @@ You have successfully set your Docker credentials in the cluster as a Secret cal 在这里： * `` 是你的私有 Docker 仓库全限定域名（FQDN）。 - DockerHub 使用 `https://index.docker.io/v2/`。 + DockerHub 使用 `https://index.docker.io/v1/`。 * `` 是你的 Docker 用户名。 * `` 是你的 Docker 密码。 * `` 是你的 Docker 邮箱。 diff --git a/content/zh/docs/tasks/configure-pod-container/quality-service-pod.md b/content/zh/docs/tasks/configure-pod-container/quality-service-pod.md index d2ae4a6b2f..b2fdccf8fb 100644 --- a/content/zh/docs/tasks/configure-pod-container/quality-service-pod.md +++ b/content/zh/docs/tasks/configure-pod-container/quality-service-pod.md @@ -59,8 +59,12 @@ kubectl create namespace qos-example For a Pod to be given a QoS class of Guaranteed: -* Every Container, including init containers, in the Pod must have a memory limit and a memory request, and they must be the same. -* Every Container, including init containers, in the Pod must have a CPU limit and a CPU request, and they must be the same. +* Every Container in the Pod must have a memory limit and a memory request. +* For every Container in the Pod, the memory limit must equal the memory request. +* Every Container in the Pod must have a CPU limit and a CPU request. +* For every Container in the Pod, the CPU limit must equal the CPU request. + +These restrictions apply to init containers and app containers equally. Here is the configuration file for a Pod that has one Container. The Container has a memory limit and a memory request, both equal to 200 MiB. The Container has a CPU limit and a CPU request, both equal to 700 milliCPU: @@ -69,8 +73,12 @@ memory request, both equal to 200 MiB. The Container has a CPU limit and a CPU r 对于 QoS 类为 Guaranteed 的 Pod： -* Pod 中的每个容器，包含初始化容器，必须指定内存请求和内存限制，并且两者要相等。 -* Pod 中的每个容器，包含初始化容器，必须指定 CPU 请求和 CPU 限制，并且两者要相等。 +* Pod 中的每个容器都必须指定内存限制和内存请求。 +* 对于 Pod 中的每个容器，内存限制必须等于内存请求。 +* Pod 中的每个容器都必须指定 CPU 限制和 CPU 请求。 +* 对于 Pod 中的每个容器，CPU 限制必须等于 CPU 请求。 + +这些限制同样适用于初始化容器和应用程序容器。 下面是包含一个容器的 Pod 配置文件。 容器设置了内存请求和内存限制，值都是 200 MiB。 diff --git a/content/zh/docs/tasks/configure-pod-container/security-context.md b/content/zh/docs/tasks/configure-pod-container/security-context.md index 42f07b5ceb..f05e678392 100644 --- a/content/zh/docs/tasks/configure-pod-container/security-context.md +++ b/content/zh/docs/tasks/configure-pod-container/security-context.md @@ -39,7 +39,7 @@ a Pod or Container. Security context settings include, but are not limited to: 为进程赋予 root 用户的部分特权而非全部特权。 * [AppArmor](/zh/docs/tutorials/clusters/apparmor/)：使用程序框架来限制个别程序的权能。 -* [Seccomp](https://en.wikipedia.org/wiki/Seccomp)：过滤进程的系统调用。 +* [Seccomp](/zh/docs/tutorials/clusters/seccomp/)：过滤进程的系统调用。 * AllowPrivilegeEscalation：控制进程是否可以获得超出其父进程的特权。 此布尔值直接控制是否为容器进程设置 [`no_new_privs`](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt)标志。 @@ -299,6 +299,41 @@ and [`emptydir`](/docs/concepts/storage/volumes/#emptydir). 这类临时性存储无效。 {{< /note >}} + +## 将卷权限和所有权更改委派给 CSI 驱动程序 +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + + +如果你部署了一个[容器存储接口 (CSI)](https://github.com/container-storage-interface/spec/blob/master/spec.md) +驱动支持 `VOLUME_MOUNT_GROUP` `NodeServiceCapability`， +在 `securityContext` 中指定 `fsGroup` 来设置文件所有权和权限的过程将由 CSI 驱动 +而不是 Kubernetes 来执行，前提是 Kubernetes 的 `DelegateFSGroupToCSIDriver` +特性门控已启用。在这种情况下，由于 Kubernetes 不执行任何 +所有权和权限更改，`fsGroupChangePolicy` 不会生效，并且 +按照 CSI 的规定，CSI 驱动应该使用所指定的 `fsGroup` 来挂载卷，从而生成了一个对 `fsGroup` 可读/可写的卷. + +更多的信息请参考 [KEP](https://github.com/gnufied/enhancements/blob/master/keps/sig-storage/2317-fsgroup-on-mount/README.md) +和 [CSI 规范](https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume) 中的字 +段 `VolumeCapability.MountVolume.volume_mount_group` 的描述 。 + ## Running an example Job @@ -638,6 +641,19 @@ driver, and then cleans up. An advantage of this approach is that the overall process gets the completion guarantee of a Job object, but maintains complete control over what Pods are created and how work is assigned to them. -## Cron Jobs {#cron-jobs} +## {{% heading "whatsnext" %}} -You can use a [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/) to create a Job that will run at specified times/dates, similar to the Unix tool `cron`. +* Learn about [Pods](/docs/concepts/workloads/pods). +* Read about different ways of running Jobs: + * [Coarse Parallel Processing Using a Work Queue](/docs/tasks/job/coarse-parallel-processing-work-queue/) + * [Fine Parallel Processing Using a Work Queue](/docs/tasks/job/fine-parallel-processing-work-queue/) + * Use an [indexed Job for parallel processing with static work assignment](/docs/tasks/job/indexed-parallel-processing-static/) (beta) + * Create multiple Jobs based on a template: [Parallel Processing using Expansions](/docs/tasks/job/parallel-processing-expansion/) +* Follow the links within [Clean up finished jobs automatically](#clean-up-finished-jobs-automatically) + to learn more about how your cluster can clean up completed and / or failed tasks. +* `Job` is part of the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/job-v1" >}} + object definition to understand the API for jobs. +* Read about [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/), which you + can use to define a series of Jobs that will run based on a schedule, similar to + the Unix tool `cron`. diff --git a/content/en/docs/concepts/workloads/controllers/replicaset.md b/content/en/docs/concepts/workloads/controllers/replicaset.md index e7b36d1b4a..70967853e8 100644 --- a/content/en/docs/concepts/workloads/controllers/replicaset.md +++ b/content/en/docs/concepts/workloads/controllers/replicaset.md @@ -410,3 +410,14 @@ selector requirements as described in the [labels user guide](/docs/concepts/ove As such, ReplicaSets are preferred over ReplicationControllers +## {{% heading "whatsnext" %}} + +* Learn about [Pods](/docs/concepts/workloads/pods). +* Learn about [Deployments](/docs/concepts/workloads/controllers/deployment/). +* [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/), + which relies on ReplicaSets to work. +* `ReplicaSet` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/replica-set-v1" >}} + object definition to understand the API for replica sets. +* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how + you can use it to manage application availability during disruptions. diff --git a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md index 5a10271b01..364df7dc72 100644 --- a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md +++ b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md @@ -284,6 +284,11 @@ machine-level function, such as machine monitoring or machine logging. These po to a machine lifetime: the pod needs to be running on the machine before other pods start, and are safe to terminate when the machine is otherwise ready to be rebooted/shutdown. -## For more information +## {{% heading "whatsnext" %}} -Read [Run Stateless Application Deployment](/docs/tasks/run-application/run-stateless-application-deployment/). +* Learn about [Pods](/docs/concepts/workloads/pods). +* Learn about [Deployment](/docs/concepts/workloads/controllers/deployment/), the replacement + for ReplicationController. +* `ReplicationController` is part of the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/replication-controller-v1" >}} + object definition to understand the API for replication controllers. diff --git a/content/en/docs/concepts/workloads/controllers/statefulset.md b/content/en/docs/concepts/workloads/controllers/statefulset.md index 3f89da5989..6b65ba1f3b 100644 --- a/content/en/docs/concepts/workloads/controllers/statefulset.md +++ b/content/en/docs/concepts/workloads/controllers/statefulset.md @@ -297,7 +297,18 @@ Please note that this field only works if you enable the `StatefulSetMinReadySec ## {{% heading "whatsnext" %}} +* Learn about [Pods](/docs/concepts/workloads/pods). +* Find out how to use StatefulSets + * Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/). + * Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/). + * Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/). + * Learn how to [scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/). + * Learn what's involved when you [delete a StatefulSet](/docs/tasks/run-application/delete-stateful-set/). + * Learn how to [configure a Pod to use a volume for storage](/docs/tasks/configure-pod-container/configure-volume-storage/). + * Learn how to [configure a Pod to use a PersistentVolume for storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/). +* `StatefulSet` is a top-level resource in the Kubernetes REST API. + Read the {{< api-reference page="workload-resources/stateful-set-v1" >}} + object definition to understand the API for stateful sets. +* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how + you can use it to manage application availability during disruptions. -* Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/). -* Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/). -* Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/). diff --git a/content/en/docs/concepts/workloads/pods/_index.md b/content/en/docs/concepts/workloads/pods/_index.md index 7132d19529..7246536eb0 100644 --- a/content/en/docs/concepts/workloads/pods/_index.md +++ b/content/en/docs/concepts/workloads/pods/_index.md @@ -309,9 +309,9 @@ in the Pod Lifecycle documentation. * Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). * Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how you can use it to manage application availability during disruptions. * Pod is a top-level resource in the Kubernetes REST API. - The [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) + The {{< api-reference page="workload-resources/pod-v1" >}} object definition describes the object in detail. -* [The Distributed System Toolkit: Patterns for Composite Containers](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns) explains common layouts for Pods with more than one container. +* [The Distributed System Toolkit: Patterns for Composite Containers](/blog/2015/06/the-distributed-system-toolkit-patterns/) explains common layouts for Pods with more than one container. To understand the context for why Kubernetes wraps a common Pod API in other resources (such as {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} or {{< glossary_tooltip text="Deployments" term_id="deployment" >}}), you can read about the prior art, including: From ba50c160da2069582bca15112ee22e2b390c7d10 Mon Sep 17 00:00:00 2001 From: Avinesh Tripathi Date: Tue, 28 Sep 2021 09:59:34 +0530 Subject: [PATCH 083/249] updated README-hi.md --- README-hi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-hi.md b/README-hi.md index 01d119a22b..1c7c0bfc1f 100644 --- a/README-hi.md +++ b/README-hi.md @@ -7,7 +7,7 @@ ## डॉक्स में योगदान देना -आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में कोई भी परिवर्तन करना चाहते हैं, और जब आप उन परिवर्तनों को हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं। +आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में परिवर्तन करने के बाद जब आप उनको हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं। एक बार जब आपका pull request बन जाता है, तो एक कुबरनेट्स समीक्षक स्पष्ट, कार्रवाई योग्य प्रतिक्रिया प्रदान करने की जिम्मेदारी लेगा। pull request के मालिक के रूप में, **यह आपकी जिम्मेदारी है कि आप कुबरनेट्स समीक्षक द्वारा प्रदान की गई प्रतिक्रिया को संबोधित करने के लिए अपने pull request को संशोधित करें।** From d3a5a71e884389abf4332f6152cca13744923cb3 Mon Sep 17 00:00:00 2001 From: Nitesh Seram Date: Tue, 28 Sep 2021 10:49:37 +0530 Subject: [PATCH 084/249] remove addonmanager.kubernetes.io labels --- .../en/examples/admin/dns/dns-horizontal-autoscaler.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml b/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml index ada91c1a35..b4bf65834a 100644 --- a/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml +++ b/content/en/examples/admin/dns/dns-horizontal-autoscaler.yaml @@ -3,15 +3,11 @@ apiVersion: v1 metadata: name: kube-dns-autoscaler namespace: kube-system - labels: - addonmanager.kubernetes.io/mode: Reconcile --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:kube-dns-autoscaler - labels: - addonmanager.kubernetes.io/mode: Reconcile rules: - apiGroups: [""] resources: ["nodes"] @@ -32,8 +28,6 @@ kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:kube-dns-autoscaler - labels: - addonmanager.kubernetes.io/mode: Reconcile subjects: - kind: ServiceAccount name: kube-dns-autoscaler @@ -52,7 +46,6 @@ metadata: labels: k8s-app: kube-dns-autoscaler kubernetes.io/cluster-service: "true" - addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: From 9fe3e942ff632a41017c601919d4b075423e856e Mon Sep 17 00:00:00 2001 From: kartik494 Date: Wed, 22 Sep 2021 11:17:54 +0530 Subject: [PATCH 085/249] Added note to improve access modes documentation --- .../concepts/storage/persistent-volumes.md | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index a029ceaeda..cd7137bedc 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -412,11 +412,22 @@ A PersistentVolume can be mounted on a host in any way supported by the resource The access modes are: -* ReadWriteOnce -- the volume can be mounted as read-write by a single node -* ReadOnlyMany -- the volume can be mounted read-only by many nodes -* ReadWriteMany -- the volume can be mounted as read-write by many nodes -* ReadWriteOncePod -- the volume can be mounted as read-write by a single Pod. - This is only supported for CSI volumes and Kubernetes version 1.22+. +`ReadWriteOnce` +: the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node. + +`ReadOnlyMany` +: the volume can be mounted as read-only by many nodes. + +`ReadWriteMany` +: the volume can be mounted as read-write by many nodes. + + `ReadWriteOncePod` +: the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+. + + + +The blog article [Introducing Single Pod Access Mode for PersistentVolumes](/blog/2021/09/13/read-write-once-pod-access-mode-alpha/) covers this in more detail. + In the CLI, the access modes are abbreviated to: From a7662f40fc7837e7221595a21905dff2951fef39 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 28 Sep 2021 18:22:54 +0100 Subject: [PATCH 086/249] Update publication date --- ....md => 2021-09-29-data-duplication-in-data-heavy-k8s-env.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename content/en/blog/_posts/{2021-09-07-data-duplication-in-data-heavy-k8s-env.md => 2021-09-29-data-duplication-in-data-heavy-k8s-env.md} (99%) diff --git a/content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md b/content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md similarity index 99% rename from content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md rename to content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md index bec8c0b892..42e9186692 100644 --- a/content/en/blog/_posts/2021-09-07-data-duplication-in-data-heavy-k8s-env.md +++ b/content/en/blog/_posts/2021-09-29-data-duplication-in-data-heavy-k8s-env.md @@ -1,7 +1,7 @@ --- layout: blog title: "How to Handle Data Duplication in Data-Heavy Kubernetes Environments" -date: 2021-09-07 +date: 2021-09-29 slug: how-to-handle-data-duplication-in-data-heavy-kubernetes-environments --- From 968dbbaaadb0ee778eb4a07b9f4a021158bd9cee Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 28 Sep 2021 18:44:42 +0100 Subject: [PATCH 087/249] Restrict width of blog figures for wide viewports Docsy uses Bootstrap to restrict the width of text paragraphs when the viewport is very wide. Also apply that to
    elements within blog articles. --- assets/scss/_custom.scss | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index e7f0902346..7d6aa6784f 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -635,6 +635,13 @@ body.td-documentation { } } +// Match Docsy-imposed max width on text body +@media (min-width: 1200px) { + body.td-blog main .td-content > figure { + max-width: 80%; + } +} + .td-content { table code { background-color: inherit !important; From 6a4dddd251f13f1c87a271544a048baaafe9a496 Mon Sep 17 00:00:00 2001 From: Jay Pipes Date: Thu, 12 Aug 2021 16:21:27 -0400 Subject: [PATCH 088/249] clarify declarative API in custom controller docs The content describing a declarative API in the custom controller section of the custom resources doc was confusing: > A declarative API allows you to declare or specify the desired state of your resource **and tries to keep the current state of Kubernetes objects in sync with the desired state**. The controller interprets the structured data as a record of the user's desired state, and continually maintains this state. (emphasis added) It is not the declarative API that tries to keep the current state of the objects in sync with the desired state. It's the controller that does that. I've reworded this paragraph to hopefully clarify this. Closes Issue #29348 Signed-off-by: Jay Pipes --- .../api-extension/custom-resources.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md index f37a71f278..0746e9f4f1 100644 --- a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md +++ b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md @@ -35,11 +35,11 @@ On their own, custom resources let you store and retrieve structured data. When you combine a custom resource with a *custom controller*, custom resources provide a true _declarative API_. -A [declarative API](/docs/concepts/overview/kubernetes-api/) -allows you to _declare_ or specify the desired state of your resource and tries to -keep the current state of Kubernetes objects in sync with the desired state. -The controller interprets the structured data as a record of the user's -desired state, and continually maintains this state. +The Kubernetes [declarative API](/docs/concepts/overview/kubernetes-api/) +enforces a separation of responsibilities. You declare the desired state of +your resource. The Kubernetes controller keeps the current state of Kubernetes +objects in sync with your declared desired state. This is in contrast to an +imperative API, where you *instruct* a server what to do. You can deploy and update a custom controller on a running cluster, independently of the cluster's lifecycle. Custom controllers can work with any kind of resource, From 2cf2b1937e647f0d034053bc8929f7d8a60f26b7 Mon Sep 17 00:00:00 2001 From: "Khaled (Kal) Henidak" Date: Mon, 20 Sep 2021 18:00:47 +0000 Subject: [PATCH 089/249] add a note re selectorless services and proxy --- content/en/docs/concepts/services-networking/service.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index db28918c98..24874355a5 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -183,6 +183,13 @@ Accessing a Service without a selector works the same as if it had a selector. In the example above, traffic is routed to the single endpoint defined in the YAML: `192.0.2.42:9376` (TCP). +{{< note >}} +The Kubernetes API server does not allow proxying to endpoints that are not mapped to +pods. Actions such as `kubectl proxy ` where the service has no +selector will fail due to this constraint. This prevents the Kubernetes API server +from being used as a proxy to endpoints the caller may not be authorized to access. +{{< /note >}} + An ExternalName Service is a special case of Service that does not have selectors and uses DNS names instead. For more information, see the [ExternalName](#externalname) section later in this document. From fd83a011414e1794fac43d6821f081dc84abdfbe Mon Sep 17 00:00:00 2001 From: Juhee Kang Date: Tue, 28 Sep 2021 09:08:49 +0900 Subject: [PATCH 090/249] [ko] Update outdated files in dev-1.22-ko.1 (p5) --- .../reference/labels-annotations-taints.md | 109 +++++++++++++++++- 1 file changed, 106 insertions(+), 3 deletions(-) diff --git a/content/ko/docs/reference/labels-annotations-taints.md b/content/ko/docs/reference/labels-annotations-taints.md index 0854c1b5cf..2577531042 100644 --- a/content/ko/docs/reference/labels-annotations-taints.md +++ b/content/ko/docs/reference/labels-annotations-taints.md @@ -200,7 +200,7 @@ kubelet이 Microsoft 윈도우에서 실행되고 있다면, 사용 중인 Windo kube-proxy 에는 커스텀 프록시를 위한 이와 같은 레이블이 있으며, 이 레이블은 서비스 컨트롤을 커스텀 프록시에 위임한다. -## experimental.windows.kubernetes.io/isolation-type +## experimental.windows.kubernetes.io/isolation-type (deprecated) {#experimental-windows-kubernetes-io-isolation-type} 예시: `experimental.windows.kubernetes.io/isolation-type: "hyperv"` @@ -210,6 +210,7 @@ Hyper-V 격리(isolation)를 사용하여 윈도우 컨테이너를 실행하려 {{< note >}} 이 어노테이션은 하나의 컨테이너로 구성된 파드에만 설정할 수 있다. +v1.20부터 이 어노테이션은 더이상 사용되지 않는다. 실험적인 Hyper-V 지원은 1.21버전에서 제거되었다. {{< /note >}} ## ingressclass.kubernetes.io/is-default-class @@ -262,11 +263,29 @@ kube-controller-manager의 잡(Job) 컨트롤러는 ## endpoints.kubernetes.io/over-capacity -예시: `endpoints.kubernetes.io/over-capacity:warning` +예시: `endpoints.kubernetes.io/over-capacity:truncated` 적용 대상: 엔드포인트(Endpoints) -v1.21 이상의 쿠버네티스 클러스터에서, 엔드포인트(Endpoints) 컨트롤러가 1000개 이상의 엔드포인트를 관리하고 있다면 각 엔드포인트 리소스에 이 어노테이션을 추가한다. 이 어노테이션은 엔드포인트 리소스가 용량 초과 되었음을 나타낸다. +v1.22 이상의 쿠버네티스 클러스터에서, 한 엔드포인트(Endpoints) 리소스가 관리하고 있는 엔드포인트의 수가 1000개 이상이면 엔드포인트 컨트롤러가 해당 엔드포인트 리소스에 이 어노테이션을 추가한다. 이 어노테이션은 해당 엔드포인트 리소스가 용량 초과 되었으며 엔드포인트 컨트롤러가 엔드포인트의 수를 1000으로 줄였음을 나타낸다. + +## batch.kubernetes.io/job-tracking + +예시: `batch.kubernetes.io/job-tracking: ""` + +적용 대상: 잡 + +잡에 어노테이션이 있으면 컨트롤 플레인은 [finalizers를 사용하여 잡 상태 추적](/ko/docs/concepts/workloads/controllers/job/#job-tracking-with-finalizers) +중임을 나타낸다. +어노테이션을 수동으로 추가하거나 제거하지 **않는다**. + +## scheduler.alpha.kubernetes.io/preferAvoidPods (deprecated) {#scheduleralphakubernetesio-preferavoidpods} + +적용 대상: 노드 + +이 어노테이션을 사용하려면 [NodePreferAvoidPods 스케줄링 플러그인](/ko/docs/reference/scheduling/config/#scheduling-plugins)이 활성화되어 있어야 한다. +해당 플러그인은 쿠버네티스 1.22에서 사용 중단되었다. +대신 [테인트와 톨러레이션](/ko/docs/concepts/scheduling-eviction/taint-and-toleration/)을 사용한다. **이 이후로 나오는 테인트는 모두 '적용 대상: 노드' 이다.** @@ -323,3 +342,87 @@ kubelet이 "외부" 클라우드 공급자에 의해 실행되었다면 노드 예시: `node.cloudprovider.kubernetes.io/shutdown:NoSchedule` 노드의 상태가 클라우드 공급자가 정의한 'shutdown' 상태이면, 이에 따라 노드에 `node.cloudprovider.kubernetes.io/shutdown` 테인트가 `NoSchedule` 값으로 설정된다. + +## pod-security.kubernetes.io/enforce + +예시: `pod-security.kubernetes.io/enforce: baseline` + +적용 대상: 네임스페이스 + +값은 **반드시** [파드 보안 표준](/docs/concepts/security/pod-security-standards) 레벨과 상응하는 +`privileged`, `baseline`, 또는 `restricted` 중 하나여야 한다. +특히 `enforce` 레이블은 표시된 수준에 정의된 요구 사항을 충족하지 않는 +레이블 네임스페이스에 모든 파드의 생성을 금지한다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. + +## pod-security.kubernetes.io/enforce-version + +예시: `pod-security.kubernetes.io/enforce-version: {{< skew latestVersion >}}` + +적용 대상: 네임스페이스 + +값은 **반드시** `latest`이거나 `v.` 형식의 유효한 쿠버네티스 버전이어야 한다. +설정된 파드의 유효성을 검사할 때 적용할 [파드 보안 표준](/docs/concepts/security/pod-security-standards) +정책의 버전이 결정된다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. + +## pod-security.kubernetes.io/audit + +예시: `pod-security.kubernetes.io/audit: baseline` + +적용 대상: 네임스페이스 + +값은 **반드시** [파드 보안 표준](/docs/concepts/security/pod-security-standards) 레벨과 상응하는 +`privileged`, `baseline`, 또는 `restricted` 중 하나여야 한다. +특히 `audit` 레이블은 표시된 수준에 정의된 요구 사항을 충족하지 않는 레이블 네임스페이스에 파드를 생성하는 것을 +방지하지 않지만, 해당 파드에 audit 어노테이션을 추가한다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. + +## pod-security.kubernetes.io/audit-version + +예시: `pod-security.kubernetes.io/audit-version: {{< skew latestVersion >}}` + +적용 대상: 네임스페이스 + +값은 **반드시** `latest`이거나 `v.` 형식의 유효한 쿠버네티스 버전이어야 한다. +설정된 파드의 유효성을 검사할 때 적용할 [파드 보안 표준](/docs/concepts/security/pod-security-standards) +정책의 버전이 결정된다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. + +## pod-security.kubernetes.io/warn + +예시: `pod-security.kubernetes.io/warn: baseline` + +적용 대상: 네임스페이스 + +값은 **반드시** [파드 보안 표준](/docs/concepts/security/pod-security-standards) 레벨과 상응하는 +`privileged`, `baseline`, 또는 `restricted` 중 하나여야 한다. +특히 `warn` 레이블은 해당 레이블이 달린 네임스페이스에, 표시된 레벨에 명시된 요구 사항을 충족하지 않는 파드를 생성하는 것을 +방지하지는 않지만, 그러한 파드가 생성되면 사용자에게 경고를 반환한다. +디플로이먼트, 잡, 스테이트풀셋 등과 같은 파드 템플릿을 포함하는 +객체를 만들거나 업데이트할 때에도 경고가 표시된다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. + +## pod-security.kubernetes.io/warn-version + +예시: `pod-security.kubernetes.io/warn-version: {{< skew latestVersion >}}` + +적용 대상: 네임스페이스 + +값은 **반드시** `latest`이거나 `v.` 형식의 유효한 쿠버네티스 버전이어야 한다. +설정된 파드의 유효성을 검사할 때 적용할 [파드 보안 표준](/docs/concepts/security/pod-security-standards) +정책의 버전이 결정된다. 디플로이먼트, 잡, 스테이트풀셋 등과 같은 파드 템플릿을 포함하는 +객체를 만들거나 업데이트할 때에도 경고가 표시된다. + +더 많은 정보는 [네임스페이스에서 파드 보안 적용](/docs/concepts/security/pod-security-admission)을 +참고한다. \ No newline at end of file From ba6cfa2784f77bcc319039ba5aed92bf9f738fcb Mon Sep 17 00:00:00 2001 From: howieyuen Date: Sun, 26 Sep 2021 16:41:21 +0800 Subject: [PATCH 091/249] [zh]translate indexed-parallel-processing-static.md --- .../job/indexed-parallel-processing-static.md | 277 ++++++++++++++++++ 1 file changed, 277 insertions(+) create mode 100644 content/zh/docs/tasks/job/indexed-parallel-processing-static.md diff --git a/content/zh/docs/tasks/job/indexed-parallel-processing-static.md b/content/zh/docs/tasks/job/indexed-parallel-processing-static.md new file mode 100644 index 0000000000..8b4e940373 --- /dev/null +++ b/content/zh/docs/tasks/job/indexed-parallel-processing-static.md @@ -0,0 +1,277 @@ +--- +title: 使用索引作业完成静态工作分配下的并行处理 +content_type: task +min-kubernetes-server-version: v1.21 +weight: 30 +--- + + +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + + + + + +在此示例中，你将运行一个使用多个并行工作进程的 Kubernetes Job。 +每个 worker 都是在自己的 Pod 中运行的不同容器。 +Pod 具有控制平面自动设置的 _索引编号（index number）_， +这些编号使得每个 Pod 能识别出要处理整个任务的哪个部分。 + + +Pod 索引在{{}} +`batch.kubernetes.io/job-completion-index` 中呈现，具体表示为一个十进制值字符串。 +为了让容器化的任务进程获得此索引，你可以使用 +[downward API](/zh/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api) +机制发布注解的值。为方便起见， +控制平面自动设置 downward API 以在 `JOB_COMPLETION_INDEX` 环境变量中公开索引。 + + +以下是此示例中步骤的概述： + +1. **定义使用带索引完成信息的 Job 清单**。 + Downward API 使你可以将 Pod 索引注释作为环境变量或文件传递给容器。 +2. **根据该清单启动一个带索引（`Indexed`）的 Job**。 + +## {{% heading "prerequisites" %}} + + +你应该已经熟悉 [Job](/zh/docs/concepts/workloads/controllers/job/) 的基本的、非并行的用法。 + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + +## 选择一种方法 + + +要从工作程序访问工作项，你有几个选择： + +1. 读取 `JOB_COMPLETION_INDEX` 环境变量。Job + {{< glossary_tooltip text="控制器" term_id="controller" >}} + 自动将此变量链接到包含完成索引的注解。 +1. 读取包含完整索引的文件。 +1. 假设你无法修改程序，你可以使用脚本包装它， + 该脚本使用上述任意方法读取索引并将其转换为程序可以用作输入的内容。 + + +对于此示例，假设你选择了方法 3 并且想要运行 +[rev](https://man7.org/linux/man-pages/man1/rev.1.html) 实用程序。 +这个程序接受一个文件作为参数并按逆序打印其内容。 + +```shell +rev data.txt +``` + + +你将使用 [`busybox`](https://hub.docker.com/_/busybox) 容器映像中的 `rev` 工具。 + + +由于这只是一个例子，每个 Pod 只做一小部分工作（反转一个短字符串）。 +例如，在实际工作负载中，你可能会创建一个表示基于场景数据制作 60 秒视频的任务的 Job 。 +视频渲染 Job 中的每个工作项都将渲染该视频剪辑的特定帧。 +索引完成意味着 Job 中的每个 Pod 都知道通过从剪辑开始计算帧数，来确定渲染和发布哪一帧，。 + + +## 定义索引作业 + + +这是一个使用 `Indexed` 完成模式的示例 Job 清单： + +{{< codenew language="yaml" file="application/job/indexed-job.yaml" >}} + + +在上面的示例中，你使用 Job 控制器为所有容器设置的内置 `JOB_COMPLETION_INDEX` 环境变量。 +[Init 容器](/zh/docs/concepts/workloads/pods/init-containers/) +将索引映射到一个静态值，并将其写入一个文件，该文件通过 +[emptyDir 卷](/zh/docs/concepts/storage/volumes/#emptydir) +与运行 worker 的容器共享。或者，你可以 +[通过 Downward API 定义自己的环境变量](/zh/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) +将索引发布到容器。你还可以选择从 +[包含 ConfigMap 的环境变量或文件](/zh/docs/tasks/configure-pod-container/configure-pod-configmap/) +加载值列表。 + + +或者也可以直接 +[使用 Downward API 将注解值作为卷文件传递](/zh/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#store-pod-fields)， +如下例所示： + +{{< codenew language="yaml" file="application/job/indexed-job-vol.yaml" >}} + + +## 执行 Job + + +现在执行 Job： + +```shell +# 使用第一种方法（依赖于 $JOB_COMPLETION_INDEX） +kubectl apply -f https://kubernetes.io/examples/application/job/indexed-job.yaml +``` + + +当你创建此 Job 时，控制平面会创建一系列 Pod，每个索引都由你指定。 +`.spec.parallelism` 的值决定了一次可以运行多少个， +而 `.spec.completions` 决定了 Job 总共创建了多少个 Pod。 + +因为 `.spec.parallelism` 小于 `.spec.completions`， +控制平面在启动更多 Pod 之前，等待部分第一批 Pod 完成。 + +创建 Job 后，稍等片刻，然后检查进度： + +```shell +kubectl describe jobs/indexed-job +``` + + +输出类似于： + +``` +Name: indexed-job +Namespace: default +Selector: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 +Labels: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 + job-name=indexed-job +Annotations: +Parallelism: 3 +Completions: 5 +Start Time: Thu, 11 Mar 2021 15:47:34 +0000 +Pods Statuses: 2 Running / 3 Succeeded / 0 Failed +Completed Indexes: 0-2 +Pod Template: + Labels: controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756 + job-name=indexed-job + Init Containers: + input: + Image: docker.io/library/bash + Port: + Host Port: + Command: + bash + -c + items=(foo bar baz qux xyz) + echo ${items[$JOB_COMPLETION_INDEX]} > /input/data.txt + + Environment: + Mounts: + /input from input (rw) + Containers: + worker: + Image: docker.io/library/busybox + Port: + Host Port: + Command: + rev + /input/data.txt + Environment: + Mounts: + /input from input (rw) + Volumes: + input: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-njkjj + Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-9kd4h + Normal SuccessfulCreate 4s job-controller Created pod: indexed-job-qjwsz + Normal SuccessfulCreate 1s job-controller Created pod: indexed-job-fdhq5 + Normal SuccessfulCreate 1s job-controller Created pod: indexed-job-ncslj +``` + + +在此示例中，你使用每个索引的自定义值运行 Job。 +你可以检查其中一个 Pod 的输出： + +```shell +kubectl logs indexed-job-fdhq5 # 更改它以匹配来自该 Job 的 Pod 的名称 +``` + + +输出类似于： + +``` +xuq +``` \ No newline at end of file From 640ca8aec89c8aee8a5150a1af541c13d5132f4f Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 24 Aug 2021 14:57:20 +0100 Subject: [PATCH 092/249] Tidy Windows node introduction Partial tidying to bring this page more in line with the Kubernetes documentation style guide. Co-authored-by: Shannon Kularathna --- .../windows/intro-windows-in-kubernetes.md | 1936 +++++++---------- 1 file changed, 783 insertions(+), 1153 deletions(-) diff --git a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md index c3757824c5..f395de9397 100644 --- a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md @@ -1,1335 +1,965 @@ --- -title: Intro to Windows support in Kubernetes -content_type: concept -weight: 65 reviewers: - jayunit100 - jsturtevant - marosset - perithompson +title: Windows containers in Kubernetes +content_type: concept +weight: 65 --- -Windows applications constitute a large portion of the services and -applications that run in many organizations. -[Windows containers](https://aka.ms/windowscontainers) provide a modern way to -encapsulate processes and package dependencies, making it easier to use DevOps -practices and follow cloud native patterns for Windows applications. -Kubernetes has become the defacto standard container orchestrator, and the -release of Kubernetes 1.14 includes production support for scheduling Windows -containers on Windows nodes in a Kubernetes cluster, enabling a vast ecosystem -of Windows applications to leverage the power of Kubernetes. Organizations -with investments in Windows-based applications and Linux-based applications -don't have to look for separate orchestrators to manage their workloads, -leading to increased operational efficiencies across their deployments, -regardless of operating system. +Windows applications constitute a large portion of the services and applications that +run in many organizations. [Windows containers](https://aka.ms/windowscontainers) +provide a way to encapsulate processes and package dependencies, making it easier +to use DevOps practices and follow cloud native patterns for Windows applications. + +Organizations with investments in Windows-based applications and Linux-based +applications don't have to look for separate orchestrators to manage their workloads, +leading to increased operational efficiencies across their deployments, regardless +of operating system. -## Windows containers in Kubernetes +## Windows nodes in Kubernetes -To enable the orchestration of Windows containers in Kubernetes, include -Windows nodes in your existing Linux cluster. Scheduling Windows containers in +To enable the orchestration of Windows containers in Kubernetes, include Windows nodes +in your existing Linux cluster. Scheduling Windows containers in {{< glossary_tooltip text="Pods" term_id="pod" >}} on Kubernetes is similar to scheduling Linux-based containers. In order to run Windows containers, your Kubernetes cluster must include -multiple operating systems, with control plane nodes running Linux and workers -running either Windows or Linux depending on your workload needs. Windows -Server 2019 is the only Windows operating system supported, enabling -[Kubernetes Node](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md#the-kubernetes-node) -on Windows (including kubelet, -[container runtime](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/containerd), -and kube-proxy). For a detailed explanation of Windows distribution channels -see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). +multiple operating systems. +While you can only run the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} on Linux, you can deploy worker nodes running either Windows or Linux depending on your workload needs. -The Kubernetes control plane, including the -[master components](/docs/concepts/overview/components/), -continues to run on Linux. -There are no plans to have a Windows-only Kubernetes cluster. +Windows {{< glossary_tooltip text="nodes" term_id="node" >}} are +[supported](#windows-os-version-support) provided that the operating system is +Windows Server 2019. -In this document, when we talk about Windows containers we mean Windows -containers with process isolation. Windows containers with -[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container) -is planned for a future release. +This document uses the term *Windows containers* to mean Windows containers with +process isolation. Kubernetes does not support running Windows containers with +[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container). -## Supported Functionality and Limitations +## Resource management -### Supported Functionality +On Linux nodes, {{< glossary_tooltip text="cgroups" term_id="cgroup" >}} are used +as a pod boundary for resource control. Containers are created within that boundary +for network, process and file system isolation. The Linux cgroup APIs can be used +to gather CPU, I/O, and memory use statistics. -#### Windows OS Version Support +In contrast, Windows uses a _job object_ per container with a system namespace filter +to contain all processes in a container and provide logical isolation from the +host. +(Job objects are a Windows process isolation mechanism and are different from +what Kubernetes refers to as a {{< glossary_tooltip term_id="job" text="Job" >}}). -Refer to the following table for Windows operating system support in -Kubernetes. A single heterogeneous Kubernetes cluster can have both Windows -and Linux worker nodes. Windows containers have to be scheduled on Windows -nodes and Linux containers on Linux nodes. +There is no way to run a Windows container without the namespace filtering in +place. This means that system privileges cannot be asserted in the context of the +host, and thus privileged containers are not available on Windows. +Containers cannot assume an identity from the host because the Security Account Manager +(SAM) is separate. -| Kubernetes version | Windows Server LTSC releases | Windows Server SAC releases | -| --- | --- | --- | --- | -| *Kubernetes v1.20* | Windows Server 2019 | Windows Server ver 1909, Windows Server ver 2004 | -| *Kubernetes v1.21* | Windows Server 2019 | Windows Server ver 2004, Windows Server ver 20H2 | -| *Kubernetes v1.22* | Windows Server 2019 | Windows Server ver 2004, Windows Server ver 20H2 | +#### Memory reservations {#resource-management-memory} -Information on the different Windows Server servicing channels including their -support models can be found at -[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). +Windows does not have an out-of-memory process killer as Linux does. Windows always +treats all user-mode memory allocations as virtual, and pagefiles are mandatory +(on Linux, the kubelet will by default not start with swap space enabled). -We don't expect all Windows customers to update the operating system for their -apps frequently. Upgrading your applications is what dictates and necessitates -upgrading or introducing new nodes to the cluster. For the customers that -chose to upgrade their operating system for containers running on Kubernetes, -we will offer guidance and step-by-step instructions when we add support for a -new operating system version. This guidance will include recommended upgrade -procedures for upgrading user applications together with cluster nodes. -Windows nodes adhere to Kubernetes -[version-skew policy](/docs/setup/release/version-skew-policy/) (node to control plane -versioning) the same way as Linux nodes do today. +Windows nodes do not overcommit memory for processes running in containers. The +net effect is that Windows won't reach out of memory conditions the same way Linux +does, and processes page to disk instead of being subject to out of memory (OOM) +termination. If memory is over-provisioned and all physical memory is exhausted, +then paging can slow down performance. +You can place bounds on memory use for workloads using the kubelet +parameters `--kubelet-reserve` and/or `--system-reserve`; these account +for memory usage on the node (outside of containers), and reduce +[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)). +As you deploy workloads, set resource limits on containers. This also subtracts from +`NodeAllocatable` and prevents the scheduler from adding more pods once a node is full. -The Windows Server Host Operating System is subject to the -[Windows Server ](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) -licensing. The Windows Container images are subject to the -[Supplemental License Terms for Windows containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/images-eula). +{{< note >}} +When you set memory resource limits for Windows containers, you should either set a +limit and leave the memory request unspecified, or set the request equal to the limit. +{{< /note >}} -Windows containers with process isolation have strict compatibility rules, -[where the host OS version must match the container base image OS version](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility). -Once we support Windows containers with Hyper-V isolation in Kubernetes, the -limitation and compatibility rules will change. +On Windows, good practice to avoid over-provisioning is to configure the kubelet +with a system reserved memory of at least 2GiB to account for Windows, Kubernetes +and container runtime overheads. -#### Pause Image +#### CPU reservations {#resource-management-cpu} -Kubernetes maintains a multi-architecture image that includes support for Windows. -For Kubernetes v1.22 the recommended pause image is `k8s.gcr.io/pause:3.5`. -The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause) -is available on GitHub. +To account for CPU use by the operating system, the container runtime, and by +Kubernetes host processes such as the kubelet, you can (and should) reserve a +percentage of total CPU. You should determine this CPU reservation taking account of +to the number of CPU cores available on the node. To decide on the CPU percentage to +reserve, identify the maximum pod density for each node and monitor the CPU usage of +the system services running there, then choose a value that meets your workload needs. -Microsoft maintains a multi-architecture image with Linux and Windows amd64 support at `mcr.microsoft.com/oss/kubernetes/pause:3.5`. -This image is built from the same source as the Kubernetes maintained image but all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft. -The Microsoft maintained image is recommended for production environments when signed binaries are required. +You can place bounds on CPU usage for workloads using the +kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to +account for CPU usage on the node (outside of containers). +This reduces `NodeAllocatable`. +The cluster-wide scheduler then takes this reservation into account when determining +pod placement. -#### Compute +On Windows, the kubelet supports a command-line flag to set the priority of the +kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get +more CPU time slices when compared to other processes running on the Windows host. +More information on the allowable values and their meaning is available at +[Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). +To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above. -From an API and kubectl perspective, Windows containers behave in much the -same way as Linux-based containers. However, there are some notable -differences in key functionality which are outlined in the -[limitation section](#limitations). +## Compatibility and limitations {#limitations} -Key Kubernetes elements work the same way in Windows as they do in Linux. In -this section, we talk about some of the key workload enablers and how they map -to Windows. +Some node features are only available if you use a specific +[container runtime](#container-runtime); others are not available on Windows nodes, +including: + +* HugePages: not supported for Windows containers +* Privileged containers: not supported for Windows containers +* TerminationGracePeriod: not implemented + +Not all features of shared namespaces are supported. See [API compatibility](#api) +for more details. + +See [Windows OS version compatibility](#windows-os-version-support) for details on +the Windows versions that Kubernetes is tested against. + +From an API and kubectl perspective, Windows containers behave in much the same +way as Linux-based containers. However, there are some notable differences in key +functionality which are outlined in this section. + +### Comparison with Linux {#compatibility-linux-similarities} + +Key Kubernetes elements work the same way in Windows as they do in Linux. This +section refers to several key workload enablers and how they map to Windows. * [Pods](/docs/concepts/workloads/pods/) - A Pod is the basic building block of Kubernetes–the smallest and simplest - unit in the Kubernetes object model that you create or deploy. You may not - deploy Windows and Linux containers in the same Pod. All containers in a Pod - are scheduled onto a single Node where each Node represents a specific - platform and architecture. The following Pod capabilities, properties and - events are supported with Windows containers: + A Pod is the basic building block of Kubernetes–the smallest and simplest unit in + the Kubernetes object model that you create or deploy. You may not deploy Windows and + Linux containers in the same Pod. All containers in a Pod are scheduled onto a single + Node where each Node represents a specific platform and architecture. The following + Pod capabilities, properties and events are supported with Windows containers: * Single or multiple containers per Pod with process isolation and volume sharing - * Pod status fields + * Pod `status` fields * Readiness and Liveness probes * postStart & preStop container lifecycle events * ConfigMap, Secrets: as environment variables or volumes - * EmptyDir + * `emptyDir` volumes * Named pipe host mounts * Resource limits -* [Controllers](/docs/concepts/workloads/controllers/) - - Kubernetes controllers handle the desired state of Pods. The following - workload controllers are supported with Windows containers: - +* [Workload resources](/docs/concepts/workloads/controllers/) including: * ReplicaSet - * ReplicationController * Deployments * StatefulSets * DaemonSet * Job * CronJob + * ReplicationController +* {{< glossary_tooltip text="Services" term_id="service" >}} + See [Load balancing and Services](#load-balancing-and-services) for more details. -* [Services](/docs/concepts/services-networking/service/) - - A Kubernetes Service is an abstraction which defines a logical set of Pods - and a policy by which to access them - sometimes called a micro-service. You - can use services for cross-operating system connectivity. In Windows, services - can utilize the following types, properties and capabilities: - - * Service Environment variables - * NodePort - * ClusterIP - * LoadBalancer - * ExternalName - * Headless services - -Pods, Controllers and Services are critical elements to managing Windows +Pods, workload resources, and Services are critical elements to managing Windows workloads on Kubernetes. However, on their own they are not enough to enable the proper lifecycle management of Windows workloads in a dynamic cloud native -environment. We added support for the following features: +environment. Kubernetes also supports: +* `kubectl exec` * Pod and container metrics -* Horizontal Pod Autoscaler support -* kubectl Exec -* Resource Quotas +* {{< glossary_tooltip text="Horizontal pod autoscaling" term_id="horizontal-pod-autoscaler" >}} +* {{< glossary_tooltip text="Resource quotas" term_id="resource-quota" >}} * Scheduler preemption -#### Container Runtime -##### Docker EE - -{{< feature-state for_k8s_version="v1.14" state="stable" >}} - -Docker EE-basic 19.03+ is the recommended container runtime for all Windows -Server versions. This works with the dockershim code included in the kubelet. - -##### CRI-ContainerD - -{{< feature-state for_k8s_version="v1.20" state="stable" >}} - -{{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ can -also be used as the container runtime for Windows Kubernetes nodes. - -Learn how to -[install ContainerD on a Windows](/docs/setup/production-environment/container-runtimes/#install-containerd). - -#### Persistent Storage - -Kubernetes [volumes](/docs/concepts/storage/volumes/) enable complex -applications, with data persistence and Pod volume sharing requirements, to be -deployed on Kubernetes. Management of persistent volumes associated with a -specific storage back-end or protocol includes actions such as: -provisioning/de-provisioning/resizing of volumes, attaching/detaching a volume -to/from a Kubernetes node and mounting/dismounting a volume to/from individual -containers in a pod that needs to persist data. The code implementing these -volume management actions for a specific storage back-end or protocol is -shipped in the form of a Kubernetes volume -[plugin](/docs/concepts/storage/volumes/#types-of-volumes). The following -broad classes of Kubernetes volume plugins are supported on Windows: - -##### In-tree Volume Plugins - -Code associated with in-tree volume plugins ship as part of the core -Kubernetes code base. Deployment of in-tree volume plugins do not require -installation of additional scripts or deployment of separate containerized -plugin components. These plugins can handle: provisioning/de-provisioning and -resizing of volumes in the storage backend, attaching/detaching of volumes -to/from a Kubernetes node and mounting/dismounting a volume to/from individual -containers in a pod. The following in-tree plugins support Windows nodes: - -* [awsElasticBlockStore](/docs/concepts/storage/volumes/#awselasticblockstore) -* [azureDisk](/docs/concepts/storage/volumes/#azuredisk) -* [azureFile](/docs/concepts/storage/volumes/#azurefile) -* [gcePersistentDisk](/docs/concepts/storage/volumes/#gcepersistentdisk) -* [vsphereVolume](/docs/concepts/storage/volumes/#vspherevolume) - -##### FlexVolume Plugins - -Code associated with [FlexVolume](/docs/concepts/storage/volumes/#flexVolume) -plugins ship as out-of-tree scripts or binaries that need to be deployed -directly on the host. FlexVolume plugins handle attaching/detaching of volumes -to/from a Kubernetes node and mounting/dismounting a volume to/from individual -containers in a pod. Provisioning/De-provisioning of persistent volumes -associated with FlexVolume plugins may be handled through an external -provisioner that is typically separate from the FlexVolume plugins. The -following FlexVolume -[plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), -deployed as powershell scripts on the host, support Windows nodes: - -* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd) -* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd) - -##### CSI Plugins - -{{< feature-state for_k8s_version="v1.22" state="stable" >}} - -Code associated with {{< glossary_tooltip text="CSI" term_id="csi" >}} plugins -ship as out-of-tree scripts and binaries that are typically distributed as -container images and deployed using standard Kubernetes constructs like -DaemonSets and StatefulSets. CSI plugins handle a wide range of volume -management actions in Kubernetes: provisioning/de-provisioning/resizing of -volumes, attaching/detaching of volumes to/from a Kubernetes node and -mounting/dismounting a volume to/from individual containers in a pod, -backup/restore of persistent data using snapshots and cloning. - -CSI plugins communicate with a CSI node plugin which performs the local storage operations. -On Windows nodes CSI node plugins typically call APIs exposed by the community-managed -[csi-proxy](https://github.com/kubernetes-csi/csi-proxy) which handles the local storage operations. - -Please refer to the deployment guide of the environment where you wish to deploy a Windows CSI plugin -for further details around installation. -You may also refer to the following [installation steps](https://github.com/kubernetes-csi/csi-proxy#installation). - -#### Networking +### Networking on Windows nodes {#compatibility-networking} Networking for Windows containers is exposed through [CNI plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). Windows containers function similarly to virtual machines in regards to -networking. Each container has a virtual network adapter (vNIC) which is -connected to a Hyper-V virtual switch (vSwitch). The Host Networking Service -(HNS) and the Host Compute Service (HCS) work together to create containers -and attach container vNICs to networks. HCS is responsible for the management -of containers whereas HNS is responsible for the management of networking -resources such as: +networking. Each container has a virtual network adapter (vNIC) which is connected +to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the +Host Compute Service (HCS) work together to create containers and attach container +vNICs to networks. HCS is responsible for the management of containers whereas HNS +is responsible for the management of networking resources such as: * Virtual networks (including creation of vSwitches) * Endpoints / vNICs * Namespaces -* Policies (Packet encapsulations, Load-balancing rules, ACLs, NAT'ing rules, etc.) +* Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules. -The following service spec types are supported: +#### Container networking {#networking} -* NodePort -* ClusterIP -* LoadBalancer -* ExternalName +The Windows HNS and vSwitch implement namespacing and can +create virtual NICs as needed for a pod or container. However, many configurations such +as DNS, routes, and metrics are stored in the Windows registry database rather than as +files inside `/etc`, which is how Linux stores those configurations. The Windows registry for the container +is separate from that of the host, so concepts like mapping `/etc/resolv.conf` from +the host into a container don't have the same effect they would on Linux. These must +be configured using Windows APIs run in the context of that container. Therefore +CNI implementations need to call the HNS instead of relying on file mappings to pass +network details into the pod or container. -##### Network modes +The following networking functionality is _not_ supported on Windows nodes: + +* Host networking mode +* Local NodePort access from the node itself (works for other nodes or external clients) +* More than 64 backend pods (or unique destination addresses) for a single Service +* IPv6 communication between Windows pods connected to overlay networks +* Local Traffic Policy in non-DSR mode +* Outbound communication using the ICMP protocol via the `win-overlay`, `win-bridge`, or using the Azure-CNI plugin.\ + Specifically, the Windows data plane ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) doesn't support ICMP packet transpositions, and this means: + * ICMP packets directed to destinations within the same network (such as pod to pod communication via ping) work as expected and without any limitations; + * TCP/UDP packets work as expected and without any limitations; + * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) cannot be transposed and thus will not be routed back to their source; + * Since TCP/UDP packets can still be transposed, you can substitute `ping ` with `curl ` to get some debugging insight into connectivity with the outside world. + +Overlay networking support in kube-proxy is a beta feature. In addition, it requires +[KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) +to be installed on Windows Server 2019. + +#### Network modes Windows supports five different networking drivers/modes: L2bridge, L2tunnel, -Overlay, Transparent, and NAT. In a heterogeneous cluster with Windows and -Linux worker nodes, you need to select a networking solution that is -compatible on both Windows and Linux. The following out-of-tree plugins are -supported on Windows, with recommendations on when to use each CNI: +Overlay (beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux +worker nodes, you need to select a networking solution that is compatible on both +Windows and Linux. The following out-of-tree plugins are supported on Windows, +with recommendations on when to use each CNI: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Network DriverDescriptionContainer Packet ModificationsNetwork PluginsNetwork Plugin Characteristics
    L2bridgeContainers are attached to an external vSwitch. Containers are attached - to the underlay network, although the physical network doesn't need to learn - the container. MACs because they are rewritten on ingress/egress. - - MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS - OutboundNAT policy. - - win-bridge, - Azure-CNI, - Flannel host-gateway uses win-bridge - - win-bridge uses L2bridge network mode, - connects containers to the underlay of hosts, offering best performance. - Requires user-defined routes (UDR) for inter-node connectivity. -
    L2Tunnel - This is a special case of l2bridge, but only used on Azure. All packets - are sent to the virtualization host where SDN policy is applied. - - MAC rewritten, IP visible on the underlay network - - Azure-CNI - - Azure-CNI allows integration of containers with Azure vNET, and allows them - to leverage the set of capabilities that - Azure Virtual Network - provides. For example, securely connect to Azure services or use Azure NSGs. - See azure-cni - for some examples. -
    Overlay (Overlay networking for Windows in Kubernetes is in Alpha stage) - Containers are given a vNIC connected to an external vSwitch. Each overlay - network gets its own IP subnet, defined by a custom IP prefix.The overlay - network driver uses VXLAN encapsulation. - - Encapsulated with an outer header. - - Win-overlay, - Flannel VXLAN (uses win-overlay) - - win-overlay should be used when virtual container networks are desired to - be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs - to be re-used for different overlay networks (which have different VNID tags) - if you are restricted on IPs in your datacenter. This option requires - KB4489899 on Windows Server - 2019. -
    - Transparent (special use case for ovn-kubernetes) - - Requires an external vSwitch. Containers are attached to an external - vSwitch which enables intra-pod communication via logical networks (logical - switches and routers). - - Packet is encapsulated either via - GENEVE, - STT tunneling to reach - pods which are not on the same host.
    Packets are forwarded or dropped - via the tunnel metadata information supplied by the ovn network controller. -
    - NAT is done for north-south communication. -     		- ovn-kubernetes - - Deploy via Ansible. - Distributed ACLs can be applied via Kubernetes policies. IPAM support. - Load-balancing can be achieved without kube-proxy. NATing is done without - using iptables/netsh. -
    NAT (not used in Kubernetes) - Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is - provided using an internal component called - WinNAT. - - MAC and IP is rewritten to host MAC/IP. - - nat - - Included here for completeness -
    +| Network Driver | Description | Container Packet Modifications | Network Plugins | Network Plugin Characteristics | +| -------------- | ----------- | ------------------------------ | --------------- | ------------------------------ | +| L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. | MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity. | +| L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/en-us/azure/aks/concepts-network#azure-cni-advanced-networking) | +| Overlay (Overlay networking for Windows in Kubernetes is in *alpha* stage) | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header. | [Win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires [KB4489899](https://support.microsoft.com/help/4489899) on Windows Server 2019. | +| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host.
    Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
    NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | +| NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://blogs.technet.microsoft.com/virtualization/2016/05/25/windows-nat-winnat-capabilities-and-limitations/) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness | -As outlined above, the [Flannel](https://github.com/coreos/flannel) CNI -[meta plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel) -is also supported on -[Windows](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel#windows-support-experimental) -via the [VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) -(**alpha support** ; delegates to win-overlay) and -[host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) -(stable support; delegates to win-bridge). This plugin supports delegating to -one of the reference CNI plugins (win-overlay, win-bridge), to work in -conjunction with Flannel daemon on Windows (Flanneld) for automatic node -subnet lease assignment and HNS network creation. This plugin reads in its own -configuration file (cni.conf), and aggregates it with the environment -variables from the FlannelD generated subnet.env file. It then delegates to -one of the reference CNI plugins for network plumbing, and sends the correct -configuration containing the node-assigned subnet to the IPAM plugin (e.g. -host-local). +As outlined above, the [Flannel](https://github.com/coreos/flannel) +CNI [meta plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel) +is also [supported](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel#windows-support-experimental) on Windows via the +[VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**alpha support** ; delegates to win-overlay) +and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge). -For the node, pod, and service objects, the following network flows are -supported for TCP/UDP traffic: +This plugin supports delegating to one of the reference CNI plugins (win-overlay, +win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for +automatic node subnet lease assignment and HNS network creation. This plugin reads +in its own configuration file (cni.conf), and aggregates it with the environment +variables from the FlannelD generated subnet.env file. It then delegates to one of +the reference CNI plugins for network plumbing, and sends the correct configuration +containing the node-assigned subnet to the IPAM plugin (for example: `host-local`). -* Pod -> Pod (IP) -* Pod -> Pod (Name) -* Pod -> Service (Cluster IP) -* Pod -> Service (PQDN, but only if there are no ".") -* Pod -> Service (FQDN) -* Pod -> External (IP) -* Pod -> External (DNS) -* Node -> Pod -* Pod -> Node +For Node, Pod, and Service objects, the following network flows are supported for +TCP/UDP traffic: -##### IP address management (IPAM) {#ipam} +* Pod → Pod (IP) +* Pod → Pod (Name) +* Pod → Service (Cluster IP) +* Pod → Service (PQDN, but only if there are no ".") +* Pod → Service (FQDN) +* Pod → external (IP) +* Pod → external (DNS) +* Node → Pod +* Pod → Node + +#### CNI plugin limitations + +* Windows reference network plugins win-bridge and win-overlay do not implement + [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) v0.4.0, + due to a missing `CHECK` implementation. +* The Flannel VXLAN CNI plugin has the following limitations on Windows: + +1. Node-pod connectivity isn't possible by design. It's only possible for local pods with Flannel v0.12.0 (or higher). +2. Flannel is restricted to using VNI 4096 and UDP port 4789. See the official + [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) + backend docs for more details on these parameters. + +#### IP address management (IPAM) {#ipam} The following IPAM options are supported on Windows: -* [Host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local) +* [host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local) * HNS IPAM (Inbox platform IPAM, this is a fallback when no IPAM is set) -* [Azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only) +* [azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only) -##### Load balancing and Services +#### Load balancing and Services + +A Kubernetes {{< glossary_tooltip text="Service" term_id="service" >}} is an abstraction +that defines a logical set of Pods and a means to access them over a network. +In a cluster that includes Windows nodes, you can use the following types of Service: + + * `NodePort` + * `ClusterIP` + * `LoadBalancer` + * `ExternalName` + +Windows container networking differs in some important ways from Linux networking. +The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) provides +additional details and background. On Windows, you can use the following settings to configure Services and load balancing behavior: {{< table caption="Windows Service Settings" >}} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FeatureDescriptionSupported Kubernetes versionSupported Windows OS buildHow to enable
    Session affinity - Ensures that connections from a particular client are passed to the same - Pod each time. - v1.20+ - Windows Server vNext Insider Preview Build 19551 (or higher) - - Set service.spec.sessionAffinity to "ClientIP" -
    Direct Server Return (DSR) - Load balancing mode where the IP address fixups and the LBNAT occurs at - the container vSwitch port directly; service traffic arrives with the source - IP set as the originating pod IP. - v1.20+ - Windows Server 2019 - - Set the following flags in kube-proxy: - --feature-gates="WinDSR=true" --enable-dsr=true -
    Preserve-Destination - Skips DNAT of service traffic, thereby preserving the virtual IP of the target - service in packets reaching the backend Pod. Also disables node-node forwarding. - v1.20+Windows Server, version 1903 (or higher) - Set "preserve-destination": "true" in service annotations - and enable DSR in kube-proxy. -
    IPv4/IPv6 dual-stack networking - Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, - and within a cluster - v1.19+Windows Server, version 2004 (or higher) - See IPv4/IPv6 dual-stack -
    Client IP preservation - Ensures that source IP of incoming ingress traffic gets preserved. Also - disables node-node forwarding. - v1.20+Windows Server, version 2019 (or higher) - Set service.spec.externalTrafficPolicy to "Local" and enable - DSR in kube-proxy. -
    - +| Feature | Description | Supported Kubernetes version | Supported Windows OS build | How to enable | +| ------- | ----------- | ----------------------------- | -------------------------- | ------------- | +| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | v1.20+ | [Windows Server vNext Insider Preview Build 19551](https://blogs.windows.com/windowsexperience/2020/01/28/announcing-windows-server-vnext-insider-preview-build-19551/) (or higher) | Set `service.spec.sessionAffinity` to "ClientIP" | +| Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | v1.20+ | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` | +| Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | v1.20+ | Windows Server, version 1903 (or higher) | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. | +| IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | v1.19+ | Windows Server, version 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) | +| Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. | v1.20+ | Windows Server, version 2019 | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy | {{< /table >}} -#### IPv4/IPv6 dual-stack - -You can enable IPv4/IPv6 dual-stack networking for `l2bridge` networks using -the `IPv6DualStack` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). See -[enable IPv4/IPv6 dual stack](/docs/concepts/services-networking/dual-stack#enable-ipv4ipv6-dual-stack) -for more details. - -On Windows, using IPv6 with Kubernetes require Windows Server, version 2004 -(kernel version 10.0.19041.610) or later. - -Overlay (VXLAN) networks on Windows do not support dual-stack networking today. - -### Limitations - -Windows is only supported as a worker node in the Kubernetes architecture and -component matrix. This means that a Kubernetes cluster must always include -Linux master nodes, zero or more Linux worker nodes, and zero or more Windows -worker nodes. - -#### Resource Handling - -Linux cgroups are used as a pod boundary for resource controls in Linux. -Containers are created within that boundary for network, process and file -system isolation. The cgroups APIs can be used to gather cpu/io/memory stats. -In contrast, Windows uses a Job object per container with a system namespace -filter to contain all processes in a container and provide logical isolation -from the host. There is no way to run a Windows container without the -namespace filtering in place. This means that system privileges cannot be -asserted in the context of the host, and thus privileged containers are not -available on Windows. Containers cannot assume an identity from the host -because the Security Account Manager (SAM) is separate. - -#### Resource Reservations - -##### Memory Reservations - -Windows does not have an out-of-memory process killer as Linux does. Windows -always treats all user-mode memory allocations as virtual, and pagefiles are -mandatory. The net effect is that Windows won't reach out of memory conditions -the same way Linux does, and processes page to disk instead of being subject -to out of memory (OOM) termination. If memory is over-provisioned and all -physical memory is exhausted, then paging can slow down performance. - -Keeping memory usage within reasonable bounds is possible using the kubelet -parameters `--kubelet-reserve` and/or `--system-reserve` to account for memory -usage on the node (outside of containers). This reduces -[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). - -As you deploy workloads, use resource limits (must set only limits or limits -must equal requests) on containers. This also subtracts from NodeAllocatable -and prevents the scheduler from adding more pods once a node is full. - -A best practice to avoid over-provisioning is to configure the kubelet with a -system reserved memory of at least 2GB to account for Windows, Docker, and -Kubernetes processes. - -##### CPU Reservations - -To account for Windows, Docker and other Kubernetes host processes it is -recommended to reserve a percentage of CPU so they are able to respond to -events. This value needs to be scaled based on the number of CPU cores -available on the Windows node.To determine this percentage a user should -identify the maximum pod density for each of their nodes and monitor the CPU -usage of the system services choosing a value that meets their workload needs. - -Keeping CPU usage within reasonable bounds is possible using the kubelet -parameters `--kubelet-reserve` and/or `--system-reserve` to account for CPU -usage on the node (outside of containers). This reduces -[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). - -#### Feature Restrictions - -* TerminationGracePeriod: not implemented -* Single file mapping: to be implemented with CRI-ContainerD -* Termination message: to be implemented with CRI-ContainerD -* Privileged Containers: not currently supported in Windows containers -* HugePages: not currently supported in Windows containers -* The existing node problem detector is Linux-only and requires privileged - containers. In general, we don't expect this to be used on Windows because - privileged containers are not supported -* Not all features of shared namespaces are supported (see API section for - more details) - -#### Difference in behavior of flags when compared to Linux - -The behavior of the following kubelet flags is different on Windows nodes as described below: - -* `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update - Node Allocatable - -* Eviction by using `--enforce-node-allocable` is not implemented. - -* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented. - -* `MemoryPressure` Condition is not implemented. - -* There are no OOM eviction actions taken by the kubelet. - -* Kubelet running on the windows node does not have memory restrictions. - `--kubelet-reserve` and `--system-reserve` do not set limits on kubelet or - processes running on the host. This means kubelet or a process on the host - could cause memory resource starvation outside the node-allocatable and - scheduler - -* An additional flag to set the priority of the kubelet process is available - on the Windows nodes called `--windows-priorityclass`. This flag allows - kubelet process to get more CPU time slices when compared to other processes - running on the Windows host. More information on the allowable values and - their meaning is available at - [Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). - In order for kubelet to always have enough CPU cycles it is recommended to set - this flag to `ABOVE_NORMAL_PRIORITY_CLASS` and above. - -#### Storage - -Windows has a layered filesystem driver to mount container layers and create a -copy filesystem based on NTFS. All file paths in the container are resolved -only within the context of that container. - -* With Docker Volume mounts can only target a directory in the container, and - not an individual file. This limitation does not exist with CRI-containerD. - -* Volume mounts cannot project files or directories back to the host - filesystem - -* Read-only filesystems are not supported because write access is always - required for the Windows registry and SAM database. However, read-only - volumes are supported - -* Volume user-masks and permissions are not available. Because the SAM is not - shared between the host & container, there's no mapping between them. All - permissions are resolved within the context of the container - -As a result, the following storage functionality is not supported on Windows nodes: - -* Volume subpath mounts. Only the entire volume can be mounted in a Windows container. -* Subpath volume mounting for Secrets -* Host mount projection -* DefaultMode (due to UID/GID dependency) -* Read-only root filesystem. Mapped volumes still support readOnly -* Block device mapping -* Memory as the storage medium -* File system features like uui/guid, per-user Linux filesystem permissions -* NFS based storage/volume support -* Expanding the mounted volume (resizefs) - -#### Networking {#networking-limitations} - -Windows Container Networking differs in some important ways from Linux -networking. The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) -contains additional details and background. - -The Windows host networking service and virtual switch implement namespacing -and can create virtual NICs as needed for a pod or container. However, many -configurations such as DNS, routes, and metrics are stored in the Windows -registry database rather than /etc/... files as they are on Linux. The Windows -registry for the container is separate from that of the host, so concepts like -mapping /etc/resolv.conf from the host into a container don't have the same -effect they would on Linux. These must be configured using Windows APIs run in -the context of that container. Therefore CNI implementations need to call the -HNS instead of relying on file mappings to pass network details into the pod -or container. - -The following networking functionality is not supported on Windows nodes - -* Host networking mode is not available for Windows pods. - -* Local NodePort access from the node itself fails (works for other nodes or - external clients). - -* Accessing service VIPs from nodes will be available with a future release of - Windows Server. - -* A single service can only support up to 64 backend pods / unique destination IPs. - -* Overlay networking support in kube-proxy is a beta feature. In addition, it - requires [KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) - to be installed on Windows Server 2019. - -* Local Traffic Policy in non-DSR mode. - -* Windows containers connected to overlay networks do not support - communicating over the IPv6 stack. There is outstanding Windows platform - work required to enable this network driver to consume IPv6 addresses and - subsequent Kubernetes work in kubelet, kube-proxy, and CNI plugins. - -* Outbound communication using the ICMP protocol via the win-overlay, - win-bridge, and Azure-CNI plugin. Specifically, the Windows data plane - ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) - doesn't support ICMP packet transpositions. This means: - - * ICMP packets directed to destinations within the same network (e.g. pod to - pod communication via ping) work as expected and without any limitations - - * TCP/UDP packets work as expected and without any limitations - - * ICMP packets directed to pass through a remote network (e.g. pod to - external internet communication via ping) cannot be transposed and thus - will not be routed back to their source - - * Since TCP/UDP packets can still be transposed, one can substitute - `ping ` with `curl ` to be able to debug connectivity - to the outside world. - -These features were added in Kubernetes v1.15: - -* `kubectl port-forward` - -##### CNI Plugins - -* Windows reference network plugins `win-bridge` and `win-overlay` do not - currently implement [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) - v0.4.0 due to missing "CHECK" implementation. - -* The Flannel VXLAN CNI has the following limitations on Windows: - - 1. Node-pod connectivity isn't possible by design. It's only possible for - local pods with Flannel v0.12.0 (or higher). - - 1. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation - is being worked on and will be overcome in a future release (open-source - flannel changes). See the official - [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) - backend docs for more details on these parameters. - -##### DNS {#dns-limitations} - -* ClusterFirstWithHostNet is not supported for DNS. Windows treats all names - with a '.' as a FQDN and skips PQDN resolution - -* On Linux, you have a DNS suffix list, which is used when trying to resolve - PQDNs. On Windows, we only have 1 DNS suffix, which is the DNS suffix - associated with that pod's namespace (mydns.svc.cluster.local for example). - Windows can resolve FQDNs and services or names resolvable with only that - suffix. For example, a pod spawned in the default namespace, will have the DNS - suffix `default.svc.cluster.local`. On a Windows pod, you can resolve both - `kubernetes.default.svc.cluster.local` and `kubernetes`, but not the - in-betweens, like `kubernetes.default` or `kubernetes.default.svc`. - -* On Windows, there are multiple DNS resolvers that can be used. As these come - with slightly different behaviors, using the `Resolve-DNSName` utility for - name query resolutions is recommended. - -##### IPv6 - -Kubernetes on Windows does not support single-stack "IPv6-only" networking. -However,dual-stack IPv4/IPv6 networking for pods and nodes with single-family -services is supported. -See [IPv4/IPv6 dual-stack networking](#ipv4ipv6-dual-stack) for more details. - ##### Session affinity Setting the maximum session sticky time for Windows services using `service.spec.sessionAffinityConfig.clientIP.timeoutSeconds` is not supported. -##### Security +#### DNS {#dns-limitations} -Secrets are written in clear text on the node's volume (as compared to -tmpfs/in-memory on linux). This means customers have to do two things: +* ClusterFirstWithHostNet is not supported for DNS. Windows treats all names with a + `.` as a FQDN and skips FQDN resolution +* On Linux, you have a DNS suffix list, which is used when trying to resolve PQDNs. On + Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that + pod's namespace (mydns.svc.cluster.local for example). Windows can resolve FQDNs + and services or names resolvable with just that suffix. For example, a pod spawned + in the default namespace, will have the DNS suffix **default.svc.cluster.local**. + Inside a Windows pod, you can resolve both **kubernetes.default.svc.cluster.local** + and **kubernetes**, but not the in-betweens, like **kubernetes.default** or + **kubernetes.default.svc**. +* On Windows, there are multiple DNS resolvers that can be used. As these come with + slightly different behaviors, using the `Resolve-DNSName` utility for name query + resolutions is recommended. -1. Use file ACLs to secure the secrets file location -1. Use volume-level encryption using - [BitLocker](https://docs.microsoft.com/en-us/windows/security/information-protection/bitlocker/bitlocker-how-to-deploy-on-windows-server) +#### IPv6 networking -[RunAsUsername](/docs/tasks/configure-pod-container/configure-runasusername) -can be specified for Windows Pod's or Container's to execute the Container -processes as a node-default user. This is roughly equivalent to -[RunAsUser](/docs/concepts/policy/pod-security-policy/#users-and-groups). +Kubernetes on Windows does not support single-stack "IPv6-only" networking. However, +dual-stack IPv4/IPv6 networking for pods and nodes with single-family services +is supported. -Linux specific pod security context privileges such as SELinux, AppArmor, -Seccomp, Capabilities (POSIX Capabilities), and others are not supported. +You can enable IPv4/IPv6 dual-stack networking for `l2bridge` networks using the +`IPv6DualStack` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +See [enable IPv4/IPv6 dual stack](/docs/concepts/services-networking/dual-stack#enable-ipv4ipv6-dual-stack) for more details. -In addition, as mentioned already, privileged containers are not supported on -Windows. +{{< note >}} +Overlay (VXLAN) networks on Windows do not support dual-stack networking. +{{< /note >}} -#### API +### Persistent storage {#compatibility-storage} -There are no differences in how most of the Kubernetes APIs work for Windows. -The subtleties around what's different come down to differences in the OS and -container runtime. In certain situations, some properties on workload APIs -such as Pod or Container were designed with an assumption that they are -implemented on Linux, failing to run on Windows. +Windows has a layered filesystem driver to mount container layers and create a copy +filesystem based on NTFS. All file paths in the container are resolved only within +the context of that container. + +* With Docker, volume mounts can only target a directory in the container, and not + an individual file. This limitation does not exist with CRI-containerD runtime. +* Volume mounts cannot project files or directories back to the host filesystem. +* Read-only filesystems are not supported because write access is always required + for the Windows registry and SAM database. However, read-only volumes are supported. +* Volume user-masks and permissions are not available. Because the SAM is not shared + between the host & container, there's no mapping between them. All permissions are + resolved within the context of the container. + +As a result, the following storage functionality is not supported on Windows nodes: + +* Volume subpath mounts: only the entire volume can be mounted in a Windows container +* Subpath volume mounting for Secrets +* Host mount projection +* Read-only root filesystem (mapped volumes still support `readOnly`) +* Block device mapping +* Memory as the storage medium (for example, `emptyDir.medium` set to `Memory`) +* File system features like uid/gid; per-user Linux filesystem permissions +* DefaultMode (due to UID/GID dependency) +* NFS based storage/volume support +* Expanding the mounted volume (resizefs) + +Kubernetes {{< glossary_tooltip text="volumes" term_id="volume" >}} enable complex +applications, with data persistence and Pod volume sharing requirements, to be deployed +on Kubernetes. Management of persistent volumes associated with a specific storage +back-end or protocol includes actions such as provisioning/de-provisioning/resizing +of volumes, attaching/detaching a volume to/from a Kubernetes node and +mounting/dismounting a volume to/from individual containers in a pod that needs to +persist data. + +The code implementing these volume management actions for a specific storage back-end +or protocol is shipped in the form of a Kubernetes volume +[plugin](/docs/concepts/storage/volumes/#types-of-volumes). +The following broad classes of Kubernetes volume plugins are supported on Windows: + +##### In-tree volume plugins + +Code associated with in-tree volume plugins ship as part of the core Kubernetes code +base. Deployment of in-tree volume plugins do not require installation of additional +scripts or deployment of separate containerized plugin components. These plugins can +handle provisioning/de-provisioning and resizing of volumes in the storage backend, +attaching/detaching of volumes to/from a Kubernetes node and mounting/dismounting a +volume to/from individual containers in a pod. The following in-tree plugins support +persistent storage on Windows nodes: + +* [`awsElasticBlockStore`](/docs/concepts/storage/volumes/#awselasticblockstore) +* [`azureDisk`](/docs/concepts/storage/volumes/#azuredisk) +* [`azureFile`](/docs/concepts/storage/volumes/#azurefile) +* [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk) +* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume) + +#### FlexVolume plugins + +Code associated with [FlexVolume](/docs/concepts/storage/volumes/#flexVolume) +plugins ship as out-of-tree scripts or binaries that need to be deployed directly +on the host. FlexVolume plugins handle attaching/detaching of volumes to/from a +Kubernetes node and mounting/dismounting a volume to/from individual containers +in a pod. Provisioning/De-provisioning of persistent volumes associated +with FlexVolume plugins may be handled through an external provisioner that +is typically separate from the FlexVolume plugins. The following FlexVolume +[plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), +deployed as PowerShell scripts on the host, support Windows nodes: + +* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd) +* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd) + +#### CSI plugins + +{{< feature-state for_k8s_version="v1.19" state="beta" >}} + +Code associated with {{< glossary_tooltip text="CSI" term_id="csi" >}} plugins ship +as out-of-tree scripts and binaries that are typically distributed as container +images and deployed using standard Kubernetes constructs like DaemonSets and +StatefulSets. +CSI plugins handle a wide range of volume management actions in Kubernetes: +provisioning/de-provisioning/resizing of volumes, attaching/detaching of volumes +to/from a Kubernetes node and mounting/dismounting a volume to/from individual +containers in a pod, backup/restore of persistent data using snapshots and cloning. +CSI plugins typically consist of node plugins (that run on each node as a DaemonSet) +and controller plugins. + +CSI node plugins (especially those associated with persistent volumes exposed as +either block devices or over a shared file-system) need to perform various privileged +operations like scanning of disk devices, mounting of file systems, etc. These +operations differ for each host operating system. For Linux worker nodes, containerized +CSI node plugins are typically deployed as privileged containers. For Windows worker +nodes, privileged operations for containerized CSI node plugins is supported using +[csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed, +stand-alone binary that needs to be pre-installed on each Windows node. + +For more details, refer to the deployment guide of the CSI plugin you wish to deploy. + +### Command line options for the kubelet {#kubelet-compatibility} + +The behavior of some kubelet command line options behave differently on Windows, as described below: + +* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process (see [CPU resource management](#resource-management-cpu)) +* The `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +* Eviction by using `--enforce-node-allocable` is not implemented +* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented +* A kubelet running on a Windows node does not have memory + restrictions. `--kubelet-reserve` and `--system-reserve` do not set limits on + kubelet or processes running on the host. This means kubelet or a process on the host + could cause memory resource starvation outside the node-allocatable and scheduler. +* The `MemoryPressure` Condition is not implemented +* The kubelet does not take OOM eviction actions + +### API compatibility {#api} + +There are no differences in how most of the Kubernetes APIs work for Windows. The +subtleties around what's different come down to differences in the OS and container +runtime. In certain situations, some properties on workload resources were designed +under the assumption that they would be implemented on Linux, and fail to run on Windows. At a high level, these OS concepts are different: -* Identity - Linux uses userID (UID) and groupID (GID) which are represented - as integer types. User and group names are not canonical - they are an alias - in `/etc/groups` or `/etc/passwd` back to UID+GID. Windows uses a larger - binary security identifier (SID) which is stored in the Windows Security - Access Manager (SAM) database. This database is not shared between the host - and containers, or between containers. - -* File permissions - Windows uses an access control list based on SIDs, rather - than a bitmask of permissions and UID+GID - -* File paths - convention on Windows is to use `\` instead of `/`. The Go IO - libraries accept both types of file path separators. However, when you're - setting a path or command line that's interpreted inside a container, `\` may - be needed. - +* Identity - Linux uses userID (UID) and groupID (GID) which + are represented as integer types. User and group names + are not canonical - they are just an alias in `/etc/groups` + or `/etc/passwd` back to UID+GID. Windows uses a larger binary + [security identifier](https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-identifiers) (SID) + which is stored in the Windows Security Access Manager (SAM) database. This + database is not shared between the host and containers, or between containers. +* File permissions - Windows uses an access control list based on (SIDs), whereas + POSIX systems such as Linux use a bitmask based on object permissions and UID+GID, + plus _optional_ access control lists. +* File paths - the convention on Windows is to use `\` instead of `/`. The Go IO + libraries typically accept both and just make it work, but when you're setting a + path or command line that's interpreted inside a container, `\` may be needed. * Signals - Windows interactive apps handle termination differently, and can implement one or more of these: - - * A UI thread handles well-defined messages including `WM_CLOSE` - - * Console apps handle ctrl-c or ctrl-break using a Control Handler - + * A UI thread handles well-defined messages including `WM_CLOSE`. + * Console apps handle Ctrl-C or Ctrl-break using a Control Handler. * Services register a Service Control Handler function that can accept - `SERVICE_CONTROL_STOP` control codes + `SERVICE_CONTROL_STOP` control codes. -Exit Codes follow the same convention where 0 is success, nonzero is failure. -The specific error codes may differ across Windows and Linux. However, exit -codes passed from the Kubernetes components (kubelet, kube-proxy) are -unchanged. +Container exit codes follow the same convention where 0 is success, and nonzero is failure. +The specific error codes may differ across Windows and Linux. However, exit codes +passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. -##### V1.Container +##### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} -* V1.Container.ResourceRequirements.limits.cpu and - V1.Container.ResourceRequirements.limits.memory - Windows doesn't use hard - limits for CPU allocations. Instead, a share system is used. The existing - fields based on millicores are scaled into relative shares that are followed - by the Windows scheduler. - See [kuberuntime/helpers_windows.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/helpers_windows.go), - and [resource controls in Microsoft docs](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/resource-controls) +The following list documents differences between how Pod container specifications +work between Windows and Linux: - * Huge pages are not implemented in the Windows container runtime, and are - not available. They require - [asserting a user privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support) - that's not configurable for containers. - -* V1.Container.ResourceRequirements.requests.cpu and - V1.Container.ResourceRequirements.requests.memory - Requests are subtracted +* `limits.cpu` and `limits.memory` - Windows doesn't use hard limits + for CPU allocations. Instead, a share system is used. + The fields based on millicores are scaled into + relative shares that are followed by the Windows scheduler + See [`kuberuntime/helpers_windows.go`](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/helpers_windows.go), + and [Implementing resource controls for Windows containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/resource-controls) + in Microsoft's virtualization documentation. +* Huge pages are not implemented in the Windows container + runtime, and are not available. They require [asserting a user + privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support) + that's not configurable for containers. +* `requests.cpu` and `requests.memory` - requests are subtracted from node available resources, so they can be used to avoid overprovisioning a - node. However, they cannot be used to guarantee resources in an - overprovisioned node. They should be applied to all containers as a best - practice if the operator wants to avoid overprovisioning entirely. + node. However, they cannot be used to guarantee resources in an overprovisioned + node. They should be applied to all containers as a best practice if the operator + wants to avoid overprovisioning entirely. +* `securityContext.allowPrivilegeEscalation` - + not possible on Windows; none of the capabilities are hooked up +* `securityContext.capabilities` - + POSIX capabilities are not implemented on Windows +* `securityContext.privileged` - + Windows doesn't support privileged containers +* `securityContext.procMount` - + Windows doesn't have a `/proc` filesystem +* `securityContext.readOnlyRootFilesystem` - + not possible on Windows; write access is required for registry & system + processes to run inside the container +* `EcurityContext.runAsGroup` - + not possible on Windows as there is no GID support +* `ecurityContext.runAsNonRoot` - + Windows does not have a root user. The closest equivalent is `ContainerAdministrator` + which is an identity that doesn't exist on the node. +* `securityContext.runAsUser` - + use [`runAsUsername`](/docs/tasks/configure-pod-container/configure-runasusername) + instead +* `securityContext.seLinuxOptions` - + not possible on Windows as SELinux is Linux-specific +* `terminationMessagePath` - + this has some limitations in that Windows doesn't support mapping single files. The + default value is `/dev/termination-log`, which does work because it does not + exist on Windows by default. -* V1.Container.SecurityContext.allowPrivilegeEscalation - not possible on - Windows, none of the capabilities are hooked up +##### Field compatibility for Pod specifications {#compatibility-v1-pod} -* V1.Container.SecurityContext.Capabilities - POSIX capabilities are not - implemented on Windows +The following list documents differences between how Pod specifications work between Windows and Linux: -* V1.Container.SecurityContext.privileged - Windows doesn't support privileged - containers - -* V1.Container.SecurityContext.procMount - Windows doesn't have a /proc filesystem - -* V1.Container.SecurityContext.readOnlyRootFilesystem - not possible on - Windows, write access is required for registry & system processes to run - inside the container - -* V1.Container.SecurityContext.runAsGroup - not possible on Windows, no GID support - -* V1.Container.SecurityContext.runAsNonRoot - Windows does not have a root - user. The closest equivalent is ContainerAdministrator which is an identity - that doesn't exist on the node. - -* V1.Container.SecurityContext.runAsUser - not possible on Windows, no UID - support as int. - -* V1.Container.SecurityContext.seLinuxOptions - not possible on Windows, no SELinux - -* V1.Container.terminationMessagePath - this has some limitations in that - Windows doesn't support mapping single files. The default value is - `/dev/termination-log`, which does work because it does not exist on Windows by - default. - -##### V1.Pod - -* V1.Pod.hostIPC, v1.pod.hostpid - host namespace sharing is not possible on Windows - -* V1.Pod.hostNetwork - There is no Windows OS support to share the host network - -* V1.Pod.dnsPolicy - `ClusterFirstWithHostNet` is not supported because Host - Networking is not supported on Windows. - -* V1.Pod.podSecurityContext - see V1.PodSecurityContext below - -* V1.Pod.shareProcessNamespace - this is a beta feature, and depends on Linux - namespaces which are not implemented on Windows. Windows cannot share - process namespaces or the container's root filesystem. Only the network can be - shared. - -* V1.Pod.terminationGracePeriodSeconds - this is not fully implemented in - Docker on Windows, see: - [reference](https://github.com/moby/moby/issues/25982). The behavior today is - that the `ENTRYPOINT` process is sent `CTRL_SHUTDOWN_EVENT`, then Windows waits 5 - seconds by default, and finally shuts down all processes using the normal - Windows shutdown behavior. The 5 second default is actually in the Windows - registry [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), +* `hostIPC` and `hostpid` - host namespace sharing is not possible on Windows +* `hostNetwork` - There is no Windows OS support to share the host network +* `dnsPolicy` - setting the Pod `dnsPolicy` to `ClusterFirstWithHostNet` is + not supported on Windows because host networking is not provided. Pods always + run with a container network. +* `podSecurityContext` (see below) +* `shareProcessNamespace` - this is a beta feature, and depends on Linux namespaces + which are not implemented on Windows. Windows cannot share process namespaces or + the container's root filesystem. Only the network can be shared. +* `terminationGracePeriodSeconds` - this is not fully implemented in Docker on Windows, + see the [GitHub issue](https://github.com/moby/moby/issues/25982). + The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, + then Windows waits 5 seconds by default, and finally shuts down + all processes using the normal Windows shutdown behavior. The 5 + second default is actually in the Windows registry + [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), so it can be overridden when the container is built. +* `volumeDevices` - this is a beta feature, and is not implemented on Windows. + Windows cannot attach raw block devices to pods. +* `volumes` + * If you define an `emptyDir` volume, you cannot set its volume source to `memory`. +* You cannot enable `mountPropagation` for volume mounts as this is not + supported on Windows. -* V1.Pod.volumeDevices - this is a beta feature, and is not implemented on - Windows. Windows cannot attach raw block devices to pods. +##### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} -* V1.Pod.volumes - EmptyDir, Secret, ConfigMap, HostPath - all work and have - tests in TestGrid +None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows. - * V1.emptyDirVolumeSource - the Node default medium is disk on Windows. - Memory is not supported, as Windows does not have a built-in RAM disk. +### Node problem detector -* V1.VolumeMount.mountPropagation - mount propagation is not supported on Windows. +The node problem detector (see +[Monitor Node Health](/docs/tasks/debug-application-cluster/monitor-node-health/)) +is not compatible with Windows. -##### V1.PodSecurityContext +### Pause container -None of the PodSecurityContext fields work on Windows. They're listed here for -reference. +In a Kubernetes Pod, an infrastructure or “pause” container is first created +to host the container. In Linux, the cgroups and namespaces that make up a pod +need a process to maintain their continued existence; the pause process provides +this. Containers that belong to the same pod, including infrastructure and worker +containers, share a common network endpoint (same IPv4 and / or IPv6 address, same +network port spaces). Kubernetes uses pause containers to allow for worker containers +crashing or restarting without losing any of the networking configuration. -* V1.PodSecurityContext.SELinuxOptions - SELinux is not available on Windows +Kubernetes maintains a multi-architecture image that includes support for Windows. +For Kubernetes v1.22 the recommended pause image is `k8s.gcr.io/pause:3.5`. +The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause) +is available on GitHub. -* V1.PodSecurityContext.RunAsUser - provides a UID, not available on Windows +Microsoft maintains a different multi-architecture image, with Linux and Windows +amd64 support, that you can find as `mcr.microsoft.com/oss/kubernetes/pause:3.5`. +This image is built from the same source as the Kubernetes maintained image but +all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft. +The Kubernetes project recommends using the Microsoft maintained image if you are +deploying to a production or production-like environment that requires signed +binaries. -* V1.PodSecurityContext.RunAsGroup - provides a GID, not available on Windows +### Container runtimes {#container-runtime} -* V1.PodSecurityContext.RunAsNonRoot - Windows does not have a root user. The - closest equivalent is ContainerAdministrator which is an identity that - doesn't exist on the node. +You need to install a +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} +into each node in the cluster so that Pods can run there. -* V1.PodSecurityContext.SupplementalGroups - provides GID, not available on Windows +The following container runtimes work with Windows: -* V1.PodSecurityContext.Sysctls - these are part of the Linux sysctl - interface. There's no equivalent on Windows. +{{% thirdparty-content %}} -#### Operating System Version Restrictions +#### cri-containerd -Windows has strict compatibility rules, where the host OS version must match -the container base image OS version. Only Windows containers with a container -operating system of Windows Server 2019 are supported. Hyper-V isolation of -containers, enabling some backward compatibility of Windows container image -versions, is planned for a future release. +{{< feature-state for_k8s_version="v1.20" state="stable" >}} -## Getting Help and Troubleshooting {#troubleshooting} +You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ +as the container runtime for Kubernetes nodes that run Windows. -Your main source of help for troubleshooting your Kubernetes cluster should -start with this -[section](/docs/tasks/debug-application-cluster/troubleshooting/). Some -additional, Windows-specific troubleshooting help is included in this section. -Logs are an important element of troubleshooting issues in Kubernetes. Make -sure to include them any time you seek troubleshooting assistance from other -contributors. Follow the instructions in the SIG-Windows -[contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs). +Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd). -* How do I know start.ps1 completed successfully? +{{< note >}} +There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations) +when using GMSA with containerd to access Windows network shares, which requires a +kernel patch. +{{< /note >}} - You should see kubelet, kube-proxy, and (if you chose Flannel as your - networking solution) flanneld host-agent processes running on your node, with - running logs being displayed in separate PowerShell windows. In addition to - this, your Windows node should be listed as "Ready" in your Kubernetes - cluster. +#### Docker EE -* Can I configure the Kubernetes node processes to run in the background as services? +{{< feature-state for_k8s_version="v1.14" state="stable" >}} - Kubelet and kube-proxy are already configured to run as native Windows - Services, offering resiliency by re-starting the services automatically in the - event of failure (for example a process crash). You have two options for - configuring these node components as services. +[Docker EE](https://docs.mirantis.com/containers/v3.0/dockeree-products/dee-intro.html)-basic 19.03+ is available as a container runtime for all Windows Server versions. This works with the legacy dockershim adapter. - * As native Windows Services +See [Install Docker](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/deploy-containers-on-server#install-docker) for more information. - Kubelet & kube-proxy can be run as native Windows Services using `sc.exe`. +## Windows OS version compatibility {#windows-os-version-support} - ```powershell - # Create the services for kubelet and kube-proxy in two separate commands - sc.exe create binPath= " --service " +On Windows nodes, strict compatibility rules apply where the host OS version must +match the container base image OS version. Only Windows containers with a container +operating system of Windows Server 2019 are fully supported. - # Please note that if the arguments contain spaces, they must be escaped. - sc.exe create kubelet binPath= "C:\kubelet.exe --service --hostname-override 'minion' " +For Kubernetes v1.22, operating system compatibility for Windows nodes (and Pods) +is as follows: - # Start the services - Start-Service kubelet - Start-Service kube-proxy +Windows Server LTSC release +: Windows Server 2019 - # Stop the service - Stop-Service kubelet (-Force) - Stop-Service kube-proxy (-Force) +Windows Server SAC release +: Windows Server version 2004, Windows Server version 20H2 - # Query the service status - Get-Service kubelet - Get-Service kube-proxy - ``` +The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) also applies. - * Using nssm.exe +## Security for Windows nodes {#security} - You can also always use alternative service managers like - [`nssm.exe`](https://nssm.cc/) to run these processes (flanneld, kubelet & - kube-proxy) in the background for you. You can use this - [sample script](https://github.com/Microsoft/SDN/tree/master/Kubernetes/flannel/register-svc.ps1), - leveraging `nssm.exe` to register kubelet, kube-proxy, and `flanneld.exe` - to run as Windows services in the background. +On Windows, data from Secrets are written out in clear text onto the node's local +storage (as compared to using tmpfs / in-memory filesystems on Linux). As a cluster +operator, you should take both of the following additional measures: - ```powershell - register-svc.ps1 -NetworkMode -ManagementIP -ClusterCIDR -KubeDnsServiceIP -LogDir - ``` +1. Use file ACLs to secure the Secrets' file location. +1. Apply volume-level encryption using [BitLocker](https://docs.microsoft.com/en-us/windows/security/information-protection/bitlocker/bitlocker-how-to-deploy-on-windows-server). - The parameters are explained below: +[RunAsUsername](/docs/tasks/configure-pod-container/configure-runasusername) +can be specified for Windows Pods or containers to execute the container +processes as a node-default user. This is roughly equivalent to +[RunAsUser](/docs/concepts/policy/pod-security-policy/#users-and-groups). - - `NetworkMode`: The network mode l2bridge (flannel host-gw, also the - default value) or overlay (flannel vxlan) chosen as a network solution - - `ManagementIP`: The IP address assigned to the Windows node. You can use - `ipconfig` to find this. - - `ClusterCIDR`: The cluster subnet range. (Default: 10.244.0.0/16) - - `KubeDnsServiceIP`: The Kubernetes DNS service IP. (Default: 10.96.0.10) - - `LogDir`: The directory where kubelet and kube-proxy logs are redirected - into their respective output files. (Default value C:\k) +Linux-specific pod security context privileges such as SELinux, AppArmor, Seccomp, or capabilities (POSIX capabilities), and others are not supported. - If the above referenced script is not suitable, you can manually configure - `nssm.exe` using the following examples. +Privileged containers are [not supported](#compatibility-v1-pod-spec-containers-securitycontext) on Windows. - Register flanneld.exe: - - ```powershell - nssm install flanneld C:\flannel\flanneld.exe - nssm set flanneld AppParameters --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 - nssm set flanneld AppEnvironmentExtra NODE_NAME= - nssm set flanneld AppDirectory C:\flannel - nssm start flanneld - ``` - - Register kubelet.exe: - - ```powershell - nssm install kubelet C:\k\kubelet.exe - nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=k8s.gcr.io/pause:3.5 --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config - nssm set kubelet AppDirectory C:\k - nssm start kubelet - ``` - - Register kube-proxy.exe (l2bridge / host-gw): - - ```powershell - nssm install kube-proxy C:\k\kube-proxy.exe - nssm set kube-proxy AppDirectory c:\k - nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --hostname-override=--kubeconfig=c:\k\config --enable-dsr=false --log-dir= --logtostderr=false - nssm.exe set kube-proxy AppEnvironmentExtra KUBE_NETWORK=cbr0 - nssm set kube-proxy DependOnService kubelet - nssm start kube-proxy - ``` - - Register kube-proxy.exe (overlay / vxlan): - - ```powershell - nssm install kube-proxy C:\k\kube-proxy.exe - nssm set kube-proxy AppDirectory c:\k - nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true" --hostname-override= --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip= --enable-dsr=false --log-dir= --logtostderr=false - nssm set kube-proxy DependOnService kubelet - nssm start kube-proxy - ``` - - For initial troubleshooting, you can use the following flags in - [`nssm.exe`](https://nssm.cc/) to redirect stdout and stderr to a output file: - - ```powershell - nssm set AppStdout C:\k\mysvc.log - nssm set AppStderr C:\k\mysvc.log - ``` - - For additional details, see official [nssm usage](https://nssm.cc/usage) docs. - -* My Windows Pods do not have network connectivity - - If you are using virtual machines, ensure that MAC spoofing is enabled on - all the VM network adapter(s). - -* My Windows Pods cannot ping external resources - - Windows Pods do not have outbound rules programmed for the ICMP protocol - today. However, TCP/UDP is supported. When trying to demonstrate connectivity - to resources outside of the cluster, please substitute `ping ` with - corresponding `curl ` commands. - - If you are still facing problems, most likely your network configuration in - [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) - deserves some extra attention. You can always edit this static file. The - configuration update will apply to any newly created Kubernetes resources. - - One of the Kubernetes networking requirements (see - [Kubernetes network model](/docs/concepts/cluster-administration/networking/)) - is for cluster communication to occur without NAT internally. To honor this - requirement, there is an - [ExceptionList](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf#L20) - for all the communication where we do not want outbound NAT to occur. However, - this also means that you need to exclude the external IP you are trying to - query from the ExceptionList. Only then will the traffic originating from your - Windows pods be SNAT'ed correctly to receive a response from the outside - world. In this regard, your ExceptionList in `cni.conf` should look as - follows: - - ```conf - "ExceptionList": [ - "10.244.0.0/16", # Cluster subnet - "10.96.0.0/12", # Service subnet - "10.127.130.0/24" # Management (host) subnet - ] - ``` - -* My Windows node cannot access NodePort service - - Local NodePort access from the node itself fails. This is a known - limitation. NodePort access works from other nodes or external clients. - -* vNICs and HNS endpoints of containers are being deleted - - This issue can be caused when the `hostname-override` parameter is not - passed to - [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/). - To resolve it, users need to pass the hostname to kube-proxy as follows: - - ```powershell - C:\k\kube-proxy.exe --hostname-override=$(hostname) - ``` - -* With flannel my nodes are having issues after rejoining a cluster - - Whenever a previously deleted node is being re-joined to the cluster, - flannelD tries to assign a new pod subnet to the node. Users should remove the - old pod subnet configuration files in the following paths: - - ```powershell - Remove-Item C:\k\SourceVip.json - Remove-Item C:\k\SourceVipRequest.json - ``` - -* After launching `start.ps1`, flanneld is stuck in "Waiting for the Network - to be created" - - There are numerous reports of this - [issue](https://github.com/coreos/flannel/issues/1066); most likely it is a - timing issue for when the management IP of the flannel network is set. A - workaround is to relaunch start.ps1 or relaunch it manually as follows: - - ```powershell - PS C:> [Environment]::SetEnvironmentVariable("NODE_NAME", "") - PS C:> C:\flannel\flanneld.exe --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 - ``` - -* My Windows Pods cannot launch because of missing `/run/flannel/subnet.env` - - This indicates that Flannel didn't launch correctly. You can either try to - restart flanneld.exe or you can copy the files over manually from - `/run/flannel/subnet.env` on the Kubernetes master to - `C:\run\flannel\subnet.env` on the Windows worker node and modify the - `FLANNEL_SUBNET` row to a different number. For example, if node subnet - 10.244.4.1/24 is desired: - - ```none - FLANNEL_NETWORK=10.244.0.0/16 - FLANNEL_SUBNET=10.244.4.1/24 - FLANNEL_MTU=1500 - FLANNEL_IPMASQ=true - ``` - -* My Windows node cannot access my services using the service IP - - This is a known limitation of the current networking stack on Windows. - Windows Pods are able to access the service IP however. - -* No network adapter is found when starting kubelet - - The Windows networking stack needs a virtual adapter for Kubernetes - networking to work. If the following commands return no results (in an admin - shell), virtual network creation — a necessary prerequisite for Kubelet to - work — has failed: - - ```powershell - Get-HnsNetwork | ? Name -ieq "cbr0" - Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*" - ``` - - Often it is worthwhile to modify the - [InterfaceName](https://github.com/microsoft/SDN/blob/master/Kubernetes/flannel/start.ps1#L7) - parameter of the start.ps1 script, in cases where the host's network adapter - isn't "Ethernet". Otherwise, consult the output of the `start-kubelet.ps1` - script to see if there are errors during virtual network creation. - -* My Pods are stuck at "Container Creating" or restarting over and over - - Check that your pause image is compatible with your OS version. The - [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) - assume that both the OS and the containers are version 1803. If you have a - later version of Windows, such as an Insider build, you need to adjust the - images accordingly. Please refer to the Microsoft's - [Docker repository](https://hub.docker.com/u/microsoft/) for images. - Regardless, both the pause image Dockerfile and the sample service expect - the image to be tagged as :latest. +## Getting help and troubleshooting {#troubleshooting} -* DNS resolution is not properly working +Your main source of help for troubleshooting your Kubernetes cluster should start +with the [Troubleshooting](/docs/tasks/debug-application-cluster/troubleshooting/) +page. - Check the [DNS limitations for Windows](#dns-limitations). +Some additional, Windows-specific troubleshooting help is included +in this section. Logs are an important element of troubleshooting +issues in Kubernetes. Make sure to include them any time you seek +troubleshooting assistance from other contributors. Follow the +instructions in the +SIG Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs). -* `kubectl port-forward` fails with "unable to do port forwarding: wincat not found" +### Node-level troubleshooting {#troubleshooting-node} - Port forwarding support for Windows requires wincat.exe to be available in the - [pause infrastructure container](#pause-image). - Ensure you are using a supported image that is compatable with your Windows OS version. - If you would like to build your own pause infrastructure container be sure to include - [wincat](https://github.com/kubernetes/kubernetes/tree/master/build/pause/windows/wincat). +1. How do I know `start.ps1` completed successfully? -* My Kubernetes installation is failing because my Windows Server node is - behind a proxy + You should see kubelet, kube-proxy, and (if you chose Flannel as your networking + solution) flanneld host-agent processes running on your node, with running logs + being displayed in separate PowerShell windows. In addition to this, your Windows + node should be listed as "Ready" in your Kubernetes cluster. - If you are behind a proxy, the following PowerShell environment variables - must be defined: - - ```PowerShell - [Environment]::SetEnvironmentVariable("HTTP_PROXY", "http://proxy.example.com:80/", [EnvironmentVariableTarget]::Machine) - [Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.example.com:443/", [EnvironmentVariableTarget]::Machine) - ``` - -* What is a `pause` container? +1. Can I configure the Kubernetes node processes to run in the background as services? - In a Kubernetes Pod, an infrastructure or "pause" container is first created - to host the container endpoint. Containers that belong to the same pod, - including infrastructure and worker containers, share a common network - namespace and endpoint (same IP and port space). Pause containers are needed - to accommodate worker containers crashing or restarting without losing any of - the networking configuration. + The kubelet and kube-proxy are already configured to run as native Windows Services, + offering resiliency by re-starting the services automatically in the event of + failure (for example a process crash). You have two options for configuring these + node components as services. - Refer to the [pause image](#pause-image) section to find the recommended version - of the pause image. + 1. As native Windows Services + + You can run the kubelet and kube-proxy as native Windows Services using `sc.exe`. + + ```powershell + # Create the services for kubelet and kube-proxy in two separate commands + sc.exe create binPath= " --service " + + # Please note that if the arguments contain spaces, they must be escaped. + sc.exe create kubelet binPath= "C:\kubelet.exe --service --hostname-override 'minion' " + + # Start the services + Start-Service kubelet + Start-Service kube-proxy + + # Stop the service + Stop-Service kubelet (-Force) + Stop-Service kube-proxy (-Force) + + # Query the service status + Get-Service kubelet + Get-Service kube-proxy + ``` + + 1. Using `nssm.exe` + + You can also always use alternative service managers like + [nssm.exe](https://nssm.cc/) to run these processes (flanneld, + kubelet & kube-proxy) in the background for you. You can use this + [sample script](https://github.com/Microsoft/SDN/tree/master/Kubernetes/flannel/register-svc.ps1), + leveraging nssm.exe to register kubelet, kube-proxy, and flanneld.exe to run + as Windows services in the background. + + ```powershell + register-svc.ps1 -NetworkMode -ManagementIP -ClusterCIDR -KubeDnsServiceIP -LogDir + + # NetworkMode = The network mode l2bridge (flannel host-gw, also the default value) or overlay (flannel vxlan) chosen as a network solution + # ManagementIP = The IP address assigned to the Windows node. You can use ipconfig to find this + # ClusterCIDR = The cluster subnet range. (Default value 10.244.0.0/16) + # KubeDnsServiceIP = The Kubernetes DNS service IP (Default value 10.96.0.10) + # LogDir = The directory where kubelet and kube-proxy logs are redirected into their respective output files (Default value C:\k) + ``` + + If the above referenced script is not suitable, you can manually configure + `nssm.exe` using the following examples. + + ```powershell + # Register flanneld.exe + nssm install flanneld C:\flannel\flanneld.exe + nssm set flanneld AppParameters --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 + nssm set flanneld AppEnvironmentExtra NODE_NAME= + nssm set flanneld AppDirectory C:\flannel + nssm start flanneld + + # Register kubelet.exe + # Microsoft releases the pause infrastructure container at mcr.microsoft.com/oss/kubernetes/pause:1.4.1 + nssm install kubelet C:\k\kubelet.exe + nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.4.1 --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config + nssm set kubelet AppDirectory C:\k + nssm start kubelet + + # Register kube-proxy.exe (l2bridge / host-gw) + nssm install kube-proxy C:\k\kube-proxy.exe + nssm set kube-proxy AppDirectory c:\k + nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --hostname-override=--kubeconfig=c:\k\config --enable-dsr=false --log-dir= --logtostderr=false + nssm.exe set kube-proxy AppEnvironmentExtra KUBE_NETWORK=cbr0 + nssm set kube-proxy DependOnService kubelet + nssm start kube-proxy + + # Register kube-proxy.exe (overlay / vxlan) + nssm install kube-proxy C:\k\kube-proxy.exe + nssm set kube-proxy AppDirectory c:\k + nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true" --hostname-override= --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip= --enable-dsr=false --log-dir= --logtostderr=false + nssm set kube-proxy DependOnService kubelet + nssm start kube-proxy + ``` + + For initial troubleshooting, you can use the following flags in [nssm.exe](https://nssm.cc/) to redirect stdout and stderr to a output file: + + ```powershell + nssm set AppStdout C:\k\mysvc.log + nssm set AppStderr C:\k\mysvc.log + ``` + + For additional details, see [NSSM - the Non-Sucking Service Manager](https://nssm.cc/usage). + +1. My Pods are stuck at "Container Creating" or restarting over and over + + Check that your pause image is compatible with your OS version. The + [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) + assume that both the OS and the containers are version 1803. If you have a later + version of Windows, such as an Insider build, you need to adjust the images + accordingly. See [Pause container](#pause-container) for more details. + +### Network troubleshooting {#troubleshooting-network} + +1. My Windows Pods do not have network connectivity + + If you are using virtual machines, ensure that MAC spoofing is **enabled** on all + the VM network adapter(s). + +1. My Windows Pods cannot ping external resources + + Windows Pods do not have outbound rules programmed for the ICMP protocol. However, + TCP/UDP is supported. When trying to demonstrate connectivity to resources + outside of the cluster, substitute `ping ` with corresponding + `curl ` commands. + + If you are still facing problems, most likely your network configuration in + [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) + deserves some extra attention. You can always edit this static file. The + configuration update will apply to any new Kubernetes resources. + + One of the Kubernetes networking requirements + (see [Kubernetes model](/docs/concepts/cluster-administration/networking/)) is + for cluster communication to occur without + NAT internally. To honor this requirement, there is an + [ExceptionList](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf#L20) + for all the communication where you do not want outbound NAT to occur. However, + this also means that you need to exclude the external IP you are trying to query + from the `ExceptionList`. Only then will the traffic originating from your Windows + pods be SNAT'ed correctly to receive a response from the outside world. In this + regard, your `ExceptionList` in `cni.conf` should look as follows: + + ```conf + "ExceptionList": [ + "10.244.0.0/16", # Cluster subnet + "10.96.0.0/12", # Service subnet + "10.127.130.0/24" # Management (host) subnet + ] + ``` + +1. My Windows node cannot access `NodePort` type Services + + Local NodePort access from the node itself fails. This is a known + limitation. NodePort access works from other nodes or external clients. + +1. vNICs and HNS endpoints of containers are being deleted + + This issue can be caused when the `hostname-override` parameter is not passed to + [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/). To resolve + it, users need to pass the hostname to kube-proxy as follows: + + ```powershell + C:\k\kube-proxy.exe --hostname-override=$(hostname) + ``` + +1. With flannel, my nodes are having issues after rejoining a cluster + + Whenever a previously deleted node is being re-joined to the cluster, flannelD + tries to assign a new pod subnet to the node. Users should remove the old pod + subnet configuration files in the following paths: + + ```powershell + Remove-Item C:\k\SourceVip.json + Remove-Item C:\k\SourceVipRequest.json + ``` + +1. After launching `start.ps1`, flanneld is stuck in "Waiting for the Network to be created" + + There are numerous reports of this [issue](https://github.com/coreos/flannel/issues/1066); most likely it is a timing issue for when the management IP of the flannel network is set. A workaround is to relaunch `start.ps1` or relaunch it manually as follows: + + ```powershell + [Environment]::SetEnvironmentVariable("NODE_NAME", "") + C:\flannel\flanneld.exe --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 + ``` + +1. My Windows Pods cannot launch because of missing `/run/flannel/subnet.env` + + This indicates that Flannel didn't launch correctly. You can either try + to restart `flanneld.exe` or you can copy the files over manually from + `/run/flannel/subnet.env` on the Kubernetes master to `C:\run\flannel\subnet.env` + on the Windows worker node and modify the `FLANNEL_SUBNET` row to a different + number. For example, if node subnet 10.244.4.1/24 is desired: + + ```env + FLANNEL_NETWORK=10.244.0.0/16 + FLANNEL_SUBNET=10.244.4.1/24 + FLANNEL_MTU=1500 + FLANNEL_IPMASQ=true + ``` + +1. My Windows node cannot access my services using the service IP + + This is a known limitation of the networking stack on Windows. However, Windows Pods can access the Service IP. + +1. No network adapter is found when starting the kubelet + + The Windows networking stack needs a virtual adapter for Kubernetes networking to work. If the following commands return no results (in an admin shell), virtual network creation — a necessary prerequisite for the kubelet to work — has failed: + + ```powershell + Get-HnsNetwork | ? Name -ieq "cbr0" + Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*" + ``` + + Often it is worthwhile to modify the [InterfaceName](https://github.com/microsoft/SDN/blob/master/Kubernetes/flannel/start.ps1#L7) parameter of the start.ps1 script, in cases where the host's network adapter isn't "Ethernet". Otherwise, consult the output of the `start-kubelet.ps1` script to see if there are errors during virtual network creation. + +1. DNS resolution is not properly working + + Check the DNS limitations for Windows in this [section](#dns-limitations). + +1. `kubectl port-forward` fails with "unable to do port forwarding: wincat not found" + + This was implemented in Kubernetes 1.15 by including `wincat.exe` in the pause infrastructure container `mcr.microsoft.com/oss/kubernetes/pause:1.4.1`. Be sure to use a supported version of Kubernetes. + If you would like to build your own pause infrastructure container be sure to include [wincat](https://github.com/kubernetes/kubernetes/tree/master/build/pause/windows/wincat). + +1. My Kubernetes installation is failing because my Windows Server node is behind a proxy + + If you are behind a proxy, the following PowerShell environment variables must be defined: + + ```PowerShell + [Environment]::SetEnvironmentVariable("HTTP_PROXY", "http://proxy.example.com:80/", [EnvironmentVariableTarget]::Machine) + [Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.example.com:443/", [EnvironmentVariableTarget]::Machine) + ``` ### Further investigation -If these steps don't resolve your problem, you can get help running Windows -containers on Windows nodes in Kubernetes through: +If these steps don't resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through: * StackOverflow [Windows Server Container](https://stackoverflow.com/questions/tagged/windows-server-container) topic - * Kubernetes Official Forum [discuss.kubernetes.io](https://discuss.kubernetes.io/) - * Kubernetes Slack [#SIG-Windows Channel](https://kubernetes.slack.com/messages/sig-windows) -## Reporting Issues and Feature Requests +### Reporting issues and feature requests -If you have what looks like a bug, or you would like to make a feature -request, please use the +If you have what looks like a bug, or you would like to +make a feature request, please use the [GitHub issue tracking system](https://github.com/kubernetes/kubernetes/issues). You can open issues on -[GitHub](https://github.com/kubernetes/kubernetes/issues/new/choose) and -assign them to SIG-Windows. You should first search the list of issues in case -it was reported previously and comment with your experience on the issue and -add additional logs. SIG-Windows Slack is also a great avenue to get some -initial support and troubleshooting ideas prior to creating a ticket. +[GitHub](https://github.com/kubernetes/kubernetes/issues/new/choose) and assign +them to SIG-Windows. You should first search the list of issues in case it was +reported previously and comment with your experience on the issue and add additional +logs. SIG-Windows Slack is also a great avenue to get some initial support and +troubleshooting ideas prior to creating a ticket. -If filing a bug, please include detailed information about how to reproduce -the problem, such as: +If filing a bug, please include detailed information about how to reproduce the problem, such as: -* Kubernetes version: kubectl version -* Environment details: Cloud provider, OS distro, networking choice and - configuration, and Docker version +* Kubernetes version: output from `kubectl version` +* Environment details: Cloud provider, OS distro, networking choice and configuration, and Docker version * Detailed steps to reproduce the problem * [Relevant logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs) -* Tag the issue sig/windows by commenting on the issue with `/sig windows` to - bring it to a SIG-Windows member's attention + +It helps if you tag the issue as **sig/windows**, by commenting on the issue with `/sig windows`. This helps to bring +the issue to a SIG Windows member's attention + ## {{% heading "whatsnext" %}} -We have a lot of features in our roadmap. An abbreviated high level list is -included below, but we encourage you to view our -[roadmap project](https://github.com/orgs/kubernetes/projects/8) and help us make -Windows support better by -[contributing](https://github.com/kubernetes/community/blob/master/sig-windows/). +### Deployment tools -### Hyper-V isolation +The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control +plane to manage the cluster it, and nodes to run your workloads. +[Adding Windows nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) +explains how to deploy Windows nodes to your cluster using kubeadm. -Hyper-V isolation is required to enable the following use cases for Windows -containers in Kubernetes: +The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes. -* Hypervisor-based isolation between pods for additional security +### Windows distribution channels -* Backwards compatibility allowing a node to run a newer Windows Server - version without requiring containers to be rebuilt +For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). -* Specific CPU/NUMA settings for a pod - -* Memory isolation and reservations - -Hyper-V isolation support will be added in a later release and will require -CRI-Containerd. - -### Deployment with kubeadm and cluster API - -Kubeadm is becoming the de facto standard for users to deploy a Kubernetes -cluster. Windows node support in kubeadm is currently a work-in-progress but a -guide is available -[here](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/). We are -also making investments in cluster API to ensure Windows nodes are properly -provisioned. +Information on the different Windows Server servicing channels +including their support models can be found at +[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). From 4033f3480b0ca6894fd8903fba0d302b2541252d Mon Sep 17 00:00:00 2001 From: EricWvi Date: Wed, 29 Sep 2021 15:11:27 +0800 Subject: [PATCH 093/249] modify kubeadm config example about joining a node to dual-stack cluster --- .../tools/kubeadm/dual-stack-support.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md b/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md index 39930cca37..7b93a1dd67 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md @@ -93,6 +93,10 @@ kind: JoinConfiguration discovery: bootstrapToken: apiServerEndpoint: 10.100.0.1:6443 + token: "clvldh.vjjwg16ucnhp94qr" + caCertHashes: + - "sha256:a4863cde706cfc580a439f842cc65d5ef112b7b2be31628513a9881cf0d9fe0e" + # change auth info above to match the actual token and CA certificate hash for your cluster nodeRegistration: kubeletExtraArgs: node-ip: 10.100.0.3,fd00:1:2:3::3 @@ -109,6 +113,10 @@ controlPlane: discovery: bootstrapToken: apiServerEndpoint: 10.100.0.1:6443 + token: "clvldh.vjjwg16ucnhp94qr" + caCertHashes: + - "sha256:a4863cde706cfc580a439f842cc65d5ef112b7b2be31628513a9881cf0d9fe0e" + # change auth info above to match the actual token and CA certificate hash for your cluster nodeRegistration: kubeletExtraArgs: node-ip: 10.100.0.4,fd00:1:2:3::4 @@ -118,7 +126,7 @@ nodeRegistration: `advertiseAddress` in JoinConfiguration.controlPlane specifies the IP address that the API Server will advertise it is listening on. The value of `advertiseAddress` equals the `--apiserver-advertise-address` flag of `kubeadm join`. ```shell -kubeadm join --config=kubeadm-config.yaml ... +kubeadm join --config=kubeadm-config.yaml ``` ### Create a single-stack cluster From 32b394f808c4fc29e3e237178094e1cb3a172ca1 Mon Sep 17 00:00:00 2001 From: Riita <42636694+riita10069@users.noreply.github.com> Date: Thu, 30 Sep 2021 00:20:06 +0900 Subject: [PATCH 094/249] Update running-cloud-controller.md --- .../docs/tasks/administer-cluster/running-cloud-controller.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/tasks/administer-cluster/running-cloud-controller.md b/content/ja/docs/tasks/administer-cluster/running-cloud-controller.md index 04dfd62f8b..def1401165 100644 --- a/content/ja/docs/tasks/administer-cluster/running-cloud-controller.md +++ b/content/ja/docs/tasks/administer-cluster/running-cloud-controller.md @@ -79,4 +79,4 @@ cloud-controller-managerは、クラウドプロバイダーのAPIにクエリ ## {{% heading "whatsnext" %}} -独自のクラウドコントローラーマネージャーを構築および開発するには[クラウドコントローラーマネージャーの開発](/docs/tasks/administer-cluster/developing-cloud-controller-manager.md)を参照してください。 +独自のクラウドコントローラーマネージャーを構築および開発するには[クラウドコントローラーマネージャーの開発](/ja/docs/tasks/administer-cluster/developing-cloud-controller-manager/)を参照してください。 From c232ecfbb4ce684e7cb76f14d16c6e1b829171ad Mon Sep 17 00:00:00 2001 From: fabriziopandini Date: Wed, 29 Sep 2021 21:34:32 +0200 Subject: [PATCH 095/249] Postpone ClusterClass blog --- ...md => 2021-10-08-clusterclass-and-managed-topologies.md} | 6 +++--- .../clusterclass.svg | 0 .../create-cluster.svg | 0 3 files changed, 3 insertions(+), 3 deletions(-) rename content/en/blog/_posts/{2021-10-04-clusterclass-and-managed-topologies.md => 2021-10-08-clusterclass-and-managed-topologies.md} (97%) rename static/images/blog/{2021-10-04-clusterclass-and-managed-topologies => 2021-10-08-clusterclass-and-managed-topologies}/clusterclass.svg (100%) rename static/images/blog/{2021-10-04-clusterclass-and-managed-topologies => 2021-10-08-clusterclass-and-managed-topologies}/create-cluster.svg (100%) diff --git a/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md similarity index 97% rename from content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md rename to content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md index ec49d73e7d..af6349ecb8 100644 --- a/content/en/blog/_posts/2021-10-04-clusterclass-and-managed-topologies.md +++ b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md @@ -1,7 +1,7 @@ --- layout: blog title: "Introducing ClusterClass and Managed Topologies in Cluster API" -date: 2021-10-04 +date: 2021-10-08 slug: capi-clusterclass-and-managed-topologies --- @@ -29,7 +29,7 @@ As the name suggests, ClusterClass and managed topologies are built in two parts The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. -![Defining a ClusterClass](/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg) +![Defining a ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg) ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape. @@ -92,7 +92,7 @@ Managed Topologies let you put the power of ClusterClass into action. Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster. -![Create a Cluster with ClusterClass](/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg) +![Create a Cluster with ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg) Here is an example: diff --git a/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg b/static/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg similarity index 100% rename from static/images/blog/2021-10-04-clusterclass-and-managed-topologies/clusterclass.svg rename to static/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg diff --git a/static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg b/static/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg similarity index 100% rename from static/images/blog/2021-10-04-clusterclass-and-managed-topologies/create-cluster.svg rename to static/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg From d113a7066f8622d10d9f0bfe36f306fa07ab0690 Mon Sep 17 00:00:00 2001 From: Arhell Date: Thu, 30 Sep 2021 00:20:11 +0300 Subject: [PATCH 096/249] [ja] Add seccomp tutorial to index --- content/ja/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/ja/docs/tutorials/_index.md b/content/ja/docs/tutorials/_index.md index 1c1d55300a..0cf5ed7f2f 100644 --- a/content/ja/docs/tutorials/_index.md +++ b/content/ja/docs/tutorials/_index.md @@ -46,6 +46,8 @@ content_type: concept * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## サービス * [Source IPを使う](/docs/tutorials/services/source-ip/) From c36a610e8e52a65adc70bd2318baad1bb639040a Mon Sep 17 00:00:00 2001 From: howieyuen Date: Thu, 30 Sep 2021 15:43:06 +0800 Subject: [PATCH 097/249] [zh] sync content/zh/docs/concepts/cluster-administration/flow-control.md --- .../cluster-administration/flow-control.md | 86 +++++++++++-------- 1 file changed, 48 insertions(+), 38 deletions(-) diff --git a/content/zh/docs/concepts/cluster-administration/flow-control.md b/content/zh/docs/concepts/cluster-administration/flow-control.md index c7805402f5..473bdda4ae 100644 --- a/content/zh/docs/concepts/cluster-administration/flow-control.md +++ b/content/zh/docs/concepts/cluster-administration/flow-control.md @@ -128,12 +128,12 @@ APF 特性包含几个不同的功能。 ### 优先级 {#Priority-Levels} @@ -219,8 +219,9 @@ server. The Priority and Fairness feature ships with a suggested configuration that should suffice for experimentation; if your cluster is likely to -experience heavy load then you should consider what configuration will work best. -The suggested configuration groups requests into five priority classes: +experience heavy load then you should consider what configuration will work +best. The suggested configuration groups requests into five priority +classes: --> ## 默认值 {#defaults} @@ -418,8 +419,8 @@ to balance progress between request flows. 公平排队算法支持通过排队配置对优先级微调。 可以在[增强建议](#whats-next)中阅读算法的详细信息，但总之： @@ -450,7 +451,7 @@ proposal](#whats-next), but in short: a small number of flows can dominate the apiserver. A larger `handSize` also potentially increases the amount of latency that a single high-traffic flow can cause. The maximum number of queued requests possible from a - single flow is `handSize *queueLengthLimit`. + single flow is `handSize * queueLengthLimit`. {{< /note >}} --> * 修改 `handSize` 允许你调整过载情况下不同流之间的冲突概率以及单个流可用的整体并发性。 @@ -459,7 +460,7 @@ proposal](#whats-next), but in short: 较大的 `handSize` 使两个单独的流程发生碰撞的可能性较小（因此，一个流可以饿死另一个流）， 但是更有可能的是少数流可以控制 apiserver。 较大的 `handSize` 还可能增加单个高并发流的延迟量。 - 单个流中可能排队的请求的最大数量为 `handSize *queueLengthLimit` 。 + 单个流中可能排队的请求的最大数量为 `handSize * queueLengthLimit` 。 {{< /note >}} * `apiserver_flowcontrol_rejected_requests_total` 是一个计数器向量， 记录被拒绝的请求数量（自服务器启动以来累计值）， @@ -725,6 +727,14 @@ poorly-behaved workloads that may be harming system health. 记录包含执行中（不在队列中等待）请求的瞬时数量， 由标签 `priority_level` 和 `flow_schema` 进一步区分。 + +* `apiserver_flowcontrol_request_concurrency_in_use` 是一个规范向量， + 包含占用座位的瞬时数量，由标签 `priority_level` 和 `flow_schema` 进一步区分。 + ### 调试端点 {#Debug-endpoints} 启用 APF 特性后， kube-apiserver 会在其 HTTP/HTTPS 端口提供以下路径： - `/debug/api_priority_and_fairness/dump_priority_levels` —— 所有优先级及其当前状态的列表。你可以这样获取： @@ -856,7 +866,7 @@ You can fetch like this: 输出类似于： - ``` + ```none PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, workload-low, 0, true, false, 0, 0, global-default, 0, true, false, 0, 0, @@ -868,8 +878,8 @@ You can fetch like this: ``` - `/debug/api_priority_and_fairness/dump_queues` —— 所有队列及其当前状态的列表。 你可以这样获取： @@ -881,7 +891,7 @@ You can fetch like this: 输出类似于： - ``` + ```none PriorityLevelName, Index, PendingRequests, ExecutingRequests, VirtualStart, workload-high, 0, 0, 0, 0.0000, workload-high, 1, 0, 0, 0.0000, @@ -892,8 +902,8 @@ You can fetch like this: ``` - `/debug/api_priority_and_fairness/dump_requests` ——当前正在队列中等待的所有请求的列表。 你可以这样获取： @@ -905,15 +915,15 @@ You can fetch like this: 输出类似于： - ``` + ```none PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, exempt, , , , , , system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z, ``` 针对每个优先级别，输出中还包含一条虚拟记录，对应豁免限制。 @@ -925,8 +935,9 @@ You can fetch like this: ``` + 输出类似于： - ``` + ```none PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, UserName, Verb, APIPath, Namespace, Name, APIVersion, Resource, SubResource, system, system-nodes, 12, 0, system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps, system, system-nodes, 12, 1, system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps, @@ -935,14 +946,13 @@ You can fetch like this: ## {{% heading "whatsnext" %}} 有关API优先级和公平性的设计细节的背景信息， -请参阅[增强建议](https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190228-priority-and-fairness.md)。 +请参阅[增强建议](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness)。 你可以通过 [SIG APIMachinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery/) 或特性的 [Slack 频道](https://kubernetes.slack.com/messages/api-priority-and-fairness/) 提出建议和特性请求。 From b033b85cd971c850683ce3c9052ea63ee7771c79 Mon Sep 17 00:00:00 2001 From: RA489 Date: Thu, 2 Sep 2021 10:13:27 +0530 Subject: [PATCH 098/249] Improvement for thirdparty tools --- content/en/docs/reference/tools/_index.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/tools/_index.md b/content/en/docs/reference/tools/_index.md index 48a9b20f41..c8d42be352 100644 --- a/content/en/docs/reference/tools/_index.md +++ b/content/en/docs/reference/tools/_index.md @@ -8,14 +8,15 @@ no_list: true --- -Kubernetes contains several built-in as well as third party tools to help you work with the Kubernetes system. +Kubernetes contains several tools to help you work with the Kubernetes system. +{{% thirdparty-content %}} ## Minikube -[`minikube`](https://minikube.sigs.k8s.io/docs/) is a built-in tool that +[`minikube`](https://minikube.sigs.k8s.io/docs/) is a tool that runs a single-node Kubernetes cluster locally on your workstation for development and testing purposes. @@ -26,7 +27,10 @@ to a Kubernetes cluster, troubleshoot them, and manage the cluster and its resou ## Helm -[`Kubernetes Helm`](https://github.com/kubernetes/helm) is a third party managed tool for managing packages of pre-configured +[`Kubernetes Helm`](https://github.com/kubernetes/helm) is a tool for managing packages of pre-configured +Kubernetes resources, aka Kubernetes charts. + +Beware Helm is a third party managed tool for managing packages of pre-configured Kubernetes resources, aka Kubernetes charts. Use Helm to: @@ -39,7 +43,7 @@ Use Helm to: ## Kompose -[`Kompose`](https://github.com/kubernetes/kompose) is a built-in tool to help Docker Compose users move to Kubernetes. +[`Kompose`](https://github.com/kubernetes/kompose) is a tool to help Docker Compose users move to Kubernetes. Use Kompose to: From 7f88af518e42bcea151ddd24cffb4ce6af202273 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Thu, 30 Sep 2021 18:09:26 +0100 Subject: [PATCH 099/249] Update compatibility statement for TerminationGracePeriod Co-authored-by: Mark Rossetti --- .../windows/intro-windows-in-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md index f395de9397..7b8e24db60 100644 --- a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md @@ -120,7 +120,7 @@ including: * HugePages: not supported for Windows containers * Privileged containers: not supported for Windows containers -* TerminationGracePeriod: not implemented +* TerminationGracePeriod: requires containerD Not all features of shared namespaces are supported. See [API compatibility](#api) for more details. From 2284c9dcec8a6483eb0d0bc8785df1ccedd6a18f Mon Sep 17 00:00:00 2001 From: Victor Paredes Date: Thu, 30 Sep 2021 18:54:05 -0400 Subject: [PATCH 100/249] concepts/services-networking/service: Removed duplicate ProxyTerminatingEndpoints term. --- content/en/docs/concepts/services-networking/service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index db28918c98..71068be5fd 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -414,7 +414,7 @@ endpoints, the kube-proxy does not forward any traffic for the relevant Service. {{< feature-state for_k8s_version="v1.22" state="alpha" >}} If you enable the `ProxyTerminatingEndpoints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`ProxyTerminatingEndpoints` for the kube-proxy, the kube-proxy checks if the node +for the kube-proxy, the kube-proxy checks if the node has local endpoints and whether or not all the local endpoints are marked as terminating. If there are local endpoints and **all** of those are terminating, then the kube-proxy ignores any external traffic policy of `Local`. Instead, whilst the node-local endpoints remain as all From db9d2110f8f1c3268a4d6eb458a9fb80ed00c333 Mon Sep 17 00:00:00 2001 From: Mohit Sharma Date: Sat, 2 Oct 2021 01:31:27 +1000 Subject: [PATCH 101/249] fix the korean doc link issue for pod deltion cost. Signed-off-by: Mohit Sharma --- content/ko/docs/concepts/workloads/controllers/replicaset.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ko/docs/concepts/workloads/controllers/replicaset.md b/content/ko/docs/concepts/workloads/controllers/replicaset.md index 7cf399d242..a04184e048 100644 --- a/content/ko/docs/concepts/workloads/controllers/replicaset.md +++ b/content/ko/docs/concepts/workloads/controllers/replicaset.md @@ -325,7 +325,7 @@ curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/fron ### 파드 삭제 비용 {{< feature-state for_k8s_version="v1.21" state="alpha" >}} -[`controller.kubernetes.io/pod-deletion-cost`](/docs/reference/labels-annotations-taints/#pod-deletion-cost) 어노테이션을 이용하여, +[`controller.kubernetes.io/pod-deletion-cost`](/ko/docs/reference/labels-annotations-taints/#pod-deletion-cost) 어노테이션을 이용하여, 레플리카셋을 스케일 다운할 때 어떤 파드부터 먼저 삭제할지에 대한 우선순위를 설정할 수 있다. 이 어노테이션은 파드에 설정되어야 하며, [-2147483647, 2147483647] 범위를 갖는다. From cd7fa760e04e60f41e97daa678ba453c688d67dc Mon Sep 17 00:00:00 2001 From: aychen99 <42681629+aychen99@users.noreply.github.com> Date: Fri, 1 Oct 2021 13:41:39 -0400 Subject: [PATCH 102/249] Update basic-stateful-set.md Fix syntax issue --- .../docs/tutorials/stateful-application/basic-stateful-set.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md b/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md index 8b7f166a69..07f6302eae 100644 --- a/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md @@ -19,7 +19,7 @@ This tutorial provides an introduction to managing applications with demonstrates how to create, delete, scale, and update the Pods of StatefulSets. --> -本教程介绍如何了使用 [StatefulSets](/zh/docs/concepts/workloads/controllers/statefulset/) 来管理应用。 +本教程介绍了如何使用 [StatefulSets](/zh/docs/concepts/workloads/controllers/statefulset/) 来管理应用。 演示了如何创建、删除、扩容/缩容和更新 StatefulSets 的 Pods。 From 7b07d97a0b0c20343c63f422f82959e738377a4a Mon Sep 17 00:00:00 2001 From: Toli Kuznets Date: Fri, 1 Oct 2021 12:21:42 -0700 Subject: [PATCH 103/249] Update job.md add a newline per bug #29840 --- content/en/docs/concepts/workloads/controllers/job.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/docs/concepts/workloads/controllers/job.md b/content/en/docs/concepts/workloads/controllers/job.md index 2ca81a5ca0..54a4104c5e 100644 --- a/content/en/docs/concepts/workloads/controllers/job.md +++ b/content/en/docs/concepts/workloads/controllers/job.md @@ -207,6 +207,7 @@ Jobs with _fixed completion count_ - that is, jobs that have non null {{< glossary_tooltip term_id="Service" >}}, Pods within the Job can use the deterministic hostnames to address each other via DNS. - From the containarized task, in the environment variable `JOB_COMPLETION_INDEX`. + The Job is considered complete when there is one successfully completed Pod for each index. For more information about how to use this mode, see [Indexed Job for Parallel Processing with Static Work Assignment](/docs/tasks/job/indexed-parallel-processing-static/). From b04b2cf2ff480628bb2b78b68a1dd7bc2fcfbe9a Mon Sep 17 00:00:00 2001 From: Arhell Date: Sat, 2 Oct 2021 00:21:32 +0300 Subject: [PATCH 104/249] [ja] Update location of PrepareNode.ps1 --- .../tasks/administer-cluster/kubeadm/adding-windows-nodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ja/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md b/content/ja/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md index 62e610b277..e332cfda7c 100644 --- a/content/ja/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md +++ b/content/ja/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md @@ -146,7 +146,7 @@ Windowsワーカーノードの(管理者)権限を持つPowerShell環境で実 1. wins、kubelet、kubeadmをインストールします。 ```PowerShell - curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1 + curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1 .\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} ``` From abadaf1ead4a36f08e8472e6854b3ada925efe9c Mon Sep 17 00:00:00 2001 From: "xiao.li" Date: Fri, 1 Oct 2021 13:54:04 -0700 Subject: [PATCH 105/249] Fix the page links to a section that doesn't exist --- .../en/docs/concepts/services-networking/dns-pod-service.md | 4 ++-- .../docs/tasks/administer-cluster/dns-debugging-resolution.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/services-networking/dns-pod-service.md b/content/en/docs/concepts/services-networking/dns-pod-service.md index f43eeff22b..f1a085176e 100644 --- a/content/en/docs/concepts/services-networking/dns-pod-service.md +++ b/content/en/docs/concepts/services-networking/dns-pod-service.md @@ -217,13 +217,13 @@ following pod-specific DNS policies. These policies are specified in the - "`Default`": The Pod inherits the name resolution configuration from the node that the pods run on. - See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node) + See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers) for more details. - "`ClusterFirst`": Any DNS query that does not match the configured cluster domain suffix, such as "`www.kubernetes.io`", is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured. - See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#effects-on-pods) + See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers) for details on how DNS queries are handled in those cases. - "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should explicitly set its DNS policy "`ClusterFirstWithHostNet`". diff --git a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md index 6bc41d2170..e9dd545955 100644 --- a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md +++ b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md @@ -67,7 +67,7 @@ If the `nslookup` command fails, check the following: ### Check the local DNS configuration first Take a look inside the resolv.conf file. -(See [Inheriting DNS from the node](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node) and +(See [Customizing DNS Service](/docs/tasks/administer-cluster/dns-custom-nameservers) and [Known issues](#known-issues) below for more information) ```shell From c170ba469f48c7594ddea431c4192427b6cb9814 Mon Sep 17 00:00:00 2001 From: Min Wang Date: Fri, 1 Oct 2021 13:35:35 -0700 Subject: [PATCH 106/249] Fix issue #29850 - drop .v1.apps in doc deployment --- .../workloads/controllers/deployment.md | 42 +++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index 791f800bbd..5338c52b9a 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -164,7 +164,7 @@ Follow the steps given below to update your Deployment: 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image. ```shell - kubectl deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubectl deployment.apps/nginx-deployment set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` or use the following command: @@ -182,7 +182,7 @@ Follow the steps given below to update your Deployment: Alternatively, you can `edit` the Deployment and change `.spec.template.spec.containers[0].image` from `nginx:1.14.2` to `nginx:1.16.1`: ```shell - kubectl edit deployment.v1.apps/nginx-deployment + kubectl edit deployment/nginx-deployment ``` The output is similar to: @@ -365,7 +365,7 @@ rolled back. * Suppose that you made a typo while updating the Deployment, by putting the image name as `nginx:1.161` instead of `nginx:1.16.1`: ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 + kubectl set image deployment/nginx-deployment nginx=nginx:1.161 ``` The output is similar to this: @@ -474,25 +474,25 @@ Follow the steps given below to check the rollout history: 1. First, check the revisions of this Deployment: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment + kubectl rollout history deployment/nginx-deployment ``` The output is similar to this: ``` deployments "nginx-deployment" REVISION CHANGE-CAUSE 1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml - 2 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 - 3 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 + 2 kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 + 3 kubectl set image deployment/nginx-deployment nginx=nginx:1.161 ``` `CHANGE-CAUSE` is copied from the Deployment annotation `kubernetes.io/change-cause` to its revisions upon creation. You can specify the`CHANGE-CAUSE` message by: - * Annotating the Deployment with `kubectl annotate deployment.v1.apps/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"` + * Annotating the Deployment with `kubectl annotate deployment/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"` * Manually editing the manifest of the resource. 2. To see the details of each revision, run: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2 + kubectl rollout history deployment/nginx-deployment --revision=2 ``` The output is similar to this: @@ -500,7 +500,7 @@ Follow the steps given below to check the rollout history: deployments "nginx-deployment" revision 2 Labels: app=nginx pod-template-hash=1159050644 - Annotations: kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + Annotations: kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 Containers: nginx: Image: nginx:1.16.1 @@ -517,7 +517,7 @@ Follow the steps given below to rollback the Deployment from the current version 1. Now you've decided to undo the current rollout and rollback to the previous revision: ```shell - kubectl rollout undo deployment.v1.apps/nginx-deployment + kubectl rollout undo deployment/nginx-deployment ``` The output is similar to this: @@ -527,7 +527,7 @@ Follow the steps given below to rollback the Deployment from the current version Alternatively, you can rollback to a specific revision by specifying it with `--to-revision`: ```shell - kubectl rollout undo deployment.v1.apps/nginx-deployment --to-revision=2 + kubectl rollout undo deployment/nginx-deployment --to-revision=2 ``` The output is similar to this: @@ -561,7 +561,7 @@ Follow the steps given below to rollback the Deployment from the current version CreationTimestamp: Sun, 02 Sep 2018 18:17:55 -0500 Labels: app=nginx Annotations: deployment.kubernetes.io/revision=4 - kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 Selector: app=nginx Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate @@ -604,7 +604,7 @@ Follow the steps given below to rollback the Deployment from the current version You can scale a Deployment by using the following command: ```shell -kubectl scale deployment.v1.apps/nginx-deployment --replicas=10 +kubectl scale deployment/nginx-deployment --replicas=10 ``` The output is similar to this: ``` @@ -616,7 +616,7 @@ in your cluster, you can setup an autoscaler for your Deployment and choose the Pods you want to run based on the CPU utilization of your existing Pods. ```shell -kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80 +kubectl autoscale deployment/nginx-deployment --min=10 --max=15 --cpu-percent=80 ``` The output is similar to this: ``` @@ -645,7 +645,7 @@ For example, you are running a Deployment with 10 replicas, [maxSurge](#max-surg * You update to a new image which happens to be unresolvable from inside the cluster. ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:sometag + kubectl set image deployment/nginx-deployment nginx=nginx:sometag ``` The output is similar to this: @@ -724,7 +724,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Pause by running the following command: ```shell - kubectl rollout pause deployment.v1.apps/nginx-deployment + kubectl rollout pause deployment/nginx-deployment ``` The output is similar to this: @@ -734,7 +734,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Then update the image of the Deployment: ```shell - kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 + kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` The output is similar to this: @@ -744,7 +744,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Notice that no new rollout started: ```shell - kubectl rollout history deployment.v1.apps/nginx-deployment + kubectl rollout history deployment/nginx-deployment ``` The output is similar to this: @@ -766,7 +766,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * You can make as many updates as you wish, for example, update the resources that will be used: ```shell - kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi + kubectl set resources deployment/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi ``` The output is similar to this: @@ -779,7 +779,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess * Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates: ```shell - kubectl rollout resume deployment.v1.apps/nginx-deployment + kubectl rollout resume deployment/nginx-deployment ``` The output is similar to this: @@ -889,7 +889,7 @@ The following `kubectl` command sets the spec with `progressDeadlineSeconds` to lack of progress for a Deployment after 10 minutes: ```shell -kubectl patch deployment.v1.apps/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}' +kubectl patch deployment/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}' ``` The output is similar to this: ``` From 5e9d700ddb45a2a64c6115ccac1abf060d907885 Mon Sep 17 00:00:00 2001 From: learner0810 Date: Sat, 2 Oct 2021 15:30:34 +0800 Subject: [PATCH 107/249] fix/typo-command-line-tools-reference/kubelet --- .../en/docs/reference/command-line-tools-reference/kubelet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index 0531f0847a..75698b3966 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -51,7 +51,7 @@ kubelet [flags] --address string     Default: 0.0.0.0 -The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in gll interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) From 8a736308a9039db3516f26619728e94e8c749f2f Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sat, 2 Oct 2021 16:28:54 +0100 Subject: [PATCH 108/249] Revise third-party content warning --- data/i18n/en/en.toml | 12 +++++++++++- layouts/docs/baseof.html | 5 +++++ layouts/partials/docs/thirdparty-disclaimer.html | 3 +++ layouts/partials/page-meta-links.html | 4 ++++ layouts/shortcodes/thirdparty-content.html | 11 ++++++++--- 5 files changed, 31 insertions(+), 4 deletions(-) create mode 100644 layouts/partials/docs/thirdparty-disclaimer.html diff --git a/data/i18n/en/en.toml b/data/i18n/en/en.toml index e32d8819d2..6ed8b8a731 100644 --- a/data/i18n/en/en.toml +++ b/data/i18n/en/en.toml @@ -217,7 +217,17 @@ other = "Subscribe" other = "Synopsis" [thirdparty_message] -other = """This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects. This page follows     CNCF website guidelines by listing projects alphabetically. To add a project to this list, read the content guide before submitting a change.""" +other = """This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.""" + +[thirdparty_message_edit_disclaimer] +other="""Third party content advice""" + + +[thirdparty_message_single_item] +other = """🛇 This item links to a third party project or product that is not part of Kubernetes itself. More information""" + +[thirdparty_message_disclaimer] +other = """

    Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the CNCF website guidelines for more details.

    You should read the content guide before proposing a change that adds an extra third-party link.

    """ [ui_search_placeholder] other = "Search" diff --git a/layouts/docs/baseof.html b/layouts/docs/baseof.html index cc6237938b..7dae9a6e90 100644 --- a/layouts/docs/baseof.html +++ b/layouts/docs/baseof.html @@ -22,6 +22,11 @@ {{ partial "deprecation-warning.html" . }} {{ end }} {{ block "main" . }}{{ end }} + {{- if .HasShortcode "thirdparty-content" -}} + {{ block "thirdparty-disclaimer" . }} + {{ partial "docs/thirdparty-disclaimer.html" . }} + {{- end -}} + {{- end -}} {{ if (and (not .Params.hide_feedback) (.Site.Params.ui.feedback.enable) (.Site.GoogleAnalytics)) }} {{ partial "feedback.html" .Site.Params.ui.feedback }} {{ end }} diff --git a/layouts/partials/docs/thirdparty-disclaimer.html b/layouts/partials/docs/thirdparty-disclaimer.html new file mode 100644 index 0000000000..0eca49d071 --- /dev/null +++ b/layouts/partials/docs/thirdparty-disclaimer.html @@ -0,0 +1,3 @@ +
    +{{ T "thirdparty_message_disclaimer" | safeHTML }} +
    diff --git a/layouts/partials/page-meta-links.html b/layouts/partials/page-meta-links.html index 529bcea29c..83571db525 100644 --- a/layouts/partials/page-meta-links.html +++ b/layouts/partials/page-meta-links.html @@ -24,6 +24,10 @@ {{ if not (.Param "auto_generated") }} {{ T "post_edit_this" }} + {{- if .HasShortcode "thirdparty-content" -}} + {{ T "thirdparty_message_edit_disclaimer" | safeHTML }} + {{- end }} + {{ T "post_create_child_page" }} {{ end }} diff --git a/layouts/shortcodes/thirdparty-content.html b/layouts/shortcodes/thirdparty-content.html index 5a98081e49..36fbdef13e 100644 --- a/layouts/shortcodes/thirdparty-content.html +++ b/layouts/shortcodes/thirdparty-content.html @@ -1,4 +1,9 @@ - +{{- else -}} + {{ T "thirdparty_message_single_item" | safeHTML }} +{{- end -}} + From feb2785238a638cdb6ba00a0087d92f73c2c085f Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sat, 2 Oct 2021 16:53:59 +0100 Subject: [PATCH 109/249] Highlight third party content disclaimer When the (new) third party content disclaimer is the current target, highlight it with a flash of yellow that fades to a yellow background, rather than the usual gray. --- assets/scss/_custom.scss | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index 7d6aa6784f..752af266cb 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -343,6 +343,26 @@ main { } } +// Highlight disclaimer when targeted as a fragment + +#third-party-content-disclaimer { + color: #000; + background: #f8f9fa; + transition: all 0.5s ease; +} + +@keyframes disclaimer-highlight { + from { background: #f8f922; color: #000; } + 50% { background: #f8f944; color: #000; } + to { background: #f8f9cb; color: #000; } +} + +#third-party-content-disclaimer:target { + color: #000; + animation: disclaimer-highlight 1.25s ease; + background: #f8f9cb; +} + .deprecation-warning { padding: 20px; margin: 20px 0; From bab43543f072053d5b0f68ca352c1abe722fba9f Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sat, 2 Oct 2021 17:01:11 +0100 Subject: [PATCH 110/249] Set custom color for third-party-content alert --- assets/scss/_custom.scss | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index 752af266cb..20a36b6175 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -338,11 +338,15 @@ main { border-left-color: #d9534f; } + h1:first-of-type + blockquote.callout { margin-top: 1.5em; } } +// Special color for third party content disclaimers +.alert.third-party-content { border-left-color: #222 }; + // Highlight disclaimer when targeted as a fragment #third-party-content-disclaimer { From a87f5070490bf7a3b1b0d9a8ae95f53a60b3bc78 Mon Sep 17 00:00:00 2001 From: Arhell Date: Sun, 3 Oct 2021 01:18:07 +0300 Subject: [PATCH 111/249] [pl] Add seccomp tutorial to index --- content/pl/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/pl/docs/tutorials/_index.md b/content/pl/docs/tutorials/_index.md index c55fd9c3ff..4dc176cd54 100644 --- a/content/pl/docs/tutorials/_index.md +++ b/content/pl/docs/tutorials/_index.md @@ -48,6 +48,8 @@ Przed zapoznaniem się z samouczkami warto stworzyć zakładkę do * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Serwisy * [Using Source IP](/docs/tutorials/services/source-ip/) From aed3469661d1c8527f627b9b9494292202cdf6f5 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Sun, 3 Oct 2021 17:17:27 +0100 Subject: [PATCH 112/249] Clean up pre-Docsy content and styles These files being removed were already deprecated and are not in fact used. Tidy them up. --- deprecated/assets/sass/_base.sass | 1904 ----------------- deprecated/assets/sass/_case-studies.sass | 99 - deprecated/assets/sass/_desktop.sass | 304 --- deprecated/assets/sass/_reset.sass | 61 - deprecated/assets/sass/_size.sass | 86 - deprecated/assets/sass/_skin.sass | 25 - deprecated/assets/sass/_tablet.sass | 250 --- deprecated/assets/sass/style.sass | 11 - .../users/application-developer/advanced.md | 120 -- .../application-developer/foundational.md | 260 --- .../application-developer/intermediate.md | 166 -- .../users/cluster-operator/foundational.md | 96 - .../users/cluster-operator/intermediate.md | 109 - 13 files changed, 3491 deletions(-) delete mode 100644 deprecated/assets/sass/_base.sass delete mode 100644 deprecated/assets/sass/_case-studies.sass delete mode 100644 deprecated/assets/sass/_desktop.sass delete mode 100755 deprecated/assets/sass/_reset.sass delete mode 100644 deprecated/assets/sass/_size.sass delete mode 100755 deprecated/assets/sass/_skin.sass delete mode 100644 deprecated/assets/sass/_tablet.sass delete mode 100755 deprecated/assets/sass/style.sass delete mode 100644 deprecated/content/en/docs/user-journeys/users/application-developer/advanced.md delete mode 100644 deprecated/content/en/docs/user-journeys/users/application-developer/foundational.md delete mode 100644 deprecated/content/en/docs/user-journeys/users/application-developer/intermediate.md delete mode 100644 deprecated/content/en/docs/user-journeys/users/cluster-operator/foundational.md delete mode 100644 deprecated/content/en/docs/user-journeys/users/cluster-operator/intermediate.md diff --git a/deprecated/assets/sass/_base.sass b/deprecated/assets/sass/_base.sass deleted file mode 100644 index 7306433631..0000000000 --- a/deprecated/assets/sass/_base.sass +++ /dev/null @@ -1,1904 +0,0 @@ -@import "reset" - -// TEXT -$h1-font-size: 32px -$h1-line-height: 40px - -$h2-font-size: 28px -$h2-line-height: 60px - -$h3-font-size: 24px -$h3-line-height: 32px - -$h4-font-size: 20px -$h4-line-height: 40px - -$h5-font-size: 16px -$h5-line-height: 36px - -$p-font-size: 14px -$p-line-height: 22px - - -// header -$full-width-paddingX: 20px - -$main-width: 100% -$main-max-width: 100% - -$header-height: 80px - -$logo-width: 180px - -$nav-buttons-margin-left: 30px - -$hamburger-size: 50px - - -// main nav -$main-nav-padding: 140px 0 30px -$main-nav-h5-margin-bottom: 1em -$main-nav-h3-margin-bottom: 0.6em -$nav-box-width: 20% -$nav-box-sibling-margin-left: calc(20% / 3) -$main-nav-main-sibling-margin-top: 60px -$main-nav-left-button-size: 50px -$main-nav-left-button-font-size: 18px - - -// hero -$hero-padding-top: 80px -$headline-wrapper-margin-bottom: 40px -$quickstart-button-padding: 0 50px -$vendor-strip-height: 88px -$vendor-strip-font-size: 16px - -// video -$video-section-height: 200px - -@import "size" - -//////////////////////////////////////////////////////////////////////////////////////////////////// -//////////////////////////////////////////////////////////////////////////////////////////////////// -//////////////////////////////////////////////////////////////////////////////////////////////////// - -body - background-color: white - -section - position: relative - background-color: white - -section, header, footer - .main-section - position: relative - margin: auto - -p - font-size: 14px - font-weight: 400 - - -.button - display: inline-block - border-radius: 6px - padding: 0 20px - line-height: 40px - color: white - background-color: $blue - text-decoration: none - font-size: 1rem - -#cellophane - position: fixed - top: 0 - left: 0 - width: 100% - height: 100% - display: none - - -// HEADER -header - position: fixed - top: 0 - left: 0 - width: 100% - z-index: 8888 - background-color: transparent - box-shadow: 0 0 0 transparent - transition: 0.3s - text-align: center - overflow: hidden - - -.logo - position: relative - float: left - display: block - width: 180px - height: 88px - top: 0 - left: 0 - transform: none - background-image: url(/images/nav_logo.svg) - background-size: contain - background-position: center center - background-repeat: no-repeat - -#blog - &.flip-nav, &.open-nav - .logo - background-image: url(../images/nav_logo2.svg) - -// Blog post tables -.blog-content - table - max-width: 100% - border: 1px solid #ccc - border-spacing: 0 - margin-top: 30px - margin-bottom: 30px - @media screen and (max-width: 425px) - display: block - overflow-x: auto - - thead - border-bottom: 2px solid #ccc - - tr - th - padding: .5rem - font-size: 1.5rem - - tbody - tr - td - padding: .5rem - border-bottom: 1px solid #ccc - border-left: 1px solid #ccc - padding-right: 5rem - -#docs - .flyout-button - position: fixed - top: 20px - left: 20px - width: 50px - height: 50px - background-image: url(/images/toc_icon.png) - background-position: center center - background-repeat: no-repeat - background-size: auto - border-radius: 50% - transition: 0.3s - z-index: 99999 - - &.open-nav .flyout-button - display: none - - .logo - position: absolute - top: 40px - left: 50% - transform: translate(-50%, -50%) - display: block - width: 45px - height: 44px - background-image: url(/images/favicon.png) - - &.flip-nav .flyout-button - background-image: url(/images/toc_icon_grey.png) - - -.nav-buttons - float: right - -#viewDocs, #tryKubernetes - display: none - -#viewDocs - border: 2px solid white - background-color: transparent - transition: 0.3s - - &:hover - background-color: white - color: $dark-grey - -#tryKubernetes - width: 0 - padding: 0 0 - border: 1px solid transparent - background-color: transparent - text-align: center - white-space: nowrap - vertical-align: middle - overflow: hidden - transition: 0.3s - -#hamburger - display: inline-block - position: relative - vertical-align: middle - padding: 0 - border: 0 - background: none - - div, &:before, &:after - position: absolute - left: 15% - width: 70% - height: 2px - background-color: $blue - transition: 0.3s - content: "" - - div - top: calc(50% - 1px) - - &:before - top: 24% - - &:after - bottom: 24% - - &:hover - div, &:before, &:after - background-color: white - -#mainNav - - h5 - color: $blue - font-weight: normal - - .main-section - white-space: nowrap - overflow: hidden - clear: both - - .nav-box - float: left - white-space: normal - - h3 - a - color: $blue - text-decoration: none - -// Global Nav - 12/9/2016 Update - -ul.global-nav - display: none - - li - display: inline-block - margin-right: 14px - - a - color: #fff - font-weight: 400 - padding: 0 - position: relative - - &.active:after - position: absolute - width: 100% - height: 2px - content: '' - bottom: -4px - left: 0 - background: #fff - - .ui-icon - filter: brightness(0) invert(1) - - ul - display: none - position: fixed - top: 40px - text-align: left - - li - display: block - height: 28px - - a - background: $dark-grey - color: #fff - padding: 7px - - li:last-child - a - border-radius: 7px - - li:hover - ul - display: block - - -.flip-nav, .open-nav - ul.global-nav li a - color: $dark-grey - - .ui-icon - filter: brightness(0) - -.flip-nav ul.global-nav li ul li a - background: #fff - color: $dark-grey - -.flip-nav ul.global-nav li a.active:after, .flip-nav ul.global-nav li ul li a.active:after, .open-nav ul.global-nav li a.active:after - background: $blue - -// FLIP NAV -.flip-nav - header - background-color: white - - -// OPEN NAV -.open-nav - body - overflow: hidden - - #cellophane - display: block - z-index: 9998 - - header - background-color: #e8e8e8 - z-index: 9999 - - #hamburger - div - opacity: 0 - - &:before, &:after - left: 12px - transform-origin: 0 1px - - &:before - transform: rotate(45deg) - - &:after - transform: rotate(-45deg) - -.open-nav, .y-enough - #tryKubernetes - width: auto - padding: 0 20px - background-color: $blue - border-color: $blue - - -.flip-nav, .open-nav - header - box-shadow: 0 1px 2px $medium-grey - - #viewDocs - border-color: $dark-grey - color: $dark-grey - - &:hover - border-color: $blue - background-color: $blue - color: white - - #hamburger:hover - div, &:before, &:after - background-color: $dark-grey - - - -// HERO -#hero - background-image: url(/images/texture.png) - background-color: $dark-grey - text-align: center - padding-left: 0 - padding-right: 0 - margin-bottom: 0 - position: relative - - &.bot-bar:after - display: block - margin-bottom: -20px - height: 8px - width: 100% - background-color: transparentize(white, 0.9) - content: '' - - &.no-sub - - h5 - display: none - - h1 - margin-bottom: 20px - -#home #hero:after - display: none - -// VENDOR STRIP -#vendorStrip - position: relative - background-color: transparentize(white, 0.9) - font-weight: 100 - white-space: nowrap - text-align: center - - li a - color: transparentize(white, 0.5) - - &.YAH - color: white - position: relative - - -// FOOTER -footer - width: 100% - background-image: url(/images/texture.png) - background-color: $dark-grey - overflow-x: hidden - - .main-section - padding: 20px 0 - - nav - a - width: 100% - text-align: center - display: inline-block - margin: 10px 0 - font-size: 24px - font-weight: 300 - color: white - text-decoration: none - - - .social - margin: 20px 0 - - div - text-align: center - margin-bottom: 20px - - div:last-child - margin: 30px 0 - - span - display: block - margin-bottom: 8px - - input - text-align: center - - - -#search, #wishField - background-color: transparent - padding: 10px - font-size: 16px - font-weight: 100 - color: white - border: 1px solid white - transition: 0.3s - - &:focus - background-color: $light-grey - color: $dark-grey - -.social a - display: inline-block - background-image: url(/images/social_sprite.svg) - background-repeat: no-repeat - background-size: 350px - width: 50px - height: 50px - border-radius: 5px - margin-right: 10px - - &:hover - background-color: #fff - - span - position: absolute - display: block - height: 0 - overflow: hidden - - &.button - background-image: none - width: auto - height: auto - - &:hover - color: $blue - -a.twitter - background-position: 0 0 - - &:hover - background-position: 0 100% - -a.stack-overflow - background-position: -50px 0 - - &:hover - background-position: -50px 100% - -a.slack - background-position: -100px 0 - - &:hover - background-position: -100px 100% - -a.github - background-position: -150px 0 - - &:hover - background-position: -150px 100% - -a.mailing-list - background-position: -200px 0 - - &:hover - background-position: -200px 100% - -a.calendar - background-position: -250px 0 - - &:hover - background-position: -250px 100% - -a.youtube - background-position: -300px 0 - - &:hover - background-position: -300px 100% - -#viewDocs - display: none - -section - background-color: white - -#hero - background-color: $dark-grey - - h5 - margin: 20px 0 - line-height: 28px - -#vendorStrip - position: relative - - ul - float: left - - li - display: inline-block - height: 100% - margin-right: 10px - &:last-child - margin-right: 0 - - a - display: block - height: 100% - color: white - font-size: 0.75em - font-weight: bold - - li + li - margin-left: 0 - - -#docs - #hero - &.light-text.no-sub - padding-bottom: 0px - - #vendorStrip - line-height: 44px - max-width: 100% - -webkit-overflow-scrolling: touch - - ul - float: none - overflow-x: auto - - #searchBox - float: none - display: block - width: 80% - margin: 0 auto - height: 44px - line-height: 44px - position: relative - - &:before - position: absolute - width: 15px - height: 15px - content: '' - right: 8px - top: 7px - background-image: url(/images/search-icon.svg) - background-repeat: no-repeat - background-size: 100% 100% - z-index: 1 - - #search - width: 100% - padding: 0 10px - height: 30px - line-height: 30px - font-size: 16px - vertical-align: top - background: #fff - border: none - border-radius: 4px - position: relative - - -#encyclopedia - position: relative - padding: 50px 20px 20px 20px - overflow: hidden - font-size: 14px - - & > div - height: 100% - -#docsToc - position: fixed - background-color: white - top: 0 - left: 0 - width: 0 - height: 100vh - overflow: hidden - padding: 50px 0 - z-index: 999999 - transition: 0.3s - - .yah - & > .title - background-color: $light-grey - border-left: 3px solid $blue - padding: 7.5px 10px 7.5px 18px - margin-left: -3px - color: $blue - -.open-toc - body - overflow: hidden - - #docsToc - padding: 50px 20px - width: 400px - max-width: 100vw - overflow-y: auto - -.pi-accordion - & > .container:first-child > .item:first-child > .title:first-child - padding-left: 0 - font-size: 1.5em - font-weight: 700 - - & > .container:first-child > .item.yah:first-child > .title:first-child - margin-left: -20px !important - - .item - overflow: hidden - - .title - color: $dark-grey - position: relative - padding: 7.5px 10px 7.5px 18px - cursor: pointer - transition: 0.3s - - &:hover - color: $blue - - a.item > .title - color: black - - &:hover - color: $blue - - div.item > .title - &:before - content: "" - position: absolute - top: 12px - left: 2px - border-style: solid - border-width: 5px 0 5px 8px - border-color: transparent transparent transparent $blue - transform: rotate(0deg) - transition: 0.3s - - .wrapper - position: relative - width: 100% - transition: height 0.3s - - .content - padding-left: 20px - opacity: 0 - transition: 0.3s - - .item.on - & > .title:before - transform: rotate(90deg) - - & > .wrapper > .content - opacity: 1 - - -dt - margin-bottom: 8px - -dd - margin-bottom: 16px - -.pi-pushmenu - display: none - position: fixed - top: 0 - width: 100% - height: 100% - opacity: 0 - transition: opacity 0.3s - - &.on - opacity: 1 - - .overlay - position: fixed - top: 0 - left: 0 - width: 100% - height: 100% - background-color: rgba(0, 0, 0, 0.4) - - .sled - position: absolute - top: 0 - width: 0 - height: 100% - background-color: white - overflow: auto - transition: 0.3s - - &.on .sled - width: 400px - max-width: 100vw - - .top-bar - height: 0 - line-height: 60px - background-color: #444 - - ul - margin-top: 25px - - li - position: relative - display: block - width: 100% - min-height: 45px - padding: 0 60px 0 20px - border-bottom: 1px solid #cccccc - - a - display: inline-block - width: 100% - height: 45px - line-height: 45px - font-family: "Roboto", sans-serif - font-size: 20px - color: $blue - - .button - background: none - padding: 0 - - ul ul - padding: 0 20px - - li - min-height: 40px - - a - height: 40px - line-height: 40px - font-size: 18px - color: lighten(#222222, 20%) - -.push-menu-close-button - position: absolute - top: 0 - right: 0 - width: 50px - height: 50px - - &:before, &:after - content: "" - position: absolute - top: calc(50% - 1px) - left: 25% - width: 50% - height: 2px - background-color: black - - &:before - transform: rotate(45deg) - - &:after - transform: rotate(-45deg) - -#docsContent - position: relative - float: right - width: 100% - - $toc-margin: 15px - $header-clearance: $header-height + 20px - - * + h2, * + h3, * + h4, * + h5, * + h6 - margin-top: 30px - - h1, h2, h3, h4, h5, h6 - line-height: normal - font-weight: 500 - margin-bottom: 30px - padding-bottom: 10px - - // Make sure anchor links aren't hidden by the header - &:before - display: block - content: " " - margin-top: -$header-clearance - height: $header-clearance - visibility: hidden - - h1,h2 - border-bottom: 1px solid #cccccc - - h1 - font-size: 32px - padding-right: 60px - - h2 - font-size: 28px - - h3 - font-size: 24px - font-weight: 300 - margin-bottom: 5px - - h4 - font-size: 20px - margin-bottom: 0px - - h5, h6 - font-size: 16px - font-weight: 500 - - p - font-size: 16px - font-weight: 300 - line-height: 1.75em - - p + p - margin-top: 10px - - code - display: inline-block - box-sizing: border-box - background-color: $light-grey - color: $dark-grey - font-family: $mono-font - vertical-align: baseline - font-size: 14px - padding: 2px 4px - - a code - color: $blue - text-decoration: underline - - pre .pi, pre .s - margin: 0 - padding: 0 - - .highlight code span, code, pre code - font-family: "Roboto Mono", monospace - - code, pre code - color: #303030 - - pre code - padding: 0 - - pre - background-color: #f7f7f7 - display: block - margin: 20px 0 - padding: 15px - position: relative - overflow-x: auto - - h1 code, h2 code, h3 code, h4 code, h5 code, h6 code - font-size: inherit - background-color: transparent - - .includecode - table-layout: fixed - - .includecode, .includecode th, .includecode td - padding: 0 !important - - .includecode th - text-align: right !important - padding: 10px !important - - .includecode th a, .includecode th a code - color: white !important - background-color: transparent !important - - .includecode pre - margin: 0 !important - - ul li - list-style: disc - - ol ul li - list-style: disc - - ol li - list-style: decimal - - ul, ol - margin: 20px 0 - padding-left: 30px - font-weight: 300 - - ul ul, ol ol, ul ol, ol ul - margin: 0.75em 0 - - li - margin-bottom: 0.75em - font-size: 16px - line-height: 1.75em - - table - width: 100% - border: 1px solid #ccc - border-spacing: 0 - margin-top: 30px - margin-bottom: 30px - - thead, tr:nth-child(even) - background-color: $light-grey - - thead - background-color: #555 - color: white - - th, td - padding: 8px - text-align: left - margin: 0 - - th - font-weight: normal - - td - font-size: 0.85em - - #editPageButton - position: absolute - top: -25px - right: 5px - width: 50px - height: 50px - line-height: 50px - border-radius: 50% - white-space: nowrap - text-indent: 50px - overflow: hidden - background: $blue url(/images/icon-pencil.svg) no-repeat - background-position: 12px 10px - background-size: 29px 29px - - #markdown-toc, #TableOfContents - margin-bottom: 20px - - ul, li - list-style: disc - color: $blue - - ul - padding: 0 15px - margin: 0 - - li - padding: 0 - line-height: 1.5em - margin-bottom: 0 - - a - position: relative - color: $blue - font-weight: 700 - - img - max-width: 100% - - #TableOfContents - & > ul > li - list-style: none - - ul, li - list-style: disk - - a.button - border-radius: 2px - text-decoration: none - - &:visited - color: white - - a.issue - margin-left: 0px - -.gridPageHome .flyout-button - display: none - -.feedback--no - margin-left: 1em - -.feedback--prompt - margin-bottom: 1em - -.feedback--response - margin-top: 1em - -.feedback--button__disabled - background-color: #f7f7f7 - color: graytext - -.feedback--response__hidden - display: none - -.fixed footer - position: fixed - bottom: 0 - -.miceType - clear: both - font-size: 11px - line-height: 18px - color: #aaa - - -html.search #docsContent - position: relative - float: none - width: 90% - max-width: 850px - margin: 0 auto - - #editPageButton - display: none - - table - border: 0 - margin-bottom: 0 - - td - padding: 0 - - h1 - margin-bottom: 0 - border-bottom: 0 - padding-bottom: 0 - padding-left: 8px - -// ocean nodes -$ocean-nodes-padding-Y: 60px -$ocean-nodes-main-margin-bottom: 60px -$ocean-nodes-h3-margin-bottom: 30px - -// video -$video-section-height: 200px - -// features -$features-h3-margin-bottom: 20px -$feature-box-div-width: 100% -$feature-box-margin-bottom: 0 -$feature-box-div-margin-bottom: 40px - - -// Home-specific - -#home - &.flip-nav, &.open-nav - .logo - background-image: url(/images/nav_logo2.svg) - - #hero - margin-bottom: 0 - padding-bottom: 1px - - .main-section - padding: 0 10px - margin-bottom: 30px - - #vendorStrip - display: none - - // Add logo to CNCF section - section#cncf - padding-top: 60px - padding-bottom: 140px - background-image: url(/images/cncf-color.svg) - background-position: center 100px - background-repeat: no-repeat - background-size: 300px - - - -// OCEAN NODES -#oceanNodes - padding-top: $ocean-nodes-padding-Y - padding-bottom: $ocean-nodes-padding-Y - - a - color: $blue - - .main-section - margin-bottom: $ocean-nodes-padding-Y - min-height: 160px - - .image-wrapper - max-width: 75% - margin: 0 auto 20px - text-align: center - - img - width: 100% - max-width: 160px - - .main-section:first-child - .image-wrapper - max-width: 100% - - img - max-width: 491px - - h3 - margin-bottom: $ocean-nodes-h3-margin-bottom - - -// Video thingy -#video - height: $video-section-height - -#video - width: 100% - position: relative - background-position: center center - background-size: cover - - - & > .light-text - display: none - position: absolute - top: 50% - left: 75% - width: 575px - padding-right: 80px - transform: translate(-50%, -50%) - color: white - - h2 - font-size: 32px - line-height: 44px - margin-bottom: 20px - - p - margin-bottom: 20px - - #desktopKCButton - position: absolute - font-size: 18px - background-color: $dark-grey - border-radius: 8px - color: $white - padding: 20px 10px 20px 10px - - #desktopShowVideoButton - position: relative - font-size: 24px - background-color: white - border-radius: 8px - color: $blue - padding: 15px 30px 15px 80px - margin-bottom: 15px - - &:before - content: "" - position: absolute - @include pureCenter(40px) - width: 0 - height: 0 - border-style: solid - border-width: 10px 0 10px 20px - border-color: transparent transparent transparent $blue - - #mobileShowVideoButton - @include pureCenter - width: 80px - height: 80px - border-radius: 50% - background-color: transparent - border: 5px solid rgba(255, 255, 255, 0.2) - overflow: visible - - &:after - @include pureCenter - left: 40px - content: "" - width: 0 - height: 0 - border-style: solid - border-width: 20px 0 20px 30px - border-color: transparent transparent transparent #ffffff - -#videoPlayer - @include fullScreen - background-color: rgba(0, 0, 0, 0.9) - display: none - - iframe - @include pureCenter - @include maintain-aspect-ratio - - #closeButton - position: absolute - top: 20px - right: 20px - width: 50px - height: 50px - border: 2px solid transparent - transition: 0.3s - - &:before, &:after - content: "" - position: absolute - top: calc(50% - 1px) - left: 10% - width: 80% - height: 2px - background-color: white - - &:before - transform: rotate(45deg) - - &:after - transform: rotate(-45deg) - - &:hover - border-color: white - -// KubeWeekly -#kubeweekly - background-color: $light-grey - padding-top: 60px - padding-bottom: 140px - background-size: auto - font-family: "Roboto Mono", monospace !important - font-size: 24px - font-weight: bold - - h5 - font-size: 20px - -.subscribe-button - border-radius: 6px - padding: 0 20px - line-height: 31px - color: white - background-color: blue - text-decoration: none - font-size: 14px - -// Features -#features - padding-top: 140px - background-color: $light-grey - background-image: url(/images/wheel.svg) - background-position: center 60px - background-repeat: no-repeat - background-size: 60px - -.feature-box - //padding: 50px 0 - width: 100% - overflow: hidden - clear: both - - h4 - line-height: normal - margin-bottom: 15px - - & > div:first-child - float: left - - & > div:last-child - float: right - -#features - h3 - margin-bottom: $features-h3-margin-bottom - - .feature-box - margin-bottom: $feature-box-margin-bottom - - & > div - width: $feature-box-div-width - margin-bottom: $feature-box-div-margin-bottom - - - - -// Community - -#community, .gridPage - &.open-nav, &.flip-nav - .logo - background-image: url(/images/nav_logo2.svg) - - #hero - padding-bottom: 20px - - #mainContent - padding: 20px 0 - - .main-section - max-width: none - - a - color: $blue - - .content - margin-bottom: 30px - padding: 30px 0 - - h1, h2, h3, h4, h5, h6, p - line-height: normal - max-width: 1200px - padding: 0 20px - margin: 0 auto 20px - - &:nth-child(even) - background-color: $light-grey - - .company-logos - text-align: center - max-width: 1200px - margin: 0 auto - - img - width: auto - margin: 10px - background-color: $light-grey - - .partner-logos - text-align: center - max-width: 1200px - margin: 0 auto - - img - width: auto - margin: 10px - background-color: $white - box-shadow: 0 5px 5px rgba(0,0,0,.24),0 0 5px rgba(0,0,0,.12) - - #calendarMeetings - position: relative - width: 80vw - height: 60vw - max-width: 1200px - max-height: 900px - margin: 20px auto - - #calendarEvents - position: relative - width: 80vw - height: 30vw - max-width: 1200px - max-height: 450px - margin: 20px auto - - iframe - position: absolute - border: 0 - width: 100% - height: 100% - -// Dialog -.ui-icon - display: inline-block !important - -#feature-state-dialog-link - text-decoration: none !important - padding: 5px !important - a:visited - color: #454545 !important - a code - display: inline-block !important - box-sizing: border-box !important - background-color: #f7f7f7 !important - color: #303030 !important - font-family: "Roboto Mono", monospace !important - vertical-align: baseline !important - font-size: 14px !important - font-weight: bold !important - padding: 0px 4px !important - -#feature-state-dialog - background: #fff !important - border: 1px solid #ddd !important - padding: 0.5em 1em !important - - ul, li - list-style: disc !important - margin: 4px 12px !important - - p - margin: 8px 0px !important - - code - display: inline-block !important - box-sizing: border-box !important - background-color: #f7f7f7 !important - color: #303030 !important - font-family: "Roboto Mono", monospace !important - vertical-align: baseline !important - font-size: 14px !important - font-weight: bold !important - padding: 0px 4px !important - -.ui-dialog - background: #f7f7f7 !important - padding: 0.5em - -.ui-dialog-content - position: relative - float: right - width: 100% - - $toc-margin: 15px - $header-clearance: $header-height + 20px - - * + h2, * + h3, * + h4, * + h5, * + h6 - margin-top: 30px - - h1, h2, h3, h4, h5, h6 - line-height: normal - font-weight: 500 - margin-bottom: 30px - padding-bottom: 10px - - // Make sure anchor links aren't hidden by the header - &:before - display: block - content: " " - margin-top: -$header-clearance - height: $header-clearance - visibility: hidden - - h1,h2 - border-bottom: 1px solid #cccccc - - h1 - font-size: 32px - padding-right: 60px - - h2 - font-size: 28px - - h3 - font-size: 24px - font-weight: 300 - margin-bottom: 5px - - h4 - font-size: 20px - margin-bottom: 0px - - h5, h6 - font-size: 16px - font-weight: 500 - - p - font-size: 16px - font-weight: 300 - line-height: 1.75em - - p + p - margin-top: 10px - - code - display: inline-block - box-sizing: border-box - background-color: $light-grey - color: $dark-grey - font-family: $mono-font - vertical-align: baseline - font-size: 14px - font-weight: bold - padding: 2px 4px - - a code - color: $blue - text-decoration: underline - - pre .pi, pre .s - margin: 0 - padding: 0 - - .highlight code span, code, pre code - font-family: "Roboto Mono", monospace - - code, pre code - color: #303030 - - pre code - padding: 0 - - pre - background-color: #f7f7f7 - display: block - margin: 20px 0 - padding: 15px - position: relative - overflow-x: auto - - h1 code, h2 code, h3 code, h4 code, h5 code, h6 code - font-family: inherit - font-size: inherit - background-color: transparent - - .includecode - table-layout: fixed - - .includecode, .includecode th, .includecode td - padding: 0 !important - - .includecode th - text-align: right !important - padding: 10px !important - - .includecode th a, .includecode th a code - color: white !important - background-color: transparent !important - - .includecode pre - margin: 0 !important - - ul li - list-style: disc - - ol li - list-style: decimal - - ul, ol - margin: 20px 0 - padding-left: 30px - font-weight: 300 - - ul ul, ol ol, ul ol, ol ul - margin: 0.75em 0 - - li - margin-bottom: 0.75em - font-size: 16px - line-height: 1.75em - - table - width: 100% - border: 1px solid #ccc - border-spacing: 0 - margin-top: 30px - margin-bottom: 30px - - thead, tr:nth-child(even) - background-color: $light-grey - - thead - background-color: #555 - color: white - - th, td - padding: 8px - text-align: left - margin: 0 - - th - font-weight: normal - - td - font-size: 0.85em - - #editPageButton - position: absolute - top: -25px - right: 5px - width: 50px - height: 50px - line-height: 50px - border-radius: 50% - white-space: nowrap - text-indent: 50px - overflow: hidden - background: $blue url(/images/icon-pencil.svg) no-repeat - background-position: 12px 10px - background-size: 29px 29px - - #markdown-toc - margin-bottom: 20px - - ul, li - list-style: disc - color: $blue - - ul - padding: 0 15px - margin: 0 - - li - padding: 0 - line-height: 1.5em - margin-bottom: 0 - - a - position: relative - color: $blue - font-weight: 700 - - img - max-width: 100% - - a - //font-weight: 700 - text-decoration: underline - - // a:visited - // color: blueviolet - - a.button - border-radius: 2px - text-decoration: none - - &:visited - color: white - - a.issue - margin-left: 0px - -.ui-dialog-buttonpane - background: #f7f7f7 !important - -// Tabs -.ui-widget-header - background: transparent !important - background-color: transparent !important - border: 0px !important - -.ui-tabs - ul, ol, li - padding: 0px !important - list-style: none !important - margin-bottom: 0px !important - margin-left: 4px !important - -.ui-tabs-panel - ul li - list-style: disc !important - - ol li - list-style: decimal !important - -.ui-widget-content - border: 0px !important - - p a - color: $blue - - table - margin: 0px !important - -.ui-tabs .ui-tabs-panel - border: 1px solid #ccc !important - -.ui-tabs-anchor - text-decoration: none !important - -// Talk to us -#talkToUs - h3, h4 - text-align: center - - h3 - margin-bottom: 15px - - h4 - line-height: normal - margin-bottom: 50px - - br - display: none - - #bigSocial - overflow: hidden - - div - width: 100% - float: left - padding: 30px - padding-top: 110px - background-position: center top - background-size: auto - background-repeat: no-repeat - - div:nth-child(1) - background-image: url(/images/twitter_icon.png) - - div:nth-child(2) - background-image: url(/images/github_icon.png) - - div:nth-child(3) - background-image: url(/images/slack_icon.png) - - div:nth-child(4) - background-image: url(/images/stackoverflow_icon.png) - - div + div - margin-top: 20px - margin-left: 0 - - a - display: inline-block - color: $blue - font-size: 24px - font-weight: 400 - text-decoration: none - margin-bottom: 15px - - a, p - text-align: center - width: 100% - - -#home - #talkToUs - .main-section - padding: 30px 0 - - h5 - font-size: 20px - - - #caseStudiesWrapper - position: relative - text-align: center - margin-bottom: 30px - - img - padding-bottom: 1rem - max-width: 100% - - div - position: relative - display: inline-block - vertical-align: top - width: 100% - min-height: 230px - margin-bottom: 60px - padding-right: 1rem - background-position: top center - - p - font-size: 20px - - a - position: absolute - bottom: -30px - left: 50% - transform: translateX(-50%) - color: $blue - font-weight: 400 - - -/* Google Search */ -.cse .gsc-control-cse, .gsc-control-cse - padding: 0 - -.gsc-control-cse table, .gsc-control-cse-en table - margin: 0px !important - -.gsc-above-wrapper-area - border-bottom: 0 - - -/* Bing Search */ -#bing-results-container - margin-top: 30px - margin-left: 20px - -.bing-result - margin-bottom: 20px - -.bing-result-name a - font-size: 16px - color: #0000CC - - -.bing-result-url - color: #008000 - font-size: 13px - -.bing-result-snippet - color: #000 - font-size: 11px - -#bing-pagination-container - margin: 10px - margin-left: 20px - -.bing-page-anchor - text-decoration: none!important - cursor: pointer - color: #0000CC - margin-right: 8px - -// This Sass ensures a sticky footer at the bottom -.page - display: flex - flex-direction: column - min-height: 100vh - - .page-content - flex: 1 - -// 404 page -.container-404 - text-align: center - margin: 3.5rem 0 - - ul li a - font-size: 1.5rem - -// GitHub info/edit buttons -#pre-footer - margin-top: 2rem - - .button - font-size: 1.1rem - - &:first-of-type - margin-right: .75rem - - .lastedit - margin-top: 1rem - font-size: 1rem diff --git a/deprecated/assets/sass/_case-studies.sass b/deprecated/assets/sass/_case-studies.sass deleted file mode 100644 index 1b359c1773..0000000000 --- a/deprecated/assets/sass/_case-studies.sass +++ /dev/null @@ -1,99 +0,0 @@ -// SASS for Case Studies pages go here: - -hr - background-color: #999999 - -h2 - margin-bottom: 15px !important - -.subhead - padding-bottom: 2% !important - padding-top: 0% !important - -.details - margin-left: 1.9% - padding-right: 5% - font-size: 16px !important - padding-bottom: 2% !important - -.section1 - margin-bottom: 3% - .cols - width: 80% !important - margin-left: 6.8% - padding-top: 1.5% - .col1 - width: 52% !important - font-weight: 300 !important - .col2 - width: 46% !important - -.banner2text - width: 63% - padding-top: 10% - padding-left: 0% !important - float: initial !important - text-align: center - margin: 0 auto - position: relative - -.banner3text, .banner4text, .banner5text - width: 63% - padding-left: 0% !important - float: initial !important - text-align: center - margin: 0 auto - position: relative - -.fullcol - float: initial !important - -body footer - background-color: #585858 !important - -.section1 - float: left !important - -.banner1 - padding-left: 11.9% !important - -.banner2 - float: initial !important - padding-bottom: 2% !important - -footer - padding: 0% 7% - -.banner4, .banner3, .banner5 - float: initial !important - -@media screen and (max-width: 910px) - .banner2text - width: 47% - padding-top: 45% - .details - margin-left: 0% - margin-bottom: 3% - .subhead - padding-bottom: 0% !important - padding-top: 0% !important - .banner3 - width: 100% !important - -@media screen and (max-width: 780px) - .section1 .cols - width: 100% !important - margin-left: 0% - padding-top: 5% - .col1, .col2 - width: 100% !important - margin-left: 0% !important - .fullcol - width: 90% !important - margin-left: 5% !important - .banner1 - padding-left: 10% !important - margin-bottom: 6% !important - .banner2text - padding-top: 60% !important - padding-bottom: 2% !important diff --git a/deprecated/assets/sass/_desktop.sass b/deprecated/assets/sass/_desktop.sass deleted file mode 100644 index 1e6a49aef7..0000000000 --- a/deprecated/assets/sass/_desktop.sass +++ /dev/null @@ -1,304 +0,0 @@ -$main-max-width: 1200px -$vendor-strip-height: 44px -$video-section-height: 550px - -@media screen and (min-width: 1100px) - #hamburger - display: none - - ul.global-nav - display: inline-block - - #docs #vendorStrip #searchBox:before - top: 15px - - #vendorStrip - height: $vendor-strip-height - line-height: $vendor-strip-height - - li - a.YAH - &:after - content: "" - display: block - position: absolute - left: 0 - bottom: 0 - width: 100% - height: 4px - background-color: $blue - - - #searchBox - float: right - - - #home - #hero - #vendorStrip - display: block - - #docs - #hero - h1, h5 - text-align: left - - #vendorStrip - ul - float: left - - #searchBox - float: right - width: 250px - - #search - vertical-align: middle - - .flyout-button - display: none - - .logo - position: relative - float: left - display: block - width: 180px - height: 88px - top: 0 - left: 0 - transform: none - background-image: url(../images/nav_logo.svg) - - &.flip-nav, &.open-nav - .logo - background-image: url(../images/nav_logo2.svg) - - - #encyclopedia - padding: 50px 50px 100px 100px - clear: both - - #docsToc - position: relative - float: left - padding: 0 20px - left: 0 - width: 350px - z-index: auto - - .push-menu-close-button - display: none - - #docsContent - width: calc(100% - 400px) - - #editPageButton - right: -25px - - section, header, footer - .main-section - max-width: $main-max-width - - header, #vendorStrip, #encyclopedia, #hero h1, #hero h5, #docs #hero h1, #docs #hero h5, - #community #hero h1, .gridPage #hero h1, #community #hero h5, .gridPage #hero h5 - padding-left: 100px - padding-right: 100px - - #vendorStrip - padding-right: 10px - - #home - section, header - .main-section - max-width: 1000px - - #oceanNodes - .main-section - position: relative - max-width: 830px - - &:nth-child(1) - max-width: 1000px - padding-right: 475px - - h3, p - text-align: left - - .image-wrapper - position: absolute - max-width: 48% - transform: translateY(-50%) - - img - width: 425px - - //.content - // width: 50% - - - #video - height: $video-section-height - position: relative - background-position: center center - background-size: cover - - - - #talkToUs - h4 - br - display: block - - #bigSocial - div - width: calc(25% - 18px) - - div + div - margin-left: 20px - // - //div:nth-child(3), div:nth-child(4) - // margin-top: 0 - - - //#home #bigSocial - // div - // width: calc(33% - 15px) - // - // div:nth-child(3) - // margin-top: inherit - - // FOOTER - footer - width: 100% - background-image: url(../images/texture.png) - background-color: $dark-grey - - .main-section - padding: 20px 0 - - nav - overflow: hidden - margin-bottom: 20px - display: flex - justify-content: space-between - - a - width: auto - float: left - font-size: 24px - font-weight: 300 - white-space: nowrap - - .social - padding: 0 - max-width: 1200px - - div - float: left - - div:last-child - float: right - - #search, #wishField - background-color: transparent - padding: 10px - font-size: 16px - font-weight: 100 - color: white - border: 1px solid white - transition: 0.3s - - &:focus - background-color: $light-grey - color: $dark-grey - - .social a - display: inline-block - background-image: url(../images/social_sprite.svg) - background-repeat: no-repeat - background-size: 350px - width: 50px - height: 50px - border-radius: 5px - margin-right: 10px - - &:hover - background-color: #fff - - span - position: absolute - display: block - height: 0 - overflow: hidden - - a.twitter - background-position: 0 0 - - &:hover - background-position: 0 100% - - a.stack-overflow - background-position: -50px 0 - - &:hover - background-position: -50px 100% - - a.slack - background-position: -100px 0 - - &:hover - background-position: -100px 100% - - a.github - background-position: -150px 0 - - &:hover - background-position: -150px 100% - - a.mailing-list - background-position: -200px 0 - - &:hover - background-position: -200px 100% - - a.calendar - background-position: -250px 0 - - &:hover - background-position: -250px 100% - - a.youtube - background-position: -300px 0 - - &:hover - background-position: -300px 100% - - #community, .gridPage - #hero - text-align: left - - h1 - padding: 20px 100px - - #tryKubernetes - width: auto - background-color: $blue - padding: 0 20px - - #bigSocial - div - width: calc(25% - 18px) - - - #home #caseStudiesWrapper - div - width: 24% - min-height: 260px - -@media screen and (min-width: 1300px) - #vendorStrip - padding-right: 100px - -@media screen and (min-width: 456px) - #vendorStrip - li + li - margin-left: 20px diff --git a/deprecated/assets/sass/_reset.sass b/deprecated/assets/sass/_reset.sass deleted file mode 100755 index 8ef13fead0..0000000000 --- a/deprecated/assets/sass/_reset.sass +++ /dev/null @@ -1,61 +0,0 @@ -html, body - margin: 0 - padding: 0 - - -input, button - outline: none - -button - cursor: pointer - -ul, li - list-style: none - -ul - margin: 0 - padding: 0 - -a - text-decoration: none - -// CONVENIENCE -.clear - display: block - clear: both - -.light-text - color: white - -.right - float: right - -.left - float: left - -.center - text-align: center - - -//mixins -@mixin fullScreen - position: fixed - top: 0 - left: 0 - width: 100vw - height: 100vh - - -@mixin pureCenter($left: 50%, $top: 50%) - position: absolute - top: $top - left: $left - transform: translate(-50%, -50%) - - -@mixin maintain-aspect-ratio ($width-factor: 16, $height-factor: 9, $target-width: 80vw, $target-height: 80vh) - width: $target-width - height: $target-width * ($height-factor / $width-factor) - max-width: $target-height * ($width-factor / $height-factor) - max-height: $target-height - diff --git a/deprecated/assets/sass/_size.sass b/deprecated/assets/sass/_size.sass deleted file mode 100644 index 470a2f5765..0000000000 --- a/deprecated/assets/sass/_size.sass +++ /dev/null @@ -1,86 +0,0 @@ -h1 - font-size: $h1-font-size - line-height: $h1-line-height - -h2 - font-size: $h2-font-size - line-height: $h2-line-height - -h3 - font-size: $h3-font-size - line-height: $h3-line-height - -h4 - font-size: $h4-font-size - line-height: $h4-line-height - -h5 - font-size: $h5-font-size - line-height: $h5-line-height - -p - font-size: $p-font-size - line-height: $p-line-height - -section, header, #vendorStrip - padding-left: $full-width-paddingX - padding-right: $full-width-paddingX - - .main-section - width: $main-width - max-width: $main-max-width - -header - height: $header-height - -.nav-buttons - height: $header-height - line-height: $header-height - - .button + * - margin-left: $nav-buttons-margin-left - -#hamburger - width: $hamburger-size - height: $hamburger-size - -#mainNav - padding: $main-nav-padding - - h5 - margin-bottom: $main-nav-h5-margin-bottom - - h3 - margin-bottom: $main-nav-h3-margin-bottom - - .nav-box - width: $nav-box-width - - .nav-box + .nav-box - margin-left: $nav-box-sibling-margin-left - - .main-secton + .main-section - margin-top: $main-nav-main-sibling-margin-top - - .left .button - height: $main-nav-left-button-size - line-height: $main-nav-left-button-size - font-size: $main-nav-left-button-font-size - -.open-nav, .y-enough - #tryKubernetes - margin-left: $nav-buttons-margin-left - -#hero - padding-top: $hero-padding-top - -#docs - #hero - h1, h5 - padding-left: $full-width-paddingX - padding-right: $full-width-paddingX - -#vendorStrip - height: $vendor-strip-height - line-height: $vendor-strip-height - font-size: $vendor-strip-font-size diff --git a/deprecated/assets/sass/_skin.sass b/deprecated/assets/sass/_skin.sass deleted file mode 100755 index 5f901d03c1..0000000000 --- a/deprecated/assets/sass/_skin.sass +++ /dev/null @@ -1,25 +0,0 @@ -$blue: #3371e3 -$light-grey: #f7f7f7 -$dark-grey: #303030 -$medium-grey: #4c4c4c -$white: #ffffff - -$base-font: 'Roboto', sans-serif -$mono-font: 'Roboto Mono', monospace - - -*, .button - box-sizing: border-box - font-family: $base-font - background: none - margin: 0 - border: 0 - -body - font-family: $base-font - -h1, h2, h5, p - font-weight: 300 - -h3, h4 - font-weight: 400 diff --git a/deprecated/assets/sass/_tablet.sass b/deprecated/assets/sass/_tablet.sass deleted file mode 100644 index a8d1ce36ed..0000000000 --- a/deprecated/assets/sass/_tablet.sass +++ /dev/null @@ -1,250 +0,0 @@ -// TEXT -//$h1-font-size: 60px -//$h1-line-height: 80px -// -//$h2-font-size: 42px -//$h2-line-height: 60px -// -//$h3-font-size: 26px -//$h3-line-height: 32px -// -//$h4-font-size: 26px -//$h4-line-height: 40px -// -//$h5-font-size: 18px -//$h5-line-height: 36px -// -//$p-font-size: 20px -//$p-line-height: 28px - - -// header -$full-width-paddingX: 20px - -$main-width: 100% -$main-max-width: 100% - -$header-height: 80px - -$logo-width: 180px - -$nav-buttons-margin-left: 30px - -$hamburger-size: 50px - - -// main nav -$main-nav-padding: 140px 0 30px -$main-nav-h5-margin-bottom: 1em -$main-nav-h3-margin-bottom: 0.6em -$nav-box-width: 20% -$nav-box-sibling-margin-left: calc(20% / 3) -$main-nav-main-sibling-margin-top: 60px -$main-nav-left-button-size: 50px -$main-nav-left-button-font-size: 18px - - -// hero -$hero-padding-top: 80px -$headline-wrapper-margin-bottom: 40px -$quickstart-button-padding: 0 50px -$vendor-strip-font-size: 16px - -//video -$video-section-height: 400px - -//features -$features-h3-margin-bottom: 40px -$feature-box-margin-bottom: 60px -$feature-box-div-margin-bottom: 0 -$feature-box-div-width: 45% - - - -//////////////////////////////////////////////////////////////////////////////////////////////////// -//////////////////////////////////////////////////////////////////////////////////////////////////// -//////////////////////////////////////////////////////////////////////////////////////////////////// - -@media screen and (min-width: 750px) - @import "size" - - p - font-size: 16px - line-height: 24px - letter-spacing: 0.1px - - h1 - font-size: 36px - line-height: 44px - - h3 - font-size: 28px - line-height: 36px - - h4 - font-size: 24px - line-height: 40px - - - - #home - #viewDocs, #tryKubernetes - display: inline-block - - #vendorStrip - display: block - text-align: center - - img - max-height: 24px - vertical-align: middle - margin: 0 30px - - #docs - #vendorStrip - li - a - font-size: 1em - font-weight: normal - - li + li - margin-left: 60px - - - #oceanNodes - h3 - text-align: left - margin-bottom: 18px - - main, .main-section - position: relative - clear: both - display: table - height: 160px - - .content - display: table-cell - position: relative - vertical-align: middle - - .image-wrapper - position: absolute - top: 50% - max-width: 25% - max-height: 100% - transform: translateY(-50%) - width: 100% - - &:nth-child(odd) - padding-right: 210px - - .image-wrapper - right: 0 - text-align: right - - &:nth-child(even) - padding-left: 210px - - .image-wrapper - left: 0 - text-align: left - - &:nth-child(1) - padding-right: 0 - - h3, p - text-align: center - - .image-wrapper - position: relative - display: block - float: none - max-width: 100% - transform: none - - .content - display: block - - img - width: 100% - - - #video - height: $video-section-height - display: block - - & > .light-text - display: block - - #mobileShowVideoButton - display: none - - #features - padding-bottom: 60px - - .feature-box - margin-bottom: 30px - - &:last-child - margin-bottom: 0 - - h3 - margin-bottom: $features-h3-margin-bottom - - .feature-box - & > div - width: $feature-box-div-width - margin-bottom: $feature-box-div-margin-bottom - - #talkToUs - #bigSocial - div - width: calc(50% - 15px) - - div + div - margin-top: 0 - - div:nth-child(2) - margin-left: 20px - - div:nth-child(3) - - div:nth-child(4) - margin-left: 20px - - a - display: inline-block - color: $blue - font-weight: 400 - text-decoration: none - - // FOOTER - footer - nav - text-align: center - a - width: auto - padding: 0 20px - - .social - text-align: center - - div - display: inline-block - - div:last-child - display: block - margin: 0 - - span - display: inline-block - margin-right: 10px - - input - text-align: left - - - - #home #caseStudiesWrapper - div - width: 48% diff --git a/deprecated/assets/sass/style.sass b/deprecated/assets/sass/style.sass deleted file mode 100755 index 25a7ef9e42..0000000000 --- a/deprecated/assets/sass/style.sass +++ /dev/null @@ -1,11 +0,0 @@ -@import "reset" -@import "skin" - -// media queries -@import "base" -@import "tablet" -@import "desktop" - -{{ if .Params.case_study_styles }} -@import "case-studies" -{{ end }} diff --git a/deprecated/content/en/docs/user-journeys/users/application-developer/advanced.md b/deprecated/content/en/docs/user-journeys/users/application-developer/advanced.md deleted file mode 100644 index dde720f1b6..0000000000 --- a/deprecated/content/en/docs/user-journeys/users/application-developer/advanced.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -reviewers: -- chenopis -layout: docsportal -css: /css/style_user_journeys.css -js: https://use.fontawesome.com/4bcc658a89.js, https://cdnjs.cloudflare.com/ajax/libs/prefixfree/1.0.7/prefixfree.min.js -title: Advanced Topics -track: "USERS › APPLICATION DEVELOPER › ADVANCED" -content_template: templates/user-journey-content ---- - -{{% capture overview %}} - -{{< note >}} -This page assumes that you're familiar with core Kubernetes concepts, and are comfortable deploying your own apps. If not, you should review the {{< link text="Intermediate App Developer" url="/docs/user-journeys/users/application-developer/intermediate/" >}} topics first. -{{< /note >}} -After checking out the current page and its linked sections, you should have a better understanding of the following: -* Advanced features that you can leverage in your application -* The various ways of extending the Kubernetes API - -{{% /capture %}} - - -{{% capture body %}} - -## Deploy an application with advanced features - -Now you know the set of API objects that Kubernetes provides. Understanding the difference between a {{< glossary_tooltip term_id="daemonset" >}} and a {{< glossary_tooltip term_id="deployment" >}} is oftentimes sufficient for app deployment. That being said, it's also worth familiarizing yourself with Kubernetes's lesser known features. They can be quite powerful when applied to the right use cases. - -#### Container-level features - -As you may know, it's an antipattern to migrate an entire app (e.g. containerized Rails app, MySQL database, and all) into a single Pod. That being said, there are some very useful patterns that go beyond a 1:1 correspondence between a container and its Pod: - -* **Sidecar container**: Although your Pod should still have a single main container, you can add a secondary container that acts as a helper (see a {{< link text="logging example" url="/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent" >}}). Two containers within a single Pod can communicate {{< link text="via a shared volume" url="/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/" >}}. -* **Init containers**: *Init containers* run before any of a Pod's *app containers* (such as main and sidecar containers). {{< link text="Read more" url="/docs/concepts/workloads/pods/init-containers/" >}}, see an {{< link text="nginx server example" url="/docs/tasks/configure-pod-container/configure-pod-initialization/" >}}, and {{< link text="learn how to debug these containers" url="/docs/tasks/debug-application-cluster/debug-init-containers/" >}}. - -#### Pod configuration - -Usually, you use {{< glossary_tooltip text="labels" term_id="label" >}} and {{< glossary_tooltip text="annotations" term_id="annotation" >}} to attach metadata to your resources. To inject data into your resources, you'd likely create {{< glossary_tooltip text="ConfigMaps" term_id="configmap" >}} (for nonconfidential data) or {{< glossary_tooltip text="Secrets" term_id="secret" >}} (for confidential data). - -Below are some other, lesser-known ways of configuring your resources' Pods: - -* **Taints and Tolerations** - These provide a way for nodes to "attract" or "repel" your Pods. They are often used when an application needs to be deployed onto specific hardware, such as GPUs for scientific computing. {{< link text="Read more" url="/docs/concepts/configuration/taint-and-toleration/" >}}. -* **Downward API** - This allows your containers to consume information about themselves or the cluster, without being overly coupled to the Kubernetes API server. This can be achieved with {{< link text="environment variables" url="/docs/tasks/inject-data-application/environment-variable-expose-pod-information/" >}} or {{< link text="DownwardAPIVolumeFiles" url="/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/" >}}. -* **Pod Presets** - Normally, to mount runtime requirements (such as environmental variables, ConfigMaps, and Secrets) into a resource, you specify them in the resource's configuration file. {{< link text="PodPresets" url="/docs/concepts/workloads/pods/podpreset/" >}} allow you to dynamically inject these requirements instead, when the resource is created. For instance, this allows team A to mount any number of new Secrets into the resources created by teams B and C, without requiring action from B and C. {{< link text="See an example" url="/docs/tasks/inject-data-application/podpreset/" >}}. - -#### Additional API Objects - -{{< note >}} -Before setting up the following resources, check to see if they are the responsibility of your organization's {{< glossary_tooltip text="cluster operators" term_id="cluster-operator" >}}. -{{< /note >}} -* **{{< glossary_tooltip text="Horizontal Pod Autoscaler (HPA)" term_id="horizontal-pod-autoscaler" >}}** - These resources are a great way to automate the process of scaling your application when CPU usage or other {{< link text="custom metrics" url="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md" >}} spike. {{< link text="See an example" url="/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/" >}} to understand how HPAs are set up. - -* **Federated cluster objects** - If you are running an application on multiple Kubernetes clusters using *federation*, you need to deploy the federated version of the standard Kubernetes API objects. For reference, check out the guides for setting up {{< link text="Federated ConfigMaps" url="/docs/tasks/administer-federation/configmap/" >}} and {{< link text="Federated Deployments" url="/docs/tasks/administer-federation/deployment/" >}}. - -## Extend the Kubernetes API - -Kubernetes is designed with extensibility in mind. If the API resources and features mentioned above are not enough for your needs, there are ways to customize its behavior without having to modify core Kubernetes code. - -#### Understand Kubernetes's default behavior - -Before making any customizations, it's important that you understand the general abstraction behind Kubernetes API objects. Although Deployments and Secrets may seem quite different, the following concepts are true for *any* object: - -* **Kubernetes objects are a way of storing structured data about your cluster.** - In the case of Deployments, this data represents desired state (such as "How many replicas should be running?"), but it can also be general metadata (such as database credentials). -* **Kubernetes objects are modified via the {{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}}**. - In other words, you can make `GET` and `POST` requests to a specific resource path (such as `/api/v1/namespaces/default/deployments`) to read and write the corresponding object type. -* **By leveraging the {{< link text="Controller pattern" url="/docs/concepts/api-extension/custom-resources/#custom-controllers" >}}, Kubernetes objects can be used to enforce desired state**. For simplicity, you can think of the Controller pattern as the following continuous loop: - -
    - 1. Check current state (number of replicas, container image, etc) - 2. Compare current state to desired state - 3. Update if there's a mismatch -
    - - These states are obtained from the Kubernetes API. - - {{< note >}} - Not all Kubernetes objects need to have a Controller. Though Deployments trigger the cluster to make state changes, ConfigMaps act purely as storage. - {{< /note >}} -#### Create Custom Resources - -Based on the ideas above, you can define a new {{< link text="Custom Resource" url="/docs/concepts/api-extension/custom-resources/#custom-resources" >}} that is just as legitimate as a Deployment. For example, you might want to define a `Backup` object for periodic backups, if `CronJobs` don't provide all the functionality you need. - -There are two main ways of setting up custom resources: -1. **Custom Resource Definitions (CRDs)** - This method requires the least amount of implementation work. See {{< link text="an example" url="/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/" >}}. -2. **API aggregation** - This method requires some {{< link text="pre-configuration" url="/docs/tasks/access-kubernetes-api/configure-aggregation-layer/" >}} before you actually {{< link text="set up a separate, extension API server" url="/docs/tasks/access-kubernetes-api/setup-extension-api-server/" >}}. - -Note that unlike standard Kubernetes objects, which rely on the built-in {{< link text="`kube-controller-manager`" url="/docs/reference/generated/kube-controller-manager/" >}}, you'll need to write and run your own {{< link text="custom controllers" url="https://github.com/kubernetes/sample-controller" >}}. - -You may also find the following info helpful: -* {{< link text="How to know if custom resources are right for your use case" url="/docs/concepts/api-extension/custom-resources/#should-i-use-a-configmap-or-a-custom-resource" >}} -* {{< link text="How to decide between CRDs and API aggregation" url="/docs/concepts/api-extension/custom-resources/#choosing-a-method-for-adding-custom-resources" >}} - -#### Service Catalog - -If you want to consume or provide complete services (rather than individual resources), **{{< glossary_tooltip text="Service Catalog" term_id="service-catalog" >}}** provides a {{< link text="specification" url="https://github.com/openservicebrokerapi/servicebroker" >}} for doing so. These services are registered using {{< glossary_tooltip text="Service Brokers" term_id="service-broker" >}} (see {{< link text="some examples" url="https://github.com/openservicebrokerapi/servicebroker/blob/master/gettingStarted.md#example-service-brokers" >}}). - -If you do not have a {{< glossary_tooltip text="cluster operator" term_id="cluster-operator" >}} to manage the installation of Service Catalog, you can do so using {{< link text="Helm" url="/docs/tasks/service-catalog/install-service-catalog-using-helm/" >}} or an {{< link text="installer binary" url="/docs/tasks/service-catalog/install-service-catalog-using-sc/" >}}. - - -## Explore additional resources - -#### References - -The following topics are also useful for building more complex applications: - -* {{< link text="Other points of extensibility within Kubernetes" url="/docs/concepts/overview/extending/" >}} - A conceptual overview of where you can hook into the Kubernetes architecture. -* {{< link text="Kubernetes Client Libraries" url="/docs/reference/using-api/client-libraries/" >}} - Useful for building apps that need to interact heavily with the Kubernetes API. - -#### What's next -Congrats on completing the Application Developer user journey! You've covered the majority of features that Kubernetes has to offer. What now? - -* If you'd like to suggest new features or keep up with the latest developments around Kubernetes app development, consider joining a {{< glossary_tooltip term_id="sig" >}} such as {{< link text="SIG Apps" url="https://github.com/kubernetes/community/tree/master/sig-apps" >}}. - -* If you are interested in learning more about the inner workings of Kubernetes (e.g. networking), consider checking out the {{< link text="Cluster Operator journey" url="/docs/user-journeys/users/cluster-operator/foundational/" >}}. - -{{% /capture %}} - - diff --git a/deprecated/content/en/docs/user-journeys/users/application-developer/foundational.md b/deprecated/content/en/docs/user-journeys/users/application-developer/foundational.md deleted file mode 100644 index 43226f1bad..0000000000 --- a/deprecated/content/en/docs/user-journeys/users/application-developer/foundational.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -reviewers: -- chenopis -layout: docsportal -css: /css/style_user_journeys.css -js: https://use.fontawesome.com/4bcc658a89.js, https://cdnjs.cloudflare.com/ajax/libs/prefixfree/1.0.7/prefixfree.min.js -title: Foundational -track: "USERS › APPLICATION DEVELOPER › FOUNDATIONAL" -content_template: templates/user-journey-content ---- - -{{% capture overview %}} -If you're a developer looking to run applications on Kubernetes, this page and its linked topics can help you get started with the fundamentals. Though this page primarily describes development workflows, {{< link text="the subsequent page in the series" url="/docs/home/?path=users&persona=app-developer&level=intermediate" >}} covers more advanced, production setups. - -{{< note >}} -**A quick note**
    This app developer "user journey" is *not* a comprehensive overview of Kubernetes. It focuses more on *what* you develop, test, and deploy to Kubernetes, rather than *how* the underlying infrastructure works.

    Though it's possible for a single person to manage both, in many organizations, it’s common to assign the latter to a dedicated {{< glossary_tooltip text="cluster operator" term_id="cluster-operator" >}}. -{{< /note >}} -{{% /capture %}} - - -{{% capture body %}} -## Get started with a cluster - -#### Web-based environment - -If you're brand new to Kubernetes and simply want to experiment without setting up a full development environment, *web-based environments* are a good place to start: - -* {{< link text="Kubernetes Basics" url="/docs/tutorials/kubernetes-basics/#basics-modules" >}} - Introduces you to six common Kubernetes workflows. Each section walks you through browser-based, interactive exercises complete with their own Kubernetes environment. - -* {{< link text="Katacoda" url="https://www.katacoda.com/courses/kubernetes/playground" >}} - The playground equivalent of the environment used in *Kubernetes Basics* above. Katacoda also provides {{< link text="more advanced tutorials" url="https://www.katacoda.com/courses/kubernetes/" >}}, such as "Liveness and Readiness Healthchecks". - - -* {{< link text="Play with Kubernetes" url="http://labs.play-with-k8s.com/" >}} - A less structured environment than the *Katacoda* playground, for those who are more comfortable with Kubernetes concepts and want to explore further. It supports the ability to spin up multiple nodes. - - -#### Minikube (recommended) - -Web-based environments are easy to access, but are not persistent. If you want to continue exploring Kubernetes in a workspace that you can come back to and change, *Minikube* is a good option. - -Minikube can be installed locally, and runs a simple, single-node Kubernetes cluster inside a virtual machine (VM). This cluster is fully functioning and contains all core Kubernetes components. Many developers have found this sufficient for local application development. - -* {{< link text="Install Minikube" url="/docs/tasks/tools/install-minikube/" >}}. - -* {{< link text="Install kubectl" url="/docs/tasks/tools/install-kubectl/" >}}. ({{< glossary_tooltip text="What is kubectl?" term_id="kubectl" >}}) - -* *(Optional)* {{< link text="Install Docker" url="/docs/setup/production-environment/container-runtimes/#docker" >}} if you plan to run your Minikube cluster as part of a local development environment. - - Minikube includes a Docker daemon, but if you're developing applications locally, you'll want an independent Docker instance to support your workflow. This allows you to create {{< glossary_tooltip text="containers" term_id="container" >}} and push them to a container registry. - - {{< note >}} - Version 1.12 is recommended for full compatibility with Kubernetes, but a few other versions are tested and known to work. - {{< /note >}} - -You can get basic information about your cluster with the commands `kubectl cluster-info` and `kubectl get nodes`. However, to get a good idea of what's really going on, you need to deploy an application to your cluster. This is covered in the next section. - -#### MicroK8s - -On Linux, *MicroK8s* is a good alternative to Minikube for a local -install of Kubernetes: - -* Runs on the native OS, so there is no overhead from running a virtual machine. -* Always provides the latest stable version of Kubernetes, using built-in auto-upgrade functionality. -* Installs in less than a minute. - -* {{< link text="Install microk8s" url="https://microk8s.io/" >}}. - -After you install MicroK8s, you can use its tab-completion -functionality. All MicroK8s commands start with `microk8s.`. Type -`microk8s.` (with the period) and then use the tab key to see a list -of available commands. - -It also includes commands to enable Kubernetes subsystems. For example: - -* the Kubernetes Dashboard -* the DNS service -* GPU passthrough (for NVIDIA) -* Ingress -* Istio -* Metrics server -* Registry -* Storage - -## Deploy an application - -#### Basic workloads - -The following examples demonstrate the fundamentals of deploying Kubernetes apps: - - * **Stateless apps**: {{< link text="Deploy a simple nginx server" url="/docs/tasks/run-application/run-stateless-application-deployment/" >}}. - - * **Stateful apps**: {{< link text="Deploy a MySQL database" url="/docs/tasks/run-application/run-single-instance-stateful-application/" >}}. - -Through these deployment tasks, you'll gain familiarity with the following: - -* General concepts - - * **Configuration files** - Written in YAML or JSON, these files describe the desired state of your application in terms of Kubernetes API objects. A file can include one or more API object descriptions (*manifests*). (See [the example YAML](/docs/tasks/run-application/run-stateless-application-deployment/#creating-and-exploring-an-nginx-deployment) from the stateless app). - - * **{{< glossary_tooltip text="Pods" term_id="pod" >}}** - This is the basic unit for all of the workloads you run on Kubernetes. These workloads, such as *Deployments* and *Jobs*, are composed of one or more Pods. To learn more, check out {{< link text="this explanation of Pods and Nodes" url="/docs/tutorials/kubernetes-basics/explore-intro/" >}}. - -* Common workload objects - * **{{< glossary_tooltip text="Deployment" term_id="deployment" >}}** - The most common way of running *X* copies (Pods) of your application. Supports rolling updates to your container images. - - * **{{< glossary_tooltip text="Service" term_id="service" >}}** - By itself, a Deployment can't receive traffic. Setting up a Service is one of the simplest ways to configure a Deployment to receive and loadbalance requests. Depending on the `type` of Service used, these requests can come from external client apps or be limited to apps within the same cluster. A Service is tied to a specific Deployment using {{< glossary_tooltip text="label" term_id="label" >}} selection. - -The subsequent topics are also useful to know for basic application deployment. - -#### Metadata - -You can also specify custom information about your Kubernetes API objects by attaching key/value fields. Kubernetes provides two ways of doing this: - -* **{{< glossary_tooltip text="Labels" term_id="label" >}}** - Identifying metadata that you can use to sort and select sets of API objects. Labels have many applications, including the following: - - * *To keep the right number of replicas (Pods) running in a Deployment.* The specified label (`app: nginx` in the {{< link text="stateless app example" url="/docs/tasks/run-application/run-stateless-application-deployment/#creating-and-exploring-an-nginx-deployment" >}}) is used to stamp the Deployment's newly created Pods (as the value of the `spec.template.labels` configuration field), and to query which Pods it already manages (as the value of `spec.selector.matchLabels`). - - * *To tie a Service to a Deployment* using the `selector` field, which is demonstrated in the {{< link text="stateful app example" url="/docs/tasks/run-application/run-single-instance-stateful-application/#deploy-mysql" >}}. - - * *To look for specific subset of Kubernetes objects, when you are using {{< glossary_tooltip text="kubectl" term_id="kubectl" >}}.* For instance, the command `kubectl get deployments --selector=app=nginx` only displays Deployments from the nginx app. - -* **{{< glossary_tooltip text="Annotations" term_id="annotation" >}}** - Nonidentifying metadata that you can attach to API objects, usually if you don't intend to use them for sorting purposes. These often serve as supplementary data about an app's deployment, such as Git SHAs, PR numbers, or URL pointers to observability dashboards. - - -#### Storage - -You'll also want to think about storage. Kubernetes provides different types of storage API objects for different storage needs: - -* **{{< glossary_tooltip text="Volumes" term_id="volume" >}}** - Let you define storage for your cluster that is tied to the lifecycle of a Pod. It is therefore more persistent than container storage. Learn {{< link text="how to configure volume storage" url="/docs/tasks/configure-pod-container/configure-volume-storage/" >}}, or {{< link text="read more about volume storage" url="/docs/concepts/storage/volumes/" >}}. - -* **{{< glossary_tooltip text="PersistentVolumes" term_id="persistent-volume" >}}** and **{{< glossary_tooltip text="PersistentVolumeClaims" term_id="persistent-volume-claim" >}}** - Let you define storage at the cluster level. Typically a cluster operator defines the PersistentVolume objects for the cluster, and cluster users (application developers, you) define the PersistentVolumeClaim objects that your application requires. Learn {{< link text="how to set up persistent storage for your cluster" url="/docs/tasks/configure-pod-container/configure-persistent-volume-storage/" >}} or {{< link text="read more about persistent volumes" url="/docs/concepts/storage/persistent-volumes/" >}}. - -#### Configuration - -To avoid having to unnecessarily rebuild your container images, you should decouple your application's *configuration data* from the code required to run it. There are a couple ways of doing this, which you should choose according to your use case: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    ApproachType of DataHow it's mountedExample
    Using a manifest's container definitionNon-confidentialEnvironment variableCommand-line flag
    Using {{< glossary_tooltip text="ConfigMaps" term_id="configmap" >}}Non-confidentialEnvironment variable OR local filenginx configuration
    Using {{< glossary_tooltip text="Secrets" term_id="secret" >}}ConfidentialEnvironment variable OR local fileDatabase credentials
    - -{{< note >}} -If you have any data that you want to keep private, you should be using a Secret. Otherwise there is nothing stopping that data from being exposed to malicious users. -{{< /note >}} - -## Understand basic Kubernetes architecture - -As an app developer, you don't need to know everything about the inner workings of Kubernetes, but you may find it helpful to understand it at a high level. - -#### What Kubernetes offers - -Say that your team is deploying an ordinary Rails application. You've run some calculations and determined that you need five instances of your app running at any given time, in order to handle external traffic. - -If you're not running Kubernetes or a similar automated system, you might find the following scenario familiar: - -{{< note >}} -1. One instance of your app (a complete machine instance or just a container) goes down. - -1. Because your team has monitoring set up, this pages the person on call. - -1. The on-call person has to go in, investigate, and manually spin up a new instance. - -1. Depending how your team handles DNS/networking, the on-call person may also need to also update the service discovery mechanism to point at the IP of the new Rails instance rather than the old. -{{< /note >}} - -This process can be tedious and also inconvenient, especially if (2) happens in the early hours of the morning! - -**If you have Kubernetes set up, however, manual intervention is not as necessary.** The Kubernetes {{< link text="control plane" url="/docs/concepts/overview/components/#master-components" >}}, which runs on your cluster's master node, gracefully handles (3) and (4) on your behalf. As a result, Kubernetes is often referred to as a *self-healing* system. - -There are two key parts of the control plane that facilitate this behavior: the *Kubernetes API server* and the *Controllers*. - -#### Kubernetes API server - -For Kubernetes to be useful, it needs to know *what* sort of cluster state you want it to maintain. Your YAML or JSON *configuration files* declare this desired state in terms of one or more API objects, such as {{< glossary_tooltip text="Deployments" term_id="deployment" >}}. To make updates to your cluster's state, you submit these files to the {{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}} server (`kube-apiserver`). - -Examples of state include but are not limited to the following: - -* The applications or other workloads to run -* The container images for your applications and workloads -* Allocation of network and disk resources - -Note that the API server is just the gateway, and that object data is actually stored in a highly available datastore called {{< link text="*etcd*" url="https://github.com/coreos/etcd" >}}. For most intents and purposes, though, you can focus on the API server. Most reads and writes to cluster state take place as API requests. - -For more information, see {{< link text="Understanding Kubernetes Objects" url="/docs/concepts/overview/working-with-objects/kubernetes-objects/" >}}. - -#### Controllers - -Once you’ve declared your desired state through the Kubernetes API, the *controllers* work to make the cluster’s current state match this desired state. - -The standard controller processes are {{< link text="`kube-controller-manager`" url="/docs/reference/generated/kube-controller-manager/" >}} and {{< link text="`cloud-controller-manager`" url="/docs/concepts/overview/components/#cloud-controller-manager" >}}, but you can also write your own controllers as well. - -All of these controllers implement a *control loop*. For simplicity, you can think of this as the following: - -{{< note >}} -1. What is the current state of the cluster (X)? - -1. What is the desired state of the cluster (Y)? - -1. X == Y ? - - * `true` - Do nothing. - * `false` - Perform tasks to get to Y, such as starting or restarting containers, - or scaling the number of replicas of a given application. Return to 1. -{{< /note >}} - -By continuously looping, these controllers ensure the cluster can pick up new updates and avoid drifting from the desired state. These ideas are covered in more detail {{< link text="here" url="/docs/concepts/" >}}. - -## Additional resources - -The Kubernetes documentation is rich in detail. Here's a curated list of resources to help you start digging deeper. - -### Basic concepts - -* {{< link text="More about the components that run Kubernetes" url="/docs/concepts/overview/components/" >}} - -* {{< link text="Understanding Kubernetes objects" url="/docs/concepts/overview/working-with-objects/kubernetes-objects/" >}} - -* {{< link text="More about Node objects" url="/docs/concepts/architecture/nodes/" >}} - -* {{< link text="More about Pod objects" url="/docs/concepts/workloads/pods/pod-overview/" >}} - -### Tutorials - -* {{< link text="Kubernetes Basics" url="/docs/tutorials/kubernetes-basics/" >}} - -* {{< link text="Hello Minikube" url="/docs/tutorials/stateless-application/hello-minikube/" >}} *(Runs on Mac only)* - -* {{< link text="Kubernetes object management" url="/docs/tutorials/object-management-kubectl/object-management/" >}} - -### What's next - -If you feel fairly comfortable with the topics on this page and want to learn more, check out the following user journeys: - -* {{< link text="Intermediate App Developer" url="/docs/user-journeys/users/application-developer/intermediate/" >}} - Dive deeper, with the next level of this journey. -* {{< link text="Foundational Cluster Operator" url="/docs/user-journeys/users/cluster-operator/foundational/" >}} - Build breadth, by exploring other journeys. - -{{% /capture %}} diff --git a/deprecated/content/en/docs/user-journeys/users/application-developer/intermediate.md b/deprecated/content/en/docs/user-journeys/users/application-developer/intermediate.md deleted file mode 100644 index 1a3915f224..0000000000 --- a/deprecated/content/en/docs/user-journeys/users/application-developer/intermediate.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -reviewers: -- chenopis -layout: docsportal -css: /css/style_user_journeys.css -js: https://use.fontawesome.com/4bcc658a89.js, https://cdnjs.cloudflare.com/ajax/libs/prefixfree/1.0.7/prefixfree.min.js -title: Intermediate -track: "USERS › APPLICATION DEVELOPER › INTERMEDIATE" -content_template: templates/user-journey-content ---- - - -{{% capture overview %}} - -{{< note >}} - This page assumes that you've experimented with Kubernetes before. At this point, you should have basic experience interacting with a Kubernetes cluster (locally with Minikube, or elsewhere), and using API objects like Deployments to run your applications.

    If not, you should review the {{< link text="Beginner App Developer" url="/docs/user-journeys/users/application-developer/foundational/" >}} topics first. -{{< /note >}} -After checking out the current page and its linked sections, you should have a better understanding of the following: - -* Additional Kubernetes workload patterns, beyond Deployments -* What it takes to make a Kubernetes application production-ready -* Community tools that can improve your development workflow - -{{% /capture %}} - - -{{% capture body %}} - -## Learn additional workload patterns - -As your Kubernetes use cases become more complex, you may find it helpful to familiarize yourself with more of the toolkit that Kubernetes provides. {{< link text="Basic workload" url="/docs/user-journeys/users/application-developer/foundational/#section-2" >}} objects like {{< glossary_tooltip text="Deployments" term_id="deployment" >}} make it straightforward to run, update, and scale applications, but they are not ideal for every scenario. - -The following API objects provide functionality for additional workload types, whether they are *persistent* or *terminating*. - -#### Persistent workloads - -Like Deployments, these API objects run indefinitely on a cluster until they are manually terminated. They are best for long-running applications. - -* **{{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}}** - Like Deployments, StatefulSets allow you to specify that a - certain number of replicas should be running for your application. - - {{< note >}} It's misleading to say that Deployments can't handle stateful workloads. Using {{< glossary_tooltip text="PersistentVolumes" term_id="persistent-volume" >}}, you can persist data beyond the lifecycle of any individual Pod in your Deployment. - {{< /note >}} - - However, StatefulSets can provide stronger guarantees about "recovery" behavior than Deployments. StatefulSets maintain a sticky, stable identity for their Pods. The following table provides some concrete examples of what this might look like: - - | | Deployment | StatefulSet | - |---|---|---| - | **Example Pod name** | `example-b1c4` | `example-0` | - | **When a Pod dies** | Reschedule on *any* node, with new name `example-a51z` | Reschedule on same node, as `example-0` | - | **When a node becomes unreachable** | Pod(s) are scheduled onto new node, with new names | Pod(s) are marked as "Unknown", and aren't rescheduled unless the Node object is forcefully deleted | - - In practice, this means that StatefulSets are best suited for scenarios where replicas (Pods) need to coordinate their workloads in a strongly consistent manner. Guaranteeing an identity for each Pod helps avoid {{< link text="split-brain" url="https://en.wikipedia.org/wiki/Split-brain_(computing)" >}} side effects in the case when a node becomes unreachable ({{< link text="network partition" url="https://en.wikipedia.org/wiki/Network_partition" >}}). This makes StatefulSets a great fit for distributed datastores like Cassandra or Elasticsearch. - - -* **{{< glossary_tooltip text="DaemonSets" term_id="daemonset" >}}** - DaemonSets run continuously on every node in your cluster, even as nodes are added or swapped in. This guarantee is particularly useful for setting up global behavior across your cluster, such as: - - * Logging and monitoring, from applications like `fluentd` - * Network proxy or {{< link text="service mesh" url="https://www.linux.com/news/whats-service-mesh-and-why-do-i-need-one" >}} - - -#### Terminating workloads - -In contrast to Deployments, these API objects are finite. They stop once the specified number of Pods have completed successfully. - -* **{{< glossary_tooltip text="Jobs" term_id="job" >}}** - You can use these for one-off tasks like running a script or setting up a work queue. These tasks can be executed sequentially or in parallel. These tasks should be relatively independent, as Jobs do not support closely communicating parallel processes. {{< link text="Read more about Job patterns" url="/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-patterns" >}}. - -* **{{< glossary_tooltip text="CronJobs" term_id="cronjob" >}}** - These are similar to Jobs, but allow you to schedule their execution for a specific time or for periodic recurrence. You might use CronJobs to send reminder emails or to run backup jobs. They are set up with a similar syntax as *crontab*. - -#### Other resources - -For more info, you can check out {{< link text="a list of additional Kubernetes resource types" url="/docs/reference/kubectl/overview/#resource-types" >}} as well as the {{< link text="API reference docs" url="{{ reference_docs_url }}" >}}. - -There may be additional features not mentioned here that you may find useful, which are covered in the {{< link text="full Kubernetes documentation" url="/docs/home/?path=browse" >}}. - -## Deploy a production-ready workload - -The beginner tutorials on this site, such as the {{< link text="Guestbook app" url="/docs/tutorials/stateless-application/guestbook/" >}}, are geared towards getting workloads up and running on your cluster. This prototyping is great for building your intuition around Kubernetes! However, in order to reliably and securely promote your workloads to production, you need to follow some additional best practices. - -#### Declarative configuration - -You are likely interacting with your Kubernetes cluster via {{< glossary_tooltip text="kubectl" term_id="kubectl" >}}. kubectl can be used to debug the current state of your cluster (such as checking the number of nodes), or to modify live Kubernetes objects (such as updating a workload's replica count with `kubectl scale`). - -When using kubectl to update your Kubernetes objects, it's important to be aware that different commands correspond to different approaches: - -* {{< link text="Purely imperative" url="/docs/tutorials/object-management-kubectl/imperative-object-management-command/" >}} -* {{< link text="Imperative with local configuration files" url="/docs/tutorials/object-management-kubectl/imperative-object-management-configuration/" >}} (typically YAML) -* {{< link text="Declarative with local configuration files" url="/docs/tutorials/object-management-kubectl/declarative-object-management-configuration/" >}} (typically YAML) - -There are pros and cons to each approach, though the declarative approach (such as `kubectl apply -f`) may be most helpful in production. With this approach, you rely on local YAML files as the source of truth about your desired state. This enables you to version control your configuration, which is helpful for code reviews and audit tracking. - -For additional configuration best practices, familiarize yourself with {{< link text="this guide" url="/docs/concepts/configuration/overview/" >}}. - -#### Security - -You may be familiar with the *principle of least privilege*---if you are too generous with permissions when writing or using software, the negative effects of a compromise can escalate out of control. Would you be cautious handing out `sudo` privileges to software on your OS? If so, you should be just as careful when granting your workload permissions to the {{< glossary_tooltip text="Kubernetes API" term_id="kubernetes-api" >}} server! The API server is the gateway for your cluster's source of truth; it provides endpoints to read or modify cluster state. - -You (or your {{< glossary_tooltip text="cluster operator" term_id="cluster-operator" >}}) can lock down API access with the following: - -* **{{< glossary_tooltip text="ServiceAccounts" term_id="service-account" >}}** - An "identity" that your Pods can be tied to -* **{{< glossary_tooltip text="RBAC" term_id="rbac" >}}** - One way of granting your ServiceAccount explicit permissions - -For even more comprehensive reading about security best practices, consider checking out the following topics: - -* {{< link text="Authentication" url="/docs/reference/access-authn-authz/authentication/" >}} (Is the user who they say they are?) -* {{< link text="Authorization" url="/docs/admin/authorization/" >}} (Does the user actually have permissions to do what they're asking?) - -#### Resource isolation and management - -If your workloads are operating in a *multi-tenant* environment with multiple teams or projects, your container(s) are not necessarily running alone on their node(s). They are sharing node resources with other containers which you do not own. - -Even if your cluster operator is managing the cluster on your behalf, it is helpful to be aware of the following: - -* **{{< glossary_tooltip text="Namespaces" term_id="namespace" >}}**, used for isolation -* **{{< link text="Resource quotas" url="/docs/concepts/policy/resource-quotas/" >}}**, which affect what your team's workloads can use -* **{{< link text="Memory" url="/docs/tasks/configure-pod-container/assign-memory-resource/" >}} and {{< link text="CPU" url="/docs/tasks/configure-pod-container/assign-cpu-resource/" >}} requests**, for a given Pod or container -* **{{< link text="Monitoring" url="/docs/tasks/debug-application-cluster/resource-usage-monitoring/" >}}**, both on the cluster level and the app level - -This list may not be completely comprehensive, but many teams have existing processes that take care of all this. If this is not the case, you'll find the Kubernetes documentation fairly rich in detail. - -## Improve your dev workflow with tooling - -As an app developer, you'll likely encounter the following tools in your workflow. - -#### kubectl - -`kubectl` is a command-line tool that allows you to easily read or modify your Kubernetes cluster. It provides convenient, short commands for common operations like scaling app instances and getting node info. How does kubectl do this? It's basically just a user-friendly wrapper for making API requests. It's written using {{< link text="client-go" url="https://github.com/kubernetes/client-go/#client-go" >}}, the Go library for the Kubernetes API. - -To learn about the most commonly used kubectl commands, check out the {{< link text="kubectl cheatsheet" url="/docs/reference/kubectl/cheatsheet/" >}}. It explains topics such as the following: - -* {{< link text="kubeconfig files" url="/docs/tasks/access-application-cluster/configure-access-multiple-clusters/" >}} - Your kubeconfig file tells kubectl what cluster to talk to, and can reference multiple clusters (such as dev and prod). -* {{< link text="The various output formats available" url="/docs/reference/kubectl/cheatsheet/#formatting-output" >}} - This is useful to know when you are using `kubectl get` to list information about certain API objects. - -* {{< link text="The JSONPath output format" url="/docs/reference/kubectl/jsonpath/" >}} - This is related to the output formats above. JSONPath is especially useful for parsing specific subfields out of `kubectl get` output (such as the URL of a {{< glossary_tooltip text="Service" term_id="service" >}}). - -* {{< link text="`kubectl run` vs `kubectl apply`" url="/docs/reference/kubectl/conventions/" >}} - This ties into the [declarative configuration](#declarative-configuration) discussion in the previous section. - -For the full list of kubectl commands and their options, check out {{< link text="the reference guide" url="/docs/reference/generated/kubectl/kubectl-commands" >}}. - -#### Helm - -To leverage pre-packaged configurations from the community, you can use **{{< glossary_tooltip text="Helm charts" term_id="helm-chart" >}}**. - -Helm charts package up YAML configurations for specific apps like Jenkins and Postgres. You can then install and run these apps on your cluster with minimal extra configuration. This approach makes the most sense for "off-the-shelf" components which do not require much custom implementation logic. - -For writing your own Kubernetes app configurations, there is a {{< link text="thriving ecosystem of tools" url="https://docs.google.com/a/heptio.com/spreadsheets/d/1FCgqz1Ci7_VCz_wdh8vBitZ3giBtac_H8SBw4uxnrsE/edit?usp=drive_web" >}} that you may find useful. - -## Explore additional resources - -#### References -Now that you're fairly familiar with Kubernetes, you may find it useful to browse the following reference pages. Doing so provides a high level view of what other features may exist: - -* {{< link text="Commonly used `kubectl` commands" url="/docs/reference/kubectl/cheatsheet/" >}} -* {{< link text="Kubernetes API reference" url="{{ reference_docs_url }}" >}} -* {{< link text="Standardized Glossary" url="/docs/reference/glossary/" >}} - -In addition, {{< link text="the Kubernetes Blog" url="https://kubernetes.io/blog/" >}} often has helpful posts on Kubernetes design patterns and case studies. - -#### What's next -If you feel fairly comfortable with the topics on this page and want to learn more, check out the following user journeys: - -* {{< link text="Advanced App Developer" url="/docs/user-journeys/users/application-developer/advanced/" >}} - Dive deeper, with the next level of this journey. -* {{< link text="Foundational Cluster Operator" url="/docs/user-journeys/users/cluster-operator/foundational/" >}} - Build breadth, by exploring other journeys. -{{% /capture %}} - - diff --git a/deprecated/content/en/docs/user-journeys/users/cluster-operator/foundational.md b/deprecated/content/en/docs/user-journeys/users/cluster-operator/foundational.md deleted file mode 100644 index e74b7c5964..0000000000 --- a/deprecated/content/en/docs/user-journeys/users/cluster-operator/foundational.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -reviewers: -- chenopis -layout: docsportal -css: /css/style_user_journeys.css -js: https://use.fontawesome.com/4bcc658a89.js, https://cdnjs.cloudflare.com/ajax/libs/prefixfree/1.0.7/prefixfree.min.js -title: Foundational -track: "USERS › CLUSTER OPERATOR › FOUNDATIONAL" -content_template: templates/user-journey-content ---- - -{{% capture overview %}} - -If you want to learn how to get started managing and operating a Kubernetes cluster, this page and the linked topics introduce you to the foundational concepts and tasks. -This page introduces you to a Kubernetes cluster and key concepts to understand and manage it. The content focuses primarily on the cluster itself rather than the software running within the cluster. - -{{% /capture %}} - - - -{{% capture body %}} - -## Get an overview of Kubernetes - -If you have not already done so, start your understanding by reading through [What is Kubernetes?](/docs/concepts/overview/what-is-kubernetes/), which introduces a number of basic concepts and terms. - -Kubernetes is quite flexible, and a cluster can be run in a wide variety of places. You can interact with Kubernetes entirely on your own laptop or local development machine with it running within a virtual machine. Kubernetes can also run on virtual machines hosted either locally or in a cloud provider, and you can run a Kubernetes cluster on bare metal. - -A cluster is made up of one or more [Nodes](/docs/concepts/architecture/nodes/); where a node is a physical or virtual machine. -If there is more than one node in your cluster then the nodes are connected with a [cluster network](/docs/concepts/cluster-administration/networking/). -Regardless of how many nodes, all Kubernetes clusters generally have the same components, which are described in [Kubernetes Components](/docs/concepts/overview/components). - - -## Learn about Kubernetes basics - -A good way to become familiar with how to manage and operate a Kubernetes cluster is by setting one up. -One of the most compact ways to experiment with a cluster is [Installing and using Minikube](/docs/tasks/tools/install-minikube/). -Minikube is a command line tool for setting up and running a single-node cluster within a virtual machine on your local laptop or development computer. Minikube is even available through your browser at the [Katacoda Kubernetes Playground](https://www.katacoda.com/courses/kubernetes/playground). -Katacoda provides a browser-based connection to a single-node cluster, using minikube behind the scenes, to support a number of tutorials to explore Kubernetes. You can also leverage the web-based [Play with Kubernetes](http://labs.play-with-k8s.com/) to the same ends - a temporary cluster to play with on the web. - -You interact with Kubernetes either through a dashboard, an API, or using a command-line tool (such as `kubectl`) that interacts with the Kubernetes API. -Be familiar with [Organizing Cluster Access](/docs/concepts/configuration/organize-cluster-access-kubeconfig/) by using configuration files. -The Kubernetes API exposes a number of resources that provide the building blocks and abstractions that are used to run software on Kubernetes. -Learn more about these resources at [Understanding Kubernetes Objects](/docs/concepts/overview/working-with-objects/kubernetes-objects). -These resources are covered in a number of articles within the Kubernetes documentation. - -* [Pod Overview](/docs/concepts/workloads/pods/pod-overview/) - * [Pods](/docs/concepts/workloads/pods/pod/) - * [ReplicaSets](/docs/concepts/workloads/controllers/replicaset/) - * [Deployments](/docs/concepts/workloads/controllers/deployment/) - * [Garbage Collection](/docs/concepts/workloads/controllers/garbage-collection/) - * [Container Images](/docs/concepts/containers/images/) - * [Container Environment Variables](/docs/concepts/containers/container-environment-variables/) -* [Labels and Selectors](/docs/concepts/overview/working-with-objects/labels/) -* [Namespaces](/docs/concepts/overview/working-with-objects/namespaces/) - * [Namespaces Walkthrough](/docs/tasks/administer-cluster/namespaces-walkthrough/) -* [Services](/docs/concepts/services-networking/service/) -* [Annotations](/docs/concepts/overview/working-with-objects/annotations/) -* [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) -* [Secrets](/docs/concepts/configuration/secret/) - -As a cluster operator you may not need to use all these resources, although you should be familiar with them to understand how the cluster is being used. -There are a number of additional resources that you should be aware of, some listed under [Intermediate Resources](/docs/user-journeys/users/cluster-operator/intermediate#section-1). -You should also be familiar with [how to manage kubernetes resources](/docs/concepts/cluster-administration/manage-deployment/) -and [supported versions and version skew between cluster components](/docs/setup/release/version-skew-policy/). - -## Get information about your cluster - -You can [access clusters using the Kubernetes API](/docs/tasks/administer-cluster/access-cluster-api/). -If you are not already familiar with how to do this, you can review the [introductory tutorial](/docs/tutorials/kubernetes-basics/explore-intro/). -Using `kubectl`, you can retrieve information about your Kubernetes cluster very quickly. -To get basic information about the nodes in your cluster run the command `kubectl get nodes`. -You can get more detailed information for the same nodes with the command `kubectl describe nodes`. -You can see the status of the core of kubernetes with the command `kubectl get componentstatuses`. - -Some additional resources for getting information about your cluster and how it is operating include: - -* [Tools for Monitoring Compute, Storage, and Network Resources](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) -* [Resource metrics pipeline](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/) - * [Metrics](/docs/concepts/cluster-administration/controller-metrics/) - -## Explore additional resources - -### Tutorials - -* [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) -* [Configuring Redis with a ConfigMap](/docs/tutorials/configuration/configure-redis-using-configmap/) -* Stateless Applications - * [Deploying PHP Guestbook with Redis](/docs/tutorials/stateless-application/guestbook/) - * [Expose an External IP address to access an application](/docs/tutorials/stateless-application/expose-external-ip-address/) - -{{% /capture %}} diff --git a/deprecated/content/en/docs/user-journeys/users/cluster-operator/intermediate.md b/deprecated/content/en/docs/user-journeys/users/cluster-operator/intermediate.md deleted file mode 100644 index b590e4862c..0000000000 --- a/deprecated/content/en/docs/user-journeys/users/cluster-operator/intermediate.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -reviewers: -- chenopis -layout: docsportal -css: /css/style_user_journeys.css -js: https://use.fontawesome.com/4bcc658a89.js, https://cdnjs.cloudflare.com/ajax/libs/prefixfree/1.0.7/prefixfree.min.js -title: Intermediate -track: "USERS > CLUSTER OPERATOR > INTERMEDIATE" -content_template: templates/user-journey-content ---- - -{{% capture overview %}} - -If you are a cluster operator looking to expand your grasp of Kubernetes, this page and its linked topics extend the information provided on the [foundational cluster operator page](/docs/user-journeys/users/cluster-operator/foundational). From this page you can get information on key Kubernetes tasks needed to manage a complete production cluster. - -{{% /capture %}} - -{{% capture body %}} - -## Work with ingress, networking, storage, and workloads - -Introductions to Kubernetes typically discuss simple stateless applications. As you move into more complex development, testing, and production environments, you need to consider more complex cases: - -Communication: Ingress and Networking - -* [Ingress](/docs/concepts/services-networking/ingress/) - -Storage: Volumes and PersistentVolumes - -* [Volumes](/docs/concepts/storage/volumes/) -* [Persistent Volumes](/docs/concepts/storage/persistent-volumes/) - -Workloads - -* [DaemonSets](/docs/concepts/workloads/controllers/daemonset/) -* [Stateful Sets](/docs/concepts/workloads/controllers/statefulset/) -* [Jobs](/docs/concepts/workloads/controllers/jobs-run-to-completion/) -* [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/) - -Pods - -* [Pod Lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/) - * [Init Containers](/docs/concepts/workloads/pods/init-containers/) - * [Pod Presets](/docs/concepts/workloads/pods/podpreset/) - * [Container Lifecycle Hooks](/docs/concepts/containers/container-lifecycle-hooks/) - -And how Pods work with scheduling, priority, disruptions: - -* [Taints and Tolerations](/docs/concepts/configuration/taint-and-toleration/) -* [Pods and Priority](/docs/concepts/configuration/pod-priority-preemption/) -* [Disruptions](/docs/concepts/workloads/pods/disruptions/) -* [Assigning Pods to Nodes](/docs/concepts/configuration/assign-pod-node/) -* [Managing Compute Resources for Containers](/docs/concepts/configuration/manage-compute-resources-container/) -* [Configuration Best Practices](/docs/concepts/configuration/overview/) - -## Implement security best practices - -Securing your cluster includes work beyond the scope of Kubernetes itself. - -In Kubernetes, you configure access control: - -* [Controlling Access to the Kubernetes API](/docs/reference/access-authn-authz/controlling-access/) -* [Authenticating](/docs/reference/access-authn-authz/authentication/) -* [Using Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/) - -You also configure authorization. That is, you determine not just how users and services authenticate to the API server, or whether they have access, but also what resources they have access to. Role-based access control (RBAC) is the recommended mechanism for controlling authorization to Kubernetes resources. Other authorization modes are available for more specific use cases. - -* [Authorization Overview](/docs/reference/access-authn-authz/authorization/) -* [Using RBAC Authorization](/docs/reference/access-authn-authz/rbac/) - -You should create Secrets to hold sensitive data such as passwords, tokens, or keys. Be aware, however, that there are limitations to the protections that a Secret can provide. See [the Risks section of the Secrets documentation](/docs/concepts/configuration/secret/#risks). - - - -## Implement custom logging and monitoring - -Monitoring the health and state of your cluster is important. Collecting metrics, logging, and providing access to that information are common needs. Kubernetes provides some basic logging structure, and you may want to use additional tools to help aggregate and analyze log data. - -Start with the [basics on Kubernetes logging](/docs/concepts/cluster-administration/logging/) to understand how containers do logging and common patterns. Cluster operators often want to add something to gather and aggregate those logs. See the following topics: - -* [Logging Using Elasticsearch and Kibana](/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) -* [Logging Using Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/) - -Like log aggregation, many clusters utilize additional software to help capture metrics and display them. There is an overview of tools at [Tools for Monitoring Compute, Storage, and Network Resources](/docs/tasks/debug-application-cluster/resource-usage-monitoring/). -Kubernetes also supports a [resource metrics pipeline](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/) which can be used by Horizontal Pod Autoscaler with custom metrics. - -[Prometheus](https://prometheus.io/), another {{< glossary_tooltip text="CNCF" term_id="cncf" >}} project, is a common choice to support capture and temporary collection of metrics. There are several options for installing Prometheus, including using the [stable/prometheus](https://github.com/kubernetes/charts/tree/master/stable/prometheus) [helm](https://helm.sh/) chart, and CoreOS provides a [prometheus operator](https://github.com/coreos/prometheus-operator) and [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus), which adds on Grafana dashboards and common configurations. - -A common configuration on [Minikube](https://github.com/kubernetes/minikube) and some Kubernetes clusters uses [Heapster](https://github.com/kubernetes/heapster) -[along with InfluxDB and Grafana](https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md). -There is a [walkthrough of how to install this configuration in your cluster](https://blog.kublr.com/how-to-utilize-the-heapster-influxdb-grafana-stack-in-kubernetes-for-monitoring-pods-4a553f4d36c9). -As of Kubernetes 1.11, Heapster is deprecated, as per [sig-instrumentation](https://github.com/kubernetes/community/tree/master/sig-instrumentation). See [Prometheus vs. Heapster vs. Kubernetes Metrics APIs](https://brancz.com/2018/01/05/prometheus-vs-heapster-vs-kubernetes-metrics-apis/) for more information alternatives. - -Hosted monitoring, APM, or data analytics services such as [Datadog](https://docs.datadoghq.com/integrations/kubernetes/) or [Instana](https://www.instana.com/supported-integrations/kubernetes-monitoring/) also offer Kubernetes integration. - -## Additional resources - -Cluster Administration: - -* [Troubleshoot Clusters](/docs/tasks/debug-application-cluster/debug-cluster/) -* [Debug Pods and Replication Controllers](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/) -* [Debug Init Containers](/docs/tasks/debug-application-cluster/debug-init-containers/) -* [Debug Stateful Sets](/docs/tasks/debug-application-cluster/debug-stateful-set/) -* [Debug Applications](/docs/tasks/debug-application-cluster/debug-application/) -* [Using explorer to investigate your cluster](https://github.com/kubernetes/examples/blob/master/staging/explorer/README.md) - -{{% /capture %}} - - From 739a72185fbc31ea98896b01b2655bafd53fada4 Mon Sep 17 00:00:00 2001 From: Ali Yousefi Sabzevar Date: Sun, 3 Oct 2021 18:42:55 +0200 Subject: [PATCH 113/249] fix a grammar mistake - English version --- content/en/docs/concepts/services-networking/service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index e3f71c7d99..ff30adf088 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -1128,7 +1128,7 @@ spec: ``` {{< note >}} -ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName +ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using [headless Services](#headless-services). {{< /note >}} From 0b814897cb3ebb9cf6a46455f7a594cc24d0c8c9 Mon Sep 17 00:00:00 2001 From: Ali Yousefi Sabzevar Date: Sun, 3 Oct 2021 19:07:02 +0200 Subject: [PATCH 114/249] fix a grammar mistake - Chinese version --- content/zh/docs/concepts/services-networking/service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/concepts/services-networking/service.md b/content/zh/docs/concepts/services-networking/service.md index 05b50c1ee1..722d5e1c70 100644 --- a/content/zh/docs/concepts/services-networking/service.md +++ b/content/zh/docs/concepts/services-networking/service.md @@ -1819,7 +1819,7 @@ spec: ``` From d4a08df1b94b492e1709b7443fa0e6632fc46f0d Mon Sep 17 00:00:00 2001 From: Shubham Kuchhal Date: Mon, 4 Oct 2021 12:40:03 +0530 Subject: [PATCH 115/249] Improvement: Correct the "code" link in Dynamic Admission Control. --- .../access-authn-authz/extensible-admission-controllers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index aa714bdaa2..513c3d3352 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -79,7 +79,7 @@ The webhook server in the e2e test is deployed in the Kubernetes cluster, via the [deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps). The test also creates a [service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core) as the front-end of the webhook server. See -[code](https://github.com/kubernetes/kubernetes/blob/v1.15.0/test/e2e/apimachinery/webhook.go#L301). +[code](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/webhook.go#L748). You may also deploy your webhooks outside of the cluster. You will need to update your webhook configurations accordingly. From 0540e157a5ca271e107608d53a42afa87b3ffa8e Mon Sep 17 00:00:00 2001 From: Maciej Filocha Date: Mon, 4 Oct 2021 11:54:25 +0200 Subject: [PATCH 116/249] Synchronize Polish localization for ver 1.22, part 4 Synchronize Polish localization with upstream up to 08d92f9137924abdf442e7d9fb7372901379b993. Part 4 --- .../kubernetes-basics/create-cluster/cluster-interactive.html | 3 ++- .../kubernetes-basics/deploy-app/deploy-interactive.html | 4 +++- .../kubernetes-basics/explore/explore-interactive.html | 4 +++- .../kubernetes-basics/expose/expose-interactive.html | 4 +++- .../tutorials/kubernetes-basics/scale/scale-interactive.html | 4 +++- .../kubernetes-basics/update/update-interactive.html | 3 ++- 6 files changed, 16 insertions(+), 6 deletions(-) diff --git a/content/pl/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html index 72409e9238..43997fa589 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/create-cluster/cluster-interactive.html @@ -25,7 +25,8 @@ weight: 20
    - Przejdź do modułu 2› + Początek + Przejdź do modułu 2 >
    diff --git a/content/pl/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html index 954bad22b3..39403ee0e9 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/deploy-app/deploy-interactive.html @@ -38,7 +38,9 @@ weight: 20
    - Przejdź do modułu 3› + < Powrót do modułu 1 + Początek + Przejdź do modułu 3 >
    diff --git a/content/pl/docs/tutorials/kubernetes-basics/explore/explore-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/explore/explore-interactive.html index 14afae5a3d..35cfb12da8 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/explore/explore-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/explore/explore-interactive.html @@ -29,7 +29,9 @@ weight: 20
    - Przejdź do modułu 4› + < Powrót do modułu 2 + Początek + Przejdź do modułu 4 >
    diff --git a/content/pl/docs/tutorials/kubernetes-basics/expose/expose-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/expose/expose-interactive.html index 1aefa3e793..278fca5f89 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/expose/expose-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/expose/expose-interactive.html @@ -26,7 +26,9 @@ weight: 20
    - Przejdź do modułu 5› + < Powrót do modułu 3 + Początek + Przejdź do modułu 5 >
    diff --git a/content/pl/docs/tutorials/kubernetes-basics/scale/scale-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/scale/scale-interactive.html index 7990fbd1a6..028457a303 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/scale/scale-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/scale/scale-interactive.html @@ -26,7 +26,9 @@ weight: 20
    - Przejdź do modułu 6› + < Powrót do modułu 4 + Początek + Przejdź do modułu 6 >
    diff --git a/content/pl/docs/tutorials/kubernetes-basics/update/update-interactive.html b/content/pl/docs/tutorials/kubernetes-basics/update/update-interactive.html index 5664abc0e6..00e329ee0c 100644 --- a/content/pl/docs/tutorials/kubernetes-basics/update/update-interactive.html +++ b/content/pl/docs/tutorials/kubernetes-basics/update/update-interactive.html @@ -26,7 +26,8 @@ weight: 20
    - Z powrotem do Podstaw Kubernetes› + < Powrót do modułu 5 + Z powrotem do Podstaw Kubernetesa
    From 1262222578f30230c3ff76f87fcacbc1c94845cb Mon Sep 17 00:00:00 2001 From: Shubham Kuchhal Date: Mon, 4 Oct 2021 15:52:46 +0530 Subject: [PATCH 117/249] Change master to v1.22.0 --- .../access-authn-authz/extensible-admission-controllers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index 513c3d3352..30841587b3 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -79,7 +79,7 @@ The webhook server in the e2e test is deployed in the Kubernetes cluster, via the [deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps). The test also creates a [service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core) as the front-end of the webhook server. See -[code](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/webhook.go#L748). +[code](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/e2e/apimachinery/webhook.go#L748). You may also deploy your webhooks outside of the cluster. You will need to update your webhook configurations accordingly. From e24cf41794c41f41f55c2af6ae2255a67b313ea6 Mon Sep 17 00:00:00 2001 From: Mauren Berti Date: Mon, 4 Oct 2021 16:49:15 -0400 Subject: [PATCH 118/249] Changes from code review. --- content/pt-br/docs/concepts/configuration/configmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/pt-br/docs/concepts/configuration/configmap.md b/content/pt-br/docs/concepts/configuration/configmap.md index e8514ec15b..e3666a8542 100644 --- a/content/pt-br/docs/concepts/configuration/configmap.md +++ b/content/pt-br/docs/concepts/configuration/configmap.md @@ -151,7 +151,7 @@ spec: ConfigMaps não diferenciam entre propriedades com valores simples ou valores complexos, que ocupam várias linhas. O importante é a forma que Pods e outros objetos consomem tais valores. -Para este exemplo, definir um volume e montar ele dentro do container `demo` no caminho `/config` +Neste exemplo, definir um volume e montar ele dentro do contêiner `demo` no caminho `/config` cria dois arquivos: `/config/game.properties` e `/config/user-interface.properties`, embora existam quatro chaves distintas no ConfigMap. Isso se deve ao fato de que a definição do Pod contém uma lista `items` na seção `volumes`. From aa7a58485e887b088980f2ff488e14fad8c87329 Mon Sep 17 00:00:00 2001 From: Arhell Date: Tue, 5 Oct 2021 00:50:05 +0300 Subject: [PATCH 119/249] [pt-br] Add seccomp tutorial to index --- content/pt-br/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/pt-br/docs/tutorials/_index.md b/content/pt-br/docs/tutorials/_index.md index bc39fd817a..637ed4e6e8 100644 --- a/content/pt-br/docs/tutorials/_index.md +++ b/content/pt-br/docs/tutorials/_index.md @@ -47,6 +47,8 @@ Antes de iniciar um tutorial, é interessante que vocẽ salve a página de [Glo * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Serviços / "Services" * [Usando IP de origem](/docs/tutorials/services/source-ip/) From a72f7fac17f4df744234e882af02c573ef6fce91 Mon Sep 17 00:00:00 2001 From: Pushkar Joglekar <3390906+PushkarJ@users.noreply.github.com> Date: Mon, 4 Oct 2021 20:59:49 -0700 Subject: [PATCH 120/249] Blog post: A closer look at NSA / CISA Kubernetes Hardening Guidance (#29791) * Blog post: NSA / CISA Hardening This is a community response blog post that acts as complementary resource that takes a closer look at the guidance. This blog post is not a substitute for reading the guidance Apply suggestions from code review Co-authored-by: Jim Angel Co-authored-by: Savitha Raghunathan Co-authored-by: Tim Bannister Co-authored-by: Shannon Kularathna Co-authored-by: Robert Co-authored-by: Rey Lejano * Changes based on NSA/CISA initial feedback Co-authored-by: Jim Angel Co-authored-by: Savitha Raghunathan Co-authored-by: Tim Bannister Co-authored-by: Shannon Kularathna Co-authored-by: Robert Co-authored-by: Rey Lejano --- .../_posts/2021-10-05-nsa-cisa-hardening.md | 417 ++++++++++++++++++ 1 file changed, 417 insertions(+) create mode 100644 content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md diff --git a/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md new file mode 100644 index 0000000000..84665c5294 --- /dev/null +++ b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md @@ -0,0 +1,417 @@ +--- +layout: blog +title: A Closer Look at NSA/CISA Kubernetes Hardening Guidance +date: 2021-10-05 +slug: nsa-cisa-kubernetes-hardening-guidance +--- + +**Authors:** Jim Angel (Google), Pushkar Joglekar (VMware), and Savitha +Raghunathan (Red Hat) + +{{% alert title="Disclaimer" %}} +The open source tools listed in this article are to serve as examples only +and are in no way a direct recommendation from the Kubernetes community or authors. +{{% /alert %}} + +## Background + +USA's National Security Agency (NSA) and the Cybersecurity and Infrastructure +Security Agency (CISA) +released, "[Kubernetes Hardening Guidance](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF)" +on August 3rd, 2021. The guidance details threats to Kubernetes environments +and provides secure configuration guidance to minimize risk. + +The following sections of this blog correlate to the sections in the NSA/CISA guidance. +Any missing sections are skipped because of limited opportunities to add +anything new to the existing content. + +_Note_: This blog post is not a substitute for reading the guide. Reading the published +guidance is recommended before proceeding as the following content is +complementary. + +## Introduction and Threat Model + +Note that the threats identified as important by the NSA/CISA, or the intended audience of this guidance, may be different from the threats that other enterprise users of Kubernetes consider important. This section +is still useful for organizations that care about data, resource theft and +service unavailability. + +The guidance highlights the following three sources of compromises: + +- Supply chain risks +- Malicious threat actors +- Insider threats (administrators, users, or cloud service providers) + +The [threat model](https://en.wikipedia.org/wiki/Threat_model) tries to take a step back and review threats that not only +exist within the boundary of a Kubernetes cluster but also include the underlying +infrastructure and surrounding workloads that Kubernetes does not manage. + +For example, when a workload outside the cluster shares the same physical +network, it has access to the kubelet and to control plane components: etcd, controller manager, scheduler and API +server. Therefore, the guidance recommends having network level isolation +separating Kubernetes clusters from other workloads that do not need connectivity +to Kubernetes control plane nodes. Specifically, scheduler, controller-manager, +etcd only need to be accessible to the API server. Any interactions with Kubernetes +from outside the cluster can happen by providing access to API server port. + +List of ports and protocols for each of these components are +defined in [Ports and Protocols](/docs/reference/ports-and-protocols/) +within the Kubernetes documentation. + +> Special note: kube-scheduler and kube-controller-manager uses different ports than the ones mentioned in the guidance + +The [Threat modelling](https://cnsmap.netlify.app/threat-modelling) section +from the CNCF [Cloud Native Security Whitepaper + Map](https://github.com/cncf/tag-security/tree/main/security-whitepaper) +provides another perspective on approaching threat modelling Kubernetes, from a +cloud native lens. + +## Kubernetes Pod security + +Kubernetes by default does not guarantee strict workload isolation between pods +running in the same node in a cluster. However, the guidance provides several +techniques to enhance existing isolation and reduce the attack surface in case of a +compromise. + +### "Non-root" containers and "rootless" container engines + +Several best practices related to basic security principle of least privilege +i.e. provide only the permissions are needed; no more, no less, are worth a +second look. + +The guide recommends setting non-root user at build time instead of relying on +setting `runAsUser` at runtime in your Pod spec. This is a good practice and provides +some level of defense in depth. For example, if the container image is built with user `10001` +and the Pod spec misses adding the `runAsuser` field in its `Deployment` object. In this +case there are certain edge cases that are worth exploring for awareness: + +1. Pods can fail to start, if the user defined at build time is different from + the one defined in pod spec and some files are as a result inaccessible. +2. Pods can end up sharing User IDs unintentionally. This can be problematic + even if the User IDs are non-zero in a situation where a container escape to + host file system is possible. Once the attacker has access to the host file + system, they get access to all the file resources that are owned by other + unrelated pods that share the same UID. +3. Pods can end up sharing User IDs, with other node level processes not managed + by Kubernetes e.g. node level daemons for auditing, vulnerability scanning, + telemetry. The threat is similar to the one above where host file system + access can give attacker full access to these node level daemons without + needing to be root on the node. + +However, none of these cases will have as severe an impact as a container +running as root being able to escape as a root user on the host, which can provide +an attacker with complete control of the worker node, further allowing lateral +movement to other worker or control plane nodes. + +Kubernetes 1.22 introduced +an [alpha feature](/docs/tasks/administer-cluster/kubelet-in-userns/) +that specifically reduces the impact of such a control plane component running +as root user to a non-root user through user namespaces. + +That ([alpha stage](/docs/reference/command-line-tools-reference/feature-gates/#feature-stages)) support for user namespaces / rootless mode is available with +the following container runtimes: + +- [Docker Engine](https://docs.docker.com/engine/security/rootless/) +- [Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics) + +Some distributions support running in rootless mode, like the following: + +- [kind](https://kind.sigs.k8s.io/docs/user/rootless/) +- [k3s](https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental) +- [Usernetes](https://github.com/rootless-containers/usernetes) + +### Immutable container filesystems + +The NSA/CISA Kubernetes Hardening Guidance highlights an often overlooked feature `readOnlyRootFileSystem`, with a +working example in [Appendix B](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=42). This example limits execution and tampering of +containers at runtime. Any read/write activity can then be limited to few +directories by using `tmpfs` volume mounts. + +However, some applications that modify the container filesystem at runtime, like exploding a WAR or JAR file at container startup, +could face issues when enabling this feature. To avoid this issue, consider making minimal changes to the filesystem at runtime +when possible. + +### Building secure container images + +Kubernetes Hardening Guidance also recommends running a scanner at deploy time as an admission controller, +to prevent vulnerable or misconfigured pods from running in the cluster. +Theoretically, this sounds like a good approach but there are several caveats to +consider before this can be implemented in practice: + +- Depending on network bandwidth, available resources and scanner of choice, + scanning for vulnerabilities for an image can take an indeterminate amount of + time. This could lead to slower or unpredictable pod start up times, which + could result in spikes of unavailability when apps are serving peak load. +- If the policy that allows or denies pod startup is made using incorrect or + incomplete data it could result in several false positive or false negative + outcomes like the following: + - inside a container image, the `openssl` package is detected as vulnerable. However, + the application is written in Golang and uses the Go `crypto` package for TLS. Therefore, this vulnerability + is not in the code execution path and as such has minimal impact if it + remains unfixed. + - A vulnerability is detected in the `openssl` package for a Debian base image. + However, the upstream Debian community considers this as a Minor impact + vulnerability and as a result does not release a patch fix for this + vulnerability. The owner of this image is now stuck with a vulnerability that + cannot be fixed and a cluster that does not allow the image to run because + of predefined policy that does not take into account whether the fix for a + vulnerability is available or not + - A Golang app is built on top of a [distroless](https://github.com/GoogleContainerTools/distroless) + image, but it is compiled with a Golang version that uses a vulnerable [standard library](https://pkg.go.dev/std). + The scanner has + no visibility into golang version but only on OS level packages. So it + allows the pod to run in the cluster in spite of the image containing an + app binary built on vulnerable golang. + +To be clear, relying on vulnerability scanners is absolutely a good idea but +policy definitions should be flexible enough to allow: + +- Creation of exception lists for images or vulnerabilities through labelling +- Overriding the severity with a risk score based on impact of a vulnerability +- Applying the same policies at build time to catch vulnerable images with + fixable vulnerabilities before they can be deployed into Kubernetes clusters + +Special considerations like offline vulnerability database fetch, may also be +needed, if the clusters run in an air-gapped environment and the scanners +require internet access to update the vulnerability database. + +### Pod Security Policies + +Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) +API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/), +but some of the guidance in this section will still apply for the next few years, until cluster operators +upgrade their clusters to newer Kubernetes versions. + +The Kubernetes project is working on a replacement for PodSecurityPolicy. +Kubernetes v1.22 includes an alpha feature called called [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +that is intended to allow enforcing a minimum level of isolation between pods. + +The built-in isolation levels for Pod Security Admission are derived +from [Pod Security Standards](/docs/concepts/security/pod-security-standards/), which is a superset of all the components mentioned in Table I [page 10](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=17) of +the guidance. + +Information about migrating from PodSecurityPolicy to the Pod Security +Admission feature is available +in +[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/). + +One important behavior mentioned in the guidance that remains the same between +Pod Security Policy and its replacement is that enforcing either of them does +not affect pods that are already running. With both PodSecurityPolicy and Pod Security Admission, +the enforcement happens during the pod creation +stage. + +### Hardening container engines + +Some container workloads are less trusted than others but may need to run in the +same cluster. In those cases, running them on dedicated nodes that include +hardened container runtimes that provide stricter pod isolation boundaries can +act as a useful security control. + +Kubernetes supports +an API called [RuntimeClass](/docs/concepts/containers/runtime-class/) that is +stable / GA (and, therefore, enabled by default) stage as of Kubernetes v1.20. +RuntimeClass allows you to ensure that Pods requiring strong isolation are scheduled onto +nodes that can offer it. + +Some third-party projects that you can use in conjunction with RuntimeClass are: + +- [kata containers](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-runtime-class-for-kata-containers) +- [gvisor](https://gvisor.dev/docs/user_guide/containerd/quick_start/) + +As discussed here and in the guidance, many features and tooling exist in and around +Kubernetes that can enhance the isolation boundaries between +pods. Based on relevant threats and risk posture, you should pick and choose +between them, instead of trying to apply all the recommendations. Having said that, cluster +level isolation i.e. running workloads in dedicated clusters, remains the strictest workload +isolation mechanism, in spite of improvements mentioned earlier here and in the guide. + +## Network Separation and Hardening + +Kubernetes Networking can be tricky and this section focuses on how to secure +and harden the relevant configurations. The guide identifies the following as key +takeaways: +- Using NetworkPolicies to create isolation between resources, +- Securing the control plane +- Encrypting traffic and sensitive data + +### Network Policies + +Network policies can be created with the help of network plugins. In order to +make the creation and visualization easier for users, Cilium supports +a [web GUI tool](https://editor.cilium.io). That web GUI lets you create Kubernetes +NetworkPolicies (a generic API that nevertheless requires a compatible CNI plugin), +and / or Cilium network policies (CiliumClusterwideNetworkPolicy and CiliumNetworkPolicy, +which only work in clusters that use the Cilium CNI plugin). +You can use these APIs to restrict network traffic between pods, and therefore minimize the +attack vector. + +Another scenario that is worth exploring is the usage of external IPs. Some +services, when misconfigured, can create random external IPs. An attacker can take +advantage of this misconfiguration and easily intercept traffic. This vulnerability +has been reported +in [CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/). +Using [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) +can mitigate this vulnerability by preventing the services from using random +external IPs. [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) +only allows creation of services that don't require external IPs or whose +external IPs are within the range specified by the administrator. + +> CVE-2020-8554 - Kubernetes API server in all versions allow an attacker +> who is able to create a ClusterIP service and set the `spec.externalIPs` field, +> to intercept traffic to that IP address. Additionally, an attacker who is able to +> patch the `status` (which is considered a privileged operation and should not +> typically be granted to users) of a LoadBalancer service can set the +> `status.loadBalancer.ingress.ip` to similar effect. + +### Resource Policies + +In addition to configuring ResourceQuotas and limits, consider restricting how many process +IDs (PIDs) a given Pod can use, and also to reserve some PIDs for node-level use to avoid +resource exhaustion. More details to apply these limits can be +found in [Process ID Limits And Reservations](/docs/concepts/policy/pid-limiting/). + +### Control Plane Hardening + +In the next section, the guide covers control plane hardening. It is worth +noting that +from [Kubernetes 1.20](https://github.com/kubernetes/kubernetes/issues/91506), +insecure port from API server, has been removed. + +### Etcd + +As a general rule, the etcd server should be configured to only trust +certificates assigned to the API server. It limits the attack surface and prevents a +malicious attacker from gaining access to the cluster. It might be beneficial to +use a separate CA for etcd, as it by default trusts all the certificates issued +by the root CA. + +### Kubeconfig Files + +In addition to specifying the token and certificates directly, `.kubeconfig` +supports dynamic retrieval of temporary tokens using auth provider plugins. +Beware of the possibility of malicious +shell [code execution](https://banzaicloud.com/blog/kubeconfig-security/) in a +`kubeconfig` file. Once attackers gain access to the cluster, they can steal ssh +keys/secrets or more. + +### Secrets +Kubernetes [Secrets](/docs/concepts/configuration/secret/) is the native way of managing secrets as a Kubernetes +API object. However, in some scenarios such as a desire to have a single source of truth for all app secrets, irrespective of whether they run on Kubernetes or not, secrets can be managed loosely coupled with +Kubernetes and consumed by pods through side-cars or init-containers with minimal usage of Kubernetes Secrets API. + +[External secrets providers](https://github.com/external-secrets/kubernetes-external-secrets) +and [csi-secrets-store](https://github.com/kubernetes-sigs/secrets-store-csi-driver) +are some of these alternatives to Kubernetes Secrets + +## Log Auditing + +The NSA/CISA guidance stresses monitoring and alerting based on logs. The key points +include logging at the host level, application level, and on the cloud. When +running Kubernetes in production, it's important to understand who's +responsible, and who's accountable, for each layer of logging. + +### Kubernetes API auditing + +One area that deserves more focus is what exactly should alert or be logged. The +document outlines a sample policy in [Appendix L: Audit Policy](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=55) that logs all +RequestResponse's including metadata and request / response bodies. While helpful for a demo, it may not be practical for production. + +Each organization needs to evaluate their +own threat model and build an audit policy that complements or helps troubleshooting incident response. Think +about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy). +It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. + +Audit logging configurations can also be tested with +kind following these [instructions](https://kind.sigs.k8s.io/docs/user/auditing). + +### Streaming logs and auditing + +Logging is important for threat and anomaly detection. As the document outlines, +it's a best practice to scan and alert on logs as close to real time as possible +and to protect logs from tampering if a compromise occurs. It's important to +reflect on the various levels of logging and identify the critical areas such as +API endpoints. + +Kubernetes API audit logging can stream to a webhook and there's an example in [Appendix N: Webhook configuration](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=58). Using a webhook could be a method that +stores logs off cluster and/or centralizes all audit logs. Once logs are +centrally managed, look to enable alerting based on critical events. Also ensure +you understand what the baseline is for normal activities. + +### Alert identification + +While the guide stressed the importance of notifications, there is not a blanket +event list to alert from. The alerting requirements vary based on your own +requirements and threat model. Examples include the following events: + +- Changes to the `securityContext` of a Pod +- Updates to admission controller configs +- Accessing certain files / URLs + +### Additional logging resources + +- [Seccomp Security Profiles and You: A Practical Guide - Duffie Cooley](https://www.youtube.com/watch?v=OPuu8wsu2Zc) +- [TGI Kubernetes 119: Gatekeeper and OPA](https://www.youtube.com/watch?v=ZJgaGJm9NJE) +- [Abusing The Lack of Kubernetes Auditing Policies](https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies/) +- [Enable seccomp for all workloads with a new v1.22 alpha feature](https://kubernetes.io/blog/2021/08/25/seccomp-default/) +- [This Week in Cloud Native: Auditing / Pod Security](https://www.twitch.tv/videos/1147889860) + +## Upgrading and Application Security practices + +Kubernetes releases three times per year, so upgrade-related toil is a common problem for +people running production clusters. In addition to this, operators must +regularly upgrade the underlying node's operating system and running +applications. This is a best practice to ensure continued support and to reduce +the likelihood of bugs or vulnerabilities. + +Kubernetes supports the three most recent stable releases. While each Kubernetes +release goes through a large number of tests before being published, some +teams aren't comfortable running the latest stable release until some time has +passed. No matter what version you're running, ensure that patch upgrades +happen frequently or automatically. More information can be found in +the [version skew](/releases/version-skew-policy/) policy +pages. + +When thinking about how you'll manage node OS upgrades, consider ephemeral +nodes. Having the ability to destroy and add nodes allows your team to respond +quicker to node issues. In addition, having deployments that tolerate node +instability (and a culture that encourages frequent deployments) allows for +easier cluster upgrades. + +Additionally, it's worth reiterating from the guidance that periodic +vulnerability scans and penetration tests can be performed on the various system +components to proactively look for insecure configurations and vulnerabilities. + +### Finding release & security information + +To find the most recent Kubernetes supported versions, refer to +[https://k8s.io/releases](https://k8s.io/releases), which includes minor versions. It's good to stay up to date with +your minor version patches. + +If you're running a managed Kubernetes offering, look for their release +documentation and find their various security channels. + +Subscribe to +the [Kubernetes Announce mailing list](https://groups.google.com/g/kubernetes-announce). +The Kubernetes Announce mailing list is searchable for terms such +as "[Security Advisories](https://groups.google.com/g/kubernetes-announce/search?q=%5BSecurity%20Advisory%5D)". +You can set up alerts and email notifications as long as you know what key +words to alert on. + +## Conclusion + +In summary, it is fantastic to see security practitioners sharing this +level of detailed guidance in public. This guidance further highlights +Kubernetes going mainstream and how securing Kubernetes clusters and the +application containers running on Kubernetes continues to need attention and focus of +practitioners. Only a few weeks after the guidance was published, an open source +tool [kubescape](https://github.com/armosec/kubescape) to validate cluster +against this guidance became available. + +This tool can be a great starting point to check the current state of your +clusters, after which you can use the information in this blog post and in the guidance to assess +where improvements can be made. + +Finally, it is worth reiterating that not all controls in this guidance will +make sense for all practitioners. The best way to know which controls matter is +to rely on the threat model of your own Kubernetes environment. + +_A special shout out and thanks to Rory McCune (@raesene) for his inputs to this blog post_ From 5c73239f39d9c9faa57d19c02e02f2a3afc2ee9a Mon Sep 17 00:00:00 2001 From: bang9211 Date: Fri, 10 Sep 2021 10:27:34 +0900 Subject: [PATCH 121/249] Translate tasks/inject-data-application/downward-api-volume-expose-pod-information in Korean --- ...nward-api-volume-expose-pod-information.md | 259 ++++++++++++++++++ .../pods/inject/dapi-volume-resources.yaml | 57 ++++ .../ko/examples/pods/inject/dapi-volume.yaml | 38 +++ 3 files changed, 354 insertions(+) create mode 100644 content/ko/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.md create mode 100644 content/ko/examples/pods/inject/dapi-volume-resources.yaml create mode 100644 content/ko/examples/pods/inject/dapi-volume.yaml diff --git a/content/ko/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.md b/content/ko/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.md new file mode 100644 index 0000000000..2a8a206d6d --- /dev/null +++ b/content/ko/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.md @@ -0,0 +1,259 @@ +--- +title: 파일로 컨테이너에 파드 정보 노출하기 +content_type: task +weight: 40 +--- + + + +본 페이지는 파드가 DownwardAPIVolumeFile을 사용하여 파드에서 실행되는 컨테이너에 +자신에 대한 정보를 노출하는 방법에 대해 설명한다. DownwardAPIVolumeFile은 파드 필드와 +컨테이너 필드를 노출할 수 있다. + + + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + + +## 다운워드(Downward) API + +실행 중인 컨테이너에 파드 및 컨테이너 필드를 노출하는 방법에는 두 가지가 있다. + +* [환경 변수](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) +* 볼륨 파일 + +파드 및 컨테이너 필드를 노출하는 이 두 가지 방법을 *다운워드 API*라고 한다. + +## 파드 필드 저장 + +이 연습에서는 하나의 컨테이너를 가진 파드를 생성한다. +다음은 파드에 대한 구성 파일이다. + +{{< codenew file="pods/inject/dapi-volume.yaml" >}} + +구성 파일에서 파드에 `downwardAPI` 볼륨이 있고 컨테이너가 `/etc/podinfo`에 볼륨을 마운트하는 +것을 확인할 수 있다. + +`downwardAPI` 아래의 배열을 살펴보자. 배열의 각 요소는 +[DownwardAPIVolumeFile](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#downwardapivolumefile-v1-core)이다. +첫 번째 요소는 파드의 `metadata.labels` 필드 값이 `labels`라는 파일에 저장되어야 함을 지정한다. +두 번째 요소는 파드의 `annotations` 필드 값이 `annotations`라는 파일에 저장되어야 함을 지정한다. + +{{< note >}} +이 예제의 필드는 파드에 있는 컨테이너의 필드가 아니라 파드 필드이다. +{{< /note >}} + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-volume.yaml +``` + +파드의 컨테이너가 실행 중인지 확인한다. + +```shell +kubectl get pods +``` + +컨테이너의 로그를 본다. + +```shell +kubectl logs kubernetes-downwardapi-volume-example +``` + +출력은 `labels` 파일과 `annotations` 파일의 내용을 보여준다. + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" + +build="two" +builder="john-doe" +``` + +파드에서 실행 중인 컨테이너의 셸을 가져오자. + +```shell +kubectl exec -it kubernetes-downwardapi-volume-example -- sh +``` + +셸에서 `labels` 파일을 보자. + +```shell +/# cat /etc/podinfo/labels +``` + +출력을 통해 모든 파드의 레이블이 `labels` 파일에 기록되었음을 확인할 수 있다. + +```shell +cluster="test-cluster1" +rack="rack-22" +zone="us-est-coast" +``` + +마찬가지로 `annotations` 파일을 확인하자. + +```shell +/# cat /etc/podinfo/annotations +``` + +`etc/podinfo` 디렉터리에 파일을 확인하자. + +```shell +/# ls -laR /etc/podinfo +``` + +출력에서 `labels` 및 `annotations` 파일이 +임시 하위 디렉터리에 있음을 알 수 있다. 이 예제에서는 +`..2982_06_02_21_47_53.299460680`이다. `/etc/podinfo` 디렉터리에서 `..data`는 +임시 하위 디렉토리에 대한 심볼릭 링크이다. `/etc/podinfo` 디렉토리에서 +`labels`와 `annotations` 또한 심볼릭 링크이다. + +``` +drwxr-xr-x ... Feb 6 21:47 ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 ..data -> ..2982_06_02_21_47_53.299460680 +lrwxrwxrwx ... Feb 6 21:47 annotations -> ..data/annotations +lrwxrwxrwx ... Feb 6 21:47 labels -> ..data/labels + +/etc/..2982_06_02_21_47_53.299460680: +total 8 +-rw-r--r-- ... Feb 6 21:47 annotations +-rw-r--r-- ... Feb 6 21:47 labels +``` + +심볼릭 링크를 사용하면 메타데이터의 동적(dynamic) 원자적(atomic) 갱신이 가능하다. +업데이트는 새 임시 디렉터리에 기록되고, `..data` 심볼릭 링크는 +[rename(2)](http://man7.org/linux/man-pages/man2/rename.2.html)을 사용하여 +원자적(atomic)으로 갱신한다. + +{{< note >}} +다운워드 API를 [subPath](/docs/concepts/storage/volumes/#using-subpath) +볼륨 마운트로 사용하는 컨테이너는 다운워드 API 업데이트를 수신하지 않는다. +{{< /note >}} + +셸을 종료한다. + +```shell +/# exit +``` + +## 컨테이너 필드 저장 + +이전 연습에서는 파드 필드를 DownwardAPIVolumeFile에 저장하였다. +이 다음 연습에서는 컨테이너 필드를 저장한다. 다음은 하나의 컨테이너를 가진 파드의 구성 파일이다. + +{{< codenew file="pods/inject/dapi-volume-resources.yaml" >}} + +구성 파일에서 파드에 `downwardAPI` 볼륨이 있고 컨테이너는 `/etc/podinfo`에 볼륨을 +마운트하는 것을 확인할 수 있다. + +`downwardAPI` 아래의 `items` 배열을 살펴보자. 배열의 각 요소는 DownwardAPIVolumeFile이다. + +첫 번째 요소는 `client-container`라는 컨테이너에서 +`1m`으로 지정된 형식의 `limits.cpu` 필드 값이 +`cpu_limit`이라는 파일에 저장되어야 함을 지정한다. `divisor` 필드는 선택 사항이며 +기본값인 `1`은 CPU에 대한 코어 및 메모리에 대한 바이트를 의미한다. + +파드를 생성한다. + +```shell +kubectl apply -f https://k8s.io/examples/pods/inject/dapi-volume-resources.yaml +``` + +파드에서 실행 중인 컨테이너의 셸을 가져온다. + +```shell +kubectl exec -it kubernetes-downwardapi-volume-example-2 -- sh +``` + +셸에서 `cpu_limit` 파일을 확인한다. + +```shell +/# cat /etc/podinfo/cpu_limit +``` +비슷한 명령을 통해 `cpu_request`, `mem_limit` 및 +`mem_request` 파일을 확인할 수 있다. + + + + + +## 다운워드 API의 기능 + +다음 정보는 환경 변수 및 `downwardAPI` 볼륨을 통해 컨테이너에서 사용할 수 있다. + +* `fieldRef`를 통해 다음 정보를 사용할 수 있다. + * `metadata.name` - 파드의 이름 + * `metadata.namespace` - 파드의 네임스페이스(Namespace) + * `metadata.uid` - 파드의 UID + * `metadata.labels['']` - 파드의 레이블 `` 값 (예를 들어, `metadata.labels['mylabel']`) + * `metadata.annotations['']` - 파드의 어노테이션 `` 값 (예를 들어, `metadata.annotations['myannotation']`) +* `resourceFieldRef`를 통해 다음 정보를 사용할 수 있다. + * 컨테이너의 CPU 한도(limit) + * 컨테이너의 CPU 요청(request) + * 컨테이너의 메모리 한도(limit) + * 컨테이너의 메모리 요청(request) + * 컨테이너의 hugepages 한도(limit) (`DownwardAPIHugePages` [기능 게이트(feature gate)](/docs/reference/command-line-tools-reference/feature-gates/)가 활성화된 경우) + * 컨테이너의 hugepages 요청(request) (`DownwardAPIHugePages` [기능 게이트(feature gate)](/docs/reference/command-line-tools-reference/feature-gates/)가 활성화된 경우) + * 컨테이너의 임시-스토리지 한도(limit) + * 컨테이너의 임시-스토리지 요청(request) + +`downwardAPI` 볼륨 `fieldRef`를 통해 다음 정보를 사용할 수 있다. + +* `metadata.labels` - 한 줄에 하나의 레이블이 있는 +`label-key="escaped-label-value"` 형식의 모든 파드 레이블 +* `metadata.annotations` - 한 줄에 하나의 어노테이션이 있는 `annotation-key="escaped-annotation-value"` 형식의 모든 파드 어노테이션 + +환경 변수를 통해 다음 정보를 사용할 수 있다. + +* `status.podIP` - 파드의 IP 주소 +* `spec.serviceAccountName` - 파드의 서비스 계정 이름, v1.4.0-alpha.3부터 사용 가능 +* `spec.nodeName` - 노드의 이름, v1.4.0-alpha.3부터 사용 가능 +* `status.hostIP` - 노드의 IP, v1.7.0-alpha.1 이후 사용 가능 + +{{< note >}} +컨테이너에 대해 CPU 및 메모리 한도(limit)가 지정되지 않은 경우 다운워드 API는 기본적으로 +CPU 및 메모리에 대해 할당 가능한 노드 값으로 설정한다. +{{< /note >}} + +## 특정 경로 및 파일 권한에 대한 프로젝트 키 + +키(key)를 파드 안의 특정 경로에, 특정 권한으로, 파일 단위로 투영(project)할 수 있다. +자세한 내용은 +[시크릿(Secrets)](/ko/docs/concepts/configuration/secret/)을 참조한다. + +## 다운워드 API에 대한 동기 + +컨테이너가 쿠버네티스에 과도하게 결합되지 않고 자체에 대한 정보를 갖는 것이 때때로 유용하다. +다운워드 API를 사용하면 컨테이너가 쿠버네티스 클라이언트 또는 API 서버를 사용하지 않고 +자체 또는 클러스터에 대한 정보를 사용할 수 있다. + +예를 들어 잘 알려진 특정 환경 변수에 고유 식별자가 있다고 가정하는 +기존 애플리케이션이 있다. 한 가지 가능성은 애플리케이션을 래핑하는 것이지만 +이는 지루하고 오류가 발생하기 쉬우며 낮은 결합 목표를 위반한다. +더 나은 옵션은 파드의 이름을 식별자로 사용하고 +파드의 이름을 잘 알려진 환경 변수에 삽입하는 것이다. + + + + +## {{% heading "whatsnext" %}} + + +* [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core) +* [볼륨](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core) +* [DownwardAPIVolumeSource](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#downwardapivolumesource-v1-core) +* [DownwardAPIVolumeFile](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#downwardapivolumefile-v1-core) +* [ResourceFieldSelector](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#resourcefieldselector-v1-core) + + + + + diff --git a/content/ko/examples/pods/inject/dapi-volume-resources.yaml b/content/ko/examples/pods/inject/dapi-volume-resources.yaml new file mode 100644 index 0000000000..ecb231e0cf --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-volume-resources.yaml @@ -0,0 +1,57 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example-2 +spec: + containers: + - name: client-container + image: k8s.gcr.io/busybox:1.24 + command: ["sh", "-c"] + args: + - while true; do + echo -en '\n'; + if [[ -e /etc/podinfo/cpu_limit ]]; then + echo -en '\n'; cat /etc/podinfo/cpu_limit; fi; + if [[ -e /etc/podinfo/cpu_request ]]; then + echo -en '\n'; cat /etc/podinfo/cpu_request; fi; + if [[ -e /etc/podinfo/mem_limit ]]; then + echo -en '\n'; cat /etc/podinfo/mem_limit; fi; + if [[ -e /etc/podinfo/mem_request ]]; then + echo -en '\n'; cat /etc/podinfo/mem_request; fi; + sleep 5; + done; + resources: + requests: + memory: "32Mi" + cpu: "125m" + limits: + memory: "64Mi" + cpu: "250m" + volumeMounts: + - name: podinfo + mountPath: /etc/podinfo + volumes: + - name: podinfo + downwardAPI: + items: + - path: "cpu_limit" + resourceFieldRef: + containerName: client-container + resource: limits.cpu + divisor: 1m + - path: "cpu_request" + resourceFieldRef: + containerName: client-container + resource: requests.cpu + divisor: 1m + - path: "mem_limit" + resourceFieldRef: + containerName: client-container + resource: limits.memory + divisor: 1Mi + - path: "mem_request" + resourceFieldRef: + containerName: client-container + resource: requests.memory + divisor: 1Mi + diff --git a/content/ko/examples/pods/inject/dapi-volume.yaml b/content/ko/examples/pods/inject/dapi-volume.yaml new file mode 100644 index 0000000000..0f0a9f2e5c --- /dev/null +++ b/content/ko/examples/pods/inject/dapi-volume.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kubernetes-downwardapi-volume-example + labels: + zone: us-est-coast + cluster: test-cluster1 + rack: rack-22 + annotations: + build: two + builder: john-doe +spec: + containers: + - name: client-container + image: k8s.gcr.io/busybox + command: ["sh", "-c"] + args: + - while true; do + if [[ -e /etc/podinfo/labels ]]; then + echo -en '\n\n'; cat /etc/podinfo/labels; fi; + if [[ -e /etc/podinfo/annotations ]]; then + echo -en '\n\n'; cat /etc/podinfo/annotations; fi; + sleep 5; + done; + volumeMounts: + - name: podinfo + mountPath: /etc/podinfo + volumes: + - name: podinfo + downwardAPI: + items: + - path: "labels" + fieldRef: + fieldPath: metadata.labels + - path: "annotations" + fieldRef: + fieldPath: metadata.annotations + From 3049f75a0127330084b29b512458607dc610d694 Mon Sep 17 00:00:00 2001 From: Christian Kotzbauer Date: Tue, 5 Oct 2021 09:00:53 +0200 Subject: [PATCH 122/249] fix: removed duplicate "called" --- content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md index 84665c5294..1af75e9ceb 100644 --- a/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md +++ b/content/en/blog/_posts/2021-10-05-nsa-cisa-hardening.md @@ -181,7 +181,7 @@ but some of the guidance in this section will still apply for the next few years upgrade their clusters to newer Kubernetes versions. The Kubernetes project is working on a replacement for PodSecurityPolicy. -Kubernetes v1.22 includes an alpha feature called called [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +Kubernetes v1.22 includes an alpha feature called [Pod Security Admission](/docs/concepts/security/pod-security-admission/) that is intended to allow enforcing a minimum level of isolation between pods. The built-in isolation levels for Pod Security Admission are derived From 5894fa3942d67b057edf21242a04b889b0e35a3c Mon Sep 17 00:00:00 2001 From: Jihoon Seo Date: Tue, 28 Sep 2021 17:27:27 +0900 Subject: [PATCH 123/249] [ko] Update outdated files in dev-1.22-ko.1 (p7) --- .../docs/concepts/storage/storage-classes.md | 31 +++++++- content/ko/docs/concepts/storage/volumes.md | 61 ++++------------ .../workloads/controllers/cron-jobs.md | 7 +- .../workloads/controllers/daemonset.md | 6 +- .../concepts/workloads/controllers/job.md | 72 ++++++++++++++----- .../workloads/controllers/replicaset.md | 8 +-- 6 files changed, 110 insertions(+), 75 deletions(-) diff --git a/content/ko/docs/concepts/storage/storage-classes.md b/content/ko/docs/concepts/storage/storage-classes.md index f4385419f1..c915b65fad 100644 --- a/content/ko/docs/concepts/storage/storage-classes.md +++ b/content/ko/docs/concepts/storage/storage-classes.md @@ -76,7 +76,7 @@ volumeBindingMode: Immediate | Glusterfs | ✓ | [Glusterfs](#glusterfs) | | iSCSI | - | - | | Quobyte | ✓ | [Quobyte](#quobyte) | -| NFS | - | - | +| NFS | - | [NFS](#nfs) | | RBD | ✓ | [Ceph RBD](#ceph-rbd) | | VsphereVolume | ✓ | [vSphere](#vsphere) | | PortworxVolume | ✓ | [Portworx 볼륨](#portworx-볼륨) | @@ -423,6 +423,29 @@ parameters: 헤드리스 서비스를 자동으로 생성한다. 퍼시스턴트 볼륨 클레임을 삭제하면 동적 엔드포인트와 서비스가 자동으로 삭제된다. +### NFS + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: example-nfs +provisioner: example.com/external-nfs +parameters: + server: nfs-server.example.com + path: /share + readOnly: false +``` + +* `server`: NFS 서버의 호스트네임 또는 IP 주소. +* `path`: NFS 서버가 익스포트(export)한 경로. +* `readOnly`: 스토리지를 읽기 전용으로 마운트할지 나타내는 플래그(기본값: false). + +쿠버네티스에는 내장 NFS 프로비저너가 없다. NFS를 위한 스토리지클래스를 생성하려면 외부 프로비저너를 사용해야 한다. +예시는 다음과 같다. +* [NFS Ganesha server and external provisioner](https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner) +* [NFS subdir external provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) + ### OpenStack Cinder ```yaml @@ -578,6 +601,12 @@ parameters: ### Quobyte +{{< feature-state for_k8s_version="v1.22" state="deprecated" >}} + +Quobyte 인-트리 스토리지 플러그인은 사용 중단되었으며, +아웃-오브-트리 Quobyte 플러그인에 대한 [예제](https://github.com/quobyte/quobyte-csi/blob/master/example/StorageClass.yaml) +`StorageClass`는 Quobyte CSI 저장소에서 찾을 수 있다. + ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass diff --git a/content/ko/docs/concepts/storage/volumes.md b/content/ko/docs/concepts/storage/volumes.md index 050566388e..349f47a55c 100644 --- a/content/ko/docs/concepts/storage/volumes.md +++ b/content/ko/docs/concepts/storage/volumes.md @@ -124,7 +124,7 @@ EBS 볼륨이 파티션된 경우, 선택적 필드인 `partition: "}} 컨트롤러 관리자와 kubelet에 의해 로드되지 않도록 `awsElasticBlockStore` 스토리지 -플러그인을 끄려면, `CSIMigrationAWSComplete` 플래그를 `true` 로 설정한다. 이 기능은 모든 워커 노드에서 `ebs.csi.aws.com` 컨테이너 스토리지 인터페이스(CSI) 드라이버 설치를 필요로 한다. +플러그인을 끄려면, `InTreePluginAWSUnregister` 플래그를 `true` 로 설정한다. ### azureDisk {#azuredisk} @@ -462,7 +462,8 @@ spec: required: nodeSelectorTerms: - matchExpressions: - - key: failure-domain.beta.kubernetes.io/zone + # 1.21 이전 버전에서는 failure-domain.beta.kubernetes.io/zone 키를 사용해야 한다. + - key: topology.kubernetes.io/zone operator: In values: - us-central1-a @@ -480,6 +481,13 @@ GCE PD의 `CSIMigration` 기능이 활성화된 경우 기존 인-트리 플러 를 설치하고 `CSIMigration` 과 `CSIMigrationGCE` 베타 기능을 활성화해야 한다. +#### GCE CSI 마이그레이션 완료 + +{{< feature-state for_k8s_version="v1.21" state="alpha" >}} + +컨트롤러 매니저와 kubelet이 `gcePersistentDisk` 스토리지 플러그인을 로드하는 것을 방지하려면, +`InTreePluginGCEUnregister` 플래그를 `true`로 설정한다. + ### gitRepo (사용 중단됨) {#gitrepo} {{< warning >}} @@ -931,7 +939,7 @@ projected 볼륨 소스를 [`subPath`](#subpath-사용하기) 볼륨으로 마 해당 볼륨 소스의 업데이트를 수신하지 않는다. {{< /note >}} -### quobyte +### quobyte (사용 중단됨) {#quobyte} `quobyte` 볼륨을 사용하면 기존 [Quobyte](https://www.quobyte.com) 볼륨을 파드에 마운트할 수 있다. @@ -967,49 +975,6 @@ RBD는 읽기-쓰기 모드에서 단일 고객만 마운트할 수 있다. 더 자세한 내용은 [RBD 예시](https://github.com/kubernetes/examples/tree/master/volumes/rbd)를 참고한다. -### scaleIO (사용 중단됨) {#scaleio} - -ScaleIO는 기존 하드웨어를 사용해서 확장 가능한 공유 블럭 네트워크 스토리지 클러스터를 -생성하는 소프트웨어 기반 스토리지 플랫폼이다. `scaleIO` 볼륨 -플러그인을 사용하면 배포된 파드가 기존 ScaleIO에 접근할 수 -있다. 퍼시스턴트 볼륨 클레임을 위해 새로운 볼륨을 동적으로 프로비저닝하는 -방법에 대한 자세한 내용은 -[ScaleIO 퍼시스턴트 볼륨](/ko/docs/concepts/storage/persistent-volumes/#scaleio)을 참고한다. - -{{< note >}} -사용하기 위해선 먼저 기존에 ScaleIO 클러스터를 먼저 설정하고 -생성한 볼륨과 함께 실행해야 한다. -{{< /note >}} - -다음의 예시는 ScaleIO를 사용하는 파드 구성이다. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: pod-0 -spec: - containers: - - image: k8s.gcr.io/test-webserver - name: pod-0 - volumeMounts: - - mountPath: /test-pd - name: vol-0 - volumes: - - name: vol-0 - scaleIO: - gateway: https://localhost:443/api - system: scaleio - protectionDomain: sd0 - storagePool: sp1 - volumeName: vol-0 - secretRef: - name: sio-secret - fsType: xfs -``` - -더 자세한 내용은 [ScaleIO](https://github.com/kubernetes/examples/tree/master/staging/volumes/scaleio) 예제를 참고한다. - ### secret `secret` 볼륨은 암호와 같은 민감한 정보를 파드에 전달하는데 @@ -1029,7 +994,7 @@ tmpfs(RAM 기반 파일시스템)로 지원되기 때문에 비 휘발성 스토 더 자세한 내용은 [시크릿 구성하기](/ko/docs/concepts/configuration/secret/)를 참고한다. -### storageOS {#storageos} +### storageOS (사용 중단됨) {#storageos} `storageos` 볼륨을 사용하면 기존 [StorageOS](https://www.storageos.com) 볼륨을 파드에 마운트할 수 있다. @@ -1177,7 +1142,7 @@ vSphere CSI 드라이버에서 생성된 새 볼륨은 이러한 파라미터를 {{< feature-state for_k8s_version="v1.19" state="beta" >}} -`vsphereVolume` 플러그인이 컨트롤러 관리자와 kubelet에 의해 로드되지 않도록 기능을 비활성화하려면, 이 기능 플래그를 `true` 로 설정해야 한다. 이를 위해서는 모든 워커 노드에 `csi.vsphere.vmware.com` {{< glossary_tooltip text="CSI" term_id="csi" >}} 드라이버가 설치해야 한다. +`vsphereVolume` 플러그인이 컨트롤러 관리자와 kubelet에 의해 로드되지 않도록 기능을 비활성화하려면, `InTreePluginvSphereUnregister` 기능 플래그를 `true` 로 설정해야 한다. 이를 위해서는 모든 워커 노드에 `csi.vsphere.vmware.com` {{< glossary_tooltip text="CSI" term_id="csi" >}} 드라이버를 설치해야 한다. ## subPath 사용하기 {#using-subpath} diff --git a/content/ko/docs/concepts/workloads/controllers/cron-jobs.md b/content/ko/docs/concepts/workloads/controllers/cron-jobs.md index 6935cf8fb4..740929b05c 100644 --- a/content/ko/docs/concepts/workloads/controllers/cron-jobs.md +++ b/content/ko/docs/concepts/workloads/controllers/cron-jobs.md @@ -36,9 +36,10 @@ kube-controller-manager 컨테이너에 설정된 시간대는 ## 크론잡 -크론잡은 백업 실행 또는 이메일 전송과 같은 정기적이고 반복적인 -작업을 만드는데 유용하다. 또한 크론잡은 클러스터가 유휴 상태일 때 잡을 -스케줄링하는 것과 같이 특정 시간 동안의 개별 작업을 스케줄할 수 있다. +크론잡은 백업, 리포트 생성 등의 정기적 작업을 수행하기 위해 사용된다. +각 작업은 무기한 반복되도록 구성해야 한다(예: +1일/1주/1달마다 1회). +작업을 시작해야 하는 해당 간격 내 특정 시점을 정의할 수 있다. ### 예시 diff --git a/content/ko/docs/concepts/workloads/controllers/daemonset.md b/content/ko/docs/concepts/workloads/controllers/daemonset.md index 1496b25ec3..dc8f543048 100644 --- a/content/ko/docs/concepts/workloads/controllers/daemonset.md +++ b/content/ko/docs/concepts/workloads/controllers/daemonset.md @@ -229,5 +229,7 @@ Kubelet이 감시하는 특정 디렉터리에 파일을 작성하는 파드를 파드가 실행되는 호스트를 정확하게 제어하는 것보다 레플리카의 수를 스케일링 업 및 다운 하고, 업데이트 롤아웃이 더 중요한 프런트 엔드와 같은 것은 스테이트리스 서비스의 -디플로이먼트를 사용한다. 파드 사본이 항상 모든 호스트 또는 특정 호스트에서 실행되는 것이 중요하고, -다른 파드의 실행 이전에 필요한 경우에는 데몬셋을 사용한다. +디플로이먼트를 사용한다. 데몬셋이 특정 노드에서 다른 파드가 올바르게 실행되도록 하는 노드 수준 기능을 제공한다면, +파드 사본이 항상 모든 호스트 또는 특정 호스트에서 실행되는 것이 중요한 경우에 데몬셋을 사용한다. + +예를 들어, [네트워크 플러그인](/ko/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)은 데몬셋으로 실행되는 컴포넌트를 포함할 수 있다. 데몬셋 컴포넌트는 작동 중인 노드가 정상적인 클러스터 네트워킹을 할 수 있도록 한다. diff --git a/content/ko/docs/concepts/workloads/controllers/job.md b/content/ko/docs/concepts/workloads/controllers/job.md index 4972df45cd..21da00ff5a 100644 --- a/content/ko/docs/concepts/workloads/controllers/job.md +++ b/content/ko/docs/concepts/workloads/controllers/job.md @@ -187,14 +187,7 @@ _작업 큐_ 잡은 `.spec.completions` 를 설정하지 않은 상태로 두고 ### 완료 모드 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} - -{{< note >}} -인덱싱된 잡을 생성하려면, [API 서버](/docs/reference/command-line-tools-reference/kube-apiserver/) -및 [컨트롤러 관리자](/docs/reference/command-line-tools-reference/kube-controller-manager/)에서 -`IndexedJob` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 -활성화해야 한다. -{{< /note >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} 완료 횟수가 _고정적인 완료 횟수_ 즉, null이 아닌 `.spec.completions` 가 있는 잡은 `.spec.completionMode` 에 지정된 완료 모드를 가질 수 있다. @@ -203,8 +196,14 @@ _작업 큐_ 잡은 `.spec.completions` 를 설정하지 않은 상태로 두고 완료된 파드가 있는 경우 작업이 완료된 것으로 간주된다. 즉, 각 파드 완료는 서로 상동하다(homologous). null `.spec.completions` 가 있는 잡은 암시적으로 `NonIndexed` 이다. -- `Indexed`: 잡의 파드는 `batch.kubernetes.io/job-completion-index` - 어노테이션에서 사용할 수 있는 0에서 `.spec.completions-1` 까지 연결된 완료 인덱스를 가져온다. +- `Indexed`: 잡의 파드는 연결된 완료 인덱스를 0에서 `.spec.completions-1` 까지 + 가져온다. 이 인덱스는 다음의 세 가지 메카니즘으로 얻을 수 있다. + - 파드 어노테이션 `batch.kubernetes.io/job-completion-index`. + - 파드 호스트네임 중 일부(`$(job-name)-$(index)` 형태). 인덱스된(Indexed) 잡과 + {{< glossary_tooltip text="서비스" term_id="Service" >}}를 결합하여 사용하고 + 있다면, 잡에 속한 파드는 DNS를 이용하여 서로를 디스커버 하기 위해 사전에 결정된 + 호스트네임을 사용할 수 있다. + - 컨테이너화된 태스크의 경우, `JOB_COMPLETION_INDEX` 환경 변수. 각 인덱스에 대해 성공적으로 완료된 파드가 하나 있으면 작업이 완료된 것으로 간주된다. 이 모드를 사용하는 방법에 대한 자세한 내용은 [정적 작업 할당을 사용한 병렬 처리를 위해 인덱싱된 잡](/docs/tasks/job/indexed-parallel-processing-static/)을 참고한다. @@ -255,7 +254,8 @@ _작업 큐_ 잡은 `.spec.completions` 를 설정하지 않은 상태로 두고 ## 잡의 종료와 정리 -잡이 완료되면 파드가 더 이상 생성되지도 않지만, 삭제되지도 않는다. 이를 유지하면 +잡이 완료되면 파드가 더 이상 생성되지도 않지만, [일반적으로는](#pod-backoff-failure-policy) 삭제되지도 않는다. +이를 유지하면 완료된 파드의 로그를 계속 보며 에러, 경고 또는 다른 기타 진단 출력을 확인할 수 있다. 잡 오브젝트는 완료된 후에도 상태를 볼 수 있도록 남아 있다. 상태를 확인한 후 이전 잡을 삭제하는 것은 사용자의 몫이다. `kubectl` 로 잡을 삭제할 수 있다 (예: `kubectl delete jobs/pi` 또는 `kubectl delete -f ./job.yaml`). `kubectl` 을 사용해서 잡을 삭제하면 생성된 모든 파드도 함께 삭제된다. @@ -402,14 +402,12 @@ spec: ### 잡 일시 중지 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} {{< note >}} -잡 일시 중지는 쿠버네티스 버전 1.21 이상에서 사용할 수 있다. 이 기능을 -사용하려면 [API 서버](/docs/reference/command-line-tools-reference/kube-apiserver/) -및 [컨트롤러 관리자](/docs/reference/command-line-tools-reference/kube-controller-manager/)에서 -`SuspendJob` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 -활성화해야 한다. +이 기능은 쿠버네티스 버전 1.21에서는 알파 상태였으며, +이 때문에 이 기능을 활성화하기 위해서는 추가적인 단계를 진행해야 한다. +[현재 사용 중인 쿠버네티스 버전과 맞는 문서](/ko/docs/home/supported-doc-versions/)를 읽고 있는 것이 맞는지 다시 한번 확인한다. {{< /note >}} 잡이 생성되면, 잡 컨트롤러는 잡의 요구 사항을 충족하기 위해 @@ -568,6 +566,46 @@ spec: `manualSelector: true` 를 설정하면 시스템에게 사용자가 무엇을 하는지 알고 있음을 알리고, 이런 불일치를 허용한다. +### 종료자(finalizers)를 이용한 잡 추적 + +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + +{{< note >}} +이 기능을 이용하기 위해서는 +[API 서버](/docs/reference/command-line-tools-reference/kube-apiserver/)와 +[컨트롤러 매니저](/docs/reference/command-line-tools-reference/kube-controller-manager/)에 대해 +`JobTrackingWithFinalizers` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 활성화해야 한다. +기본적으로는 비활성화되어 있다. + +이 기능이 활성화되면, 컨트롤 플레인은 아래에 설명할 동작을 이용하여 새로운 잡이 생성되는지 추적한다. +기존에 존재하던 잡은 영향을 받지 않는다. +사용자가 느낄 수 있는 유일한 차이점은 컨트롤 플레인이 잡 종료를 좀 더 정확하게 추적할 수 있다는 것이다. +{{< /note >}} + +이 기능이 활성화되지 않으면, 잡 +{{< glossary_tooltip text="컨트롤러" term_id="controller" >}}는 +`succeeded`와 `failed` 파드의 수를 세어 잡 상태를 추적한다. +그런데, 파드는 다음과 같은 이유로 제거될 수 있다. +- 노드가 다운되었을 때 가비지 콜렉터가 버려진(orphan) 파드를 제거 +- 가비지 콜렉터가 (`Succeeded` 또는 `Failed` 단계에 있는) 완료된 파드를 + 일정 임계값 이후에 제거 +- 잡에 속한 파드를 사용자가 임의로 제거 +- (쿠버네티스에 속하지 않는) 외부 컨트롤러가 파드를 제거하거나 + 교체 + +클러스터에서 `JobTrackingWithFinalizers` 기능을 활성화하면, +컨트롤 플레인은 잡에 속하는 파드의 상태를 추적하고 +API 서버에서 파드가 제거되면 이를 알아챈다. +이를 위해, 잡 컨트롤러는 `batch.kubernetes.io/job-tracking` 종료자를 갖는 파드를 생성한다. +컨트롤러는 파드의 상태 변화가 잡 상태에 반영된 후에만 종료자를 제거하므로, +이후 다른 컨트롤러나 사용자가 파드를 제거할 수 있다. + +잡 컨트롤러는 새로운 잡에 대해서만 새로운 알고리즘을 적용한다. +이 기능이 활성화되기 전에 생성된 잡은 영향을 받지 않는다. +잡에 `batch.kubernetes.io/job-tracking` 어노테이션이 있는지 확인하여, +잡 컨트롤러가 파드 종료자를 이용하여 잡을 추적하고 있는지 여부를 확인할 수 있다. +이 어노테이션을 잡에 수동으로 추가하거나 제거해서는 **안 된다**. + ## 대안 ### 베어(Bare) 파드 diff --git a/content/ko/docs/concepts/workloads/controllers/replicaset.md b/content/ko/docs/concepts/workloads/controllers/replicaset.md index 7cf399d242..14056eef1d 100644 --- a/content/ko/docs/concepts/workloads/controllers/replicaset.md +++ b/content/ko/docs/concepts/workloads/controllers/replicaset.md @@ -323,7 +323,7 @@ curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/fron 모든 기준에 대해 동등하다면, 스케일 다운할 파드가 임의로 선택된다. ### 파드 삭제 비용 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} [`controller.kubernetes.io/pod-deletion-cost`](/docs/reference/labels-annotations-taints/#pod-deletion-cost) 어노테이션을 이용하여, 레플리카셋을 스케일 다운할 때 어떤 파드부터 먼저 삭제할지에 대한 우선순위를 설정할 수 있다. @@ -335,9 +335,9 @@ curl -X DELETE 'localhost:8080/apis/apps/v1/namespaces/default/replicasets/fron 파드에 대해 이 값을 명시하지 않으면 기본값은 0이다. 음수로도 설정할 수 있다. 유효하지 않은 값은 API 서버가 거부한다. -이 기능은 알파 상태이며 기본적으로는 비활성화되어 있다. -kube-apiserver와 kube-controller-manager에서 `PodDeletionCost` -[기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 켜서 활성화할 수 있다. +이 기능은 베타 상태이며 기본적으로 활성화되어 있다. +kube-apiserver와 kube-controller-manager에 대해 `PodDeletionCost` +[기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 이용하여 비활성화할 수 있다. {{< note >}} - 이 기능은 best-effort 방식으로 동작하므로, 파드 삭제 순서를 보장하지는 않는다. From 0d589766950787ab98764d9c7186f9431210c1d8 Mon Sep 17 00:00:00 2001 From: mehabhalodiya Date: Tue, 5 Oct 2021 17:43:17 +0530 Subject: [PATCH 124/249] Migrate Contributing new content overview into section index --- .../en/docs/contribute/new-content/_index.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/content/en/docs/contribute/new-content/_index.md b/content/en/docs/contribute/new-content/_index.md index 4992a37654..01677af0ed 100644 --- a/content/en/docs/contribute/new-content/_index.md +++ b/content/en/docs/contribute/new-content/_index.md @@ -1,4 +1,64 @@ --- title: Contributing new content +content_type: concept +main_menu: true weight: 20 --- + + + +This section contains information you should know before contributing new content. + + + + + + +## Contributing basics + +- Write Kubernetes documentation in Markdown and build the Kubernetes site using [Hugo](https://gohugo.io/). +- The source is in [GitHub](https://github.com/kubernetes/website). You can find Kubernetes documentation at `/content/en/docs/`. Some of the reference documentation is automatically generated from scripts in the `update-imported-docs/` directory. +- [Page content types](/docs/contribute/style/page-content-types/) describe the presentation of documentation content in Hugo. +- In addition to the standard Hugo shortcodes, we use a number of + [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our documentation to control the presentation of content. +- Documentation source is available in multiple languages in `/content/`. Each + language has its own folder with a two-letter code determined by the + [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php). For + example, English documentation source is stored in `/content/en/docs/`. +- For more information about contributing to documentation in multiple languages or starting a new translation, see [localization](/docs/contribute/localization). + +## Before you begin {#before-you-begin} + +### Sign the CNCF CLA {#sign-the-cla} + +All Kubernetes contributors **must** read the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md) and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md). + +Pull requests from contributors who haven't signed the CLA fail the automated tests. The name and email you provide must match those found in your `git config`, and your git name and email must match those used for the CNCF CLA. + +### Choose which Git branch to use + +When opening a pull request, you need to know in advance which branch to base your work on. + +Scenario | Branch +:---------|:------------ +Existing or new English language content for the current release | `main` +Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch. +Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information. + + +If you're still not sure which branch to choose, ask in `#sig-docs` on Slack. + +{{< note >}} +If you already submitted your pull request and you know that the base branch +was wrong, you (and only you, the submitter) can change it. +{{< /note >}} + +### Languages per PR + +Limit pull requests to one language per PR. If you need to make an identical change to the same code sample in multiple languages, open a separate PR for each language. + +## Tools for contributors + +The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools) directory in the `kubernetes/website` repository contains tools to help your contribution journey go more smoothly. + + From 716da39b217508ba71efea3d5b3351b3322e19a1 Mon Sep 17 00:00:00 2001 From: mehabhalodiya Date: Tue, 5 Oct 2021 17:51:45 +0530 Subject: [PATCH 125/249] Redirect overview to new content --- static/_redirects | 1 + 1 file changed, 1 insertion(+) diff --git a/static/_redirects b/static/_redirects index 3274c895a7..6868f56b79 100644 --- a/static/_redirects +++ b/static/_redirects @@ -169,6 +169,7 @@ /docs/contribute/write-new-topic/ /docs/home/contribute/write-new-topic/ 301 /docs/contribute/start/ /docs/contribute/ 301 /docs/contribute/intermediate/ /docs/contribute/ 301 +/docs/contribute/new-content/overview/ /docs/contribute/new-content/ 301 /docs/deprecate/ /docs/reference/using-api/deprecation-policy/ 301 /docs/deprecated/ /docs/reference/using-api/deprecation-policy/ 301 From fb2cc8ed6d7bca5e16daa215645f7ae1a21b4951 Mon Sep 17 00:00:00 2001 From: mehabhalodiya Date: Tue, 5 Oct 2021 17:56:08 +0530 Subject: [PATCH 126/249] Delete overview.md --- .../docs/contribute/new-content/overview.md | 65 ------------------- 1 file changed, 65 deletions(-) delete mode 100644 content/en/docs/contribute/new-content/overview.md diff --git a/content/en/docs/contribute/new-content/overview.md b/content/en/docs/contribute/new-content/overview.md deleted file mode 100644 index 8b4da4970a..0000000000 --- a/content/en/docs/contribute/new-content/overview.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: Contributing new content overview -linktitle: Overview -content_type: concept -main_menu: true -weight: 5 ---- - - - -This section contains information you should know before contributing new content. - - - - - - -## Contributing basics - -- Write Kubernetes documentation in Markdown and build the Kubernetes site using [Hugo](https://gohugo.io/). -- The source is in [GitHub](https://github.com/kubernetes/website). You can find Kubernetes documentation at `/content/en/docs/`. Some of the reference documentation is automatically generated from scripts in the `update-imported-docs/` directory. -- [Page content types](/docs/contribute/style/page-content-types/) describe the presentation of documentation content in Hugo. -- In addition to the standard Hugo shortcodes, we use a number of - [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our documentation to control the presentation of content. -- Documentation source is available in multiple languages in `/content/`. Each - language has its own folder with a two-letter code determined by the - [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php). For - example, English documentation source is stored in `/content/en/docs/`. -- For more information about contributing to documentation in multiple languages or starting a new translation, see [localization](/docs/contribute/localization). - -## Before you begin {#before-you-begin} - -### Sign the CNCF CLA {#sign-the-cla} - -All Kubernetes contributors **must** read the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md) and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md). - -Pull requests from contributors who haven't signed the CLA fail the automated tests. The name and email you provide must match those found in your `git config`, and your git name and email must match those used for the CNCF CLA. - -### Choose which Git branch to use - -When opening a pull request, you need to know in advance which branch to base your work on. - -Scenario | Branch -:---------|:------------ -Existing or new English language content for the current release | `main` -Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch. -Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information. - - -If you're still not sure which branch to choose, ask in `#sig-docs` on Slack. - -{{< note >}} -If you already submitted your pull request and you know that the base branch -was wrong, you (and only you, the submitter) can change it. -{{< /note >}} - -### Languages per PR - -Limit pull requests to one language per PR. If you need to make an identical change to the same code sample in multiple languages, open a separate PR for each language. - -## Tools for contributors - -The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools) directory in the `kubernetes/website` repository contains tools to help your contribution journey go more smoothly. - - From adad63fa5995274e0a230a60445a0c996ecf5696 Mon Sep 17 00:00:00 2001 From: Seokho Son Date: Wed, 6 Oct 2021 00:58:54 +0900 Subject: [PATCH 127/249] Update outdated files in dev-1.22-ko.1 (p1) --- content/ko/community/_index.html | 4 +-- content/ko/docs/concepts/containers/images.md | 13 ++++----- .../docs/concepts/containers/runtime-class.md | 5 +++- .../compute-storage-net/device-plugins.md | 25 +++++++++++------ .../concepts/overview/what-is-kubernetes.md | 2 +- .../working-with-objects/annotations.md | 5 ++++ .../overview/working-with-objects/labels.md | 6 ++--- .../overview/working-with-objects/names.md | 18 +++++++++++-- .../docs/concepts/policy/resource-quotas.md | 6 ++--- .../scheduling-eviction/assign-pod-node.md | 6 ++--- .../scheduling-eviction/kube-scheduler.md | 2 +- .../pod-priority-preemption.md | 27 ++++++++++++------- .../scheduler-perf-tuning.md | 4 +-- .../taint-and-toleration.md | 6 ++--- content/ko/docs/concepts/security/overview.md | 9 ++++--- 15 files changed, 87 insertions(+), 51 deletions(-) diff --git a/content/ko/community/_index.html b/content/ko/community/_index.html index 0fa8d8658a..9666cd4a14 100644 --- a/content/ko/community/_index.html +++ b/content/ko/community/_index.html @@ -12,8 +12,8 @@ cid: community

    -

    사용자, 기여자, 그리고 우리가 함께 구축한 문화로 구성된 쿠버네티스 커뮤니티는 이 오픈소스 프로젝트가 급부상하는 가장 큰 이유 중 하나입니다. 프로젝트 자체가 성장 하고 변화함에 따라 우리의 문화와 가치관이 계속 성장하고 변화하고 있습니다. 우리 모두는 프로젝트의 지속적인 개선과 작업 방식을 위해 함께 노력합니다. -

    우리는 이슈를 제기하고 풀 리퀘스트하고, SIG 미팅과 쿠버네티스 모임 그리고 KubeCon에 참석하고 채택과 혁신을 옹호하며, kubectl get pods 을 실행하고, 다른 수천가지 중요한 방법으로 기여하는 사람들 입니다. 여러분이 어떻게 이 놀라운 공동체의 일부가 될 수 있는지 계속 읽어보세요.

    +

    사용자, 기여자, 그리고 우리가 함께 구축한 문화를 통해 구성된 쿠버네티스 커뮤니티는 본 오픈소스 프로젝트가 급부상하는 가장 큰 이유 중 하나입니다. 프로젝트 자체가 성장하고 변화함에 따라 우리의 문화와 가치관 또한 지속적으로 성장하고 변화하고 있습니다. 우리 모두는 프로젝트와 작업 방식을 지속적으로 개선하기 위해 함께 노력합니다. +

    우리는 이슈(issue)와 풀 리퀘스트(pull request)를 제출하고, SIG 미팅과 쿠버네티스 모임 그리고 KubeCon에 참석하고, 도입(adoption)과 혁신(innovation)을 지지하며, kubectl get pods 를 실행하고, 다른 수천가지 중요한 방법으로 기여하는 사람들 입니다. 어떻게 하면 이 놀라운 공동체의 일부가 될 수 있는지 계속 읽어보세요.


    diff --git a/content/ko/docs/concepts/containers/images.md b/content/ko/docs/concepts/containers/images.md index 886f8247a3..9a22f2f3e5 100644 --- a/content/ko/docs/concepts/containers/images.md +++ b/content/ko/docs/concepts/containers/images.md @@ -1,4 +1,7 @@ --- + + + title: 이미지 content_type: concept weight: 10 @@ -16,9 +19,6 @@ weight: 10 이 페이지는 컨테이너 이미지 개념의 개요를 제공한다. - - - ## 이미지 이름 @@ -210,10 +210,6 @@ kubectl describe pods/private-image-test-1 | grep 'Failed' ### 미리 내려받은 이미지 -{{< note >}} -Google 쿠버네티스 엔진에서 동작 중이라면, 이미 각 노드에 Google 컨테이너 레지스트리에 대한 자격 증명과 함께 `.dockercfg`가 있을 것이다. 그렇다면 이 방법은 쓸 수 없다. -{{< /note >}} - {{< note >}} 이 방법은 노드의 구성을 제어할 수 있는 경우에만 적합하다. 이 방법은 클라우드 제공자가 노드를 관리하고 자동으로 교체한다면 안정적으로 @@ -334,4 +330,5 @@ Kubelet은 모든 `imagePullSecrets` 파일을 하나의 가상 `.docker/config. ## {{% heading "whatsnext" %}} -* [OCI 이미지 매니페스트 명세](https://github.com/opencontainers/image-spec/blob/master/manifest.md) 읽어보기 +* [OCI 이미지 매니페스트 명세](https://github.com/opencontainers/image-spec/blob/master/manifest.md) 읽어보기. +* [컨테이너 이미지 가비지 수집(garbage collection)](/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection)에 대해 배우기. diff --git a/content/ko/docs/concepts/containers/runtime-class.md b/content/ko/docs/concepts/containers/runtime-class.md index 953571ec62..a2142521cd 100644 --- a/content/ko/docs/concepts/containers/runtime-class.md +++ b/content/ko/docs/concepts/containers/runtime-class.md @@ -1,4 +1,7 @@ --- + + + title: 런타임클래스(RuntimeClass) content_type: concept weight: 20 @@ -115,7 +118,7 @@ dockershim은 사용자 정의 런타임 핸들러를 지원하지 않는다. 유효한 핸들러는 runtimes 단락 아래에서 설정한다. ``` -[plugins.cri.containerd.runtimes.${HANDLER_NAME}] +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}] ``` 더 자세한 containerd의 구성 문서를 살펴본다. diff --git a/content/ko/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/ko/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 13313adf58..9d4ad5525c 100644 --- a/content/ko/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/ko/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -197,31 +197,39 @@ service PodResourcesLister { } ``` -`List` 엔드포인트는 독점적으로 할당된 CPU의 ID, 장치 플러그인에 의해 보고된 장치 ID, -이러한 장치가 할당된 NUMA 노드의 ID와 같은 세부 정보와 함께 -실행 중인 파드의 리소스에 대한 정보를 제공한다. +`List` 엔드포인트는 실행 중인 파드의 리소스에 대한 정보를 제공하며, +독점적으로 할당된 CPU의 ID, 장치 플러그인에 의해 보고된 장치 ID, +이러한 장치가 할당된 NUMA 노드의 ID와 같은 세부 정보를 함께 제공한다. 또한, NUMA 기반 머신의 경우, 컨테이너를 위해 예약된 메모리와 hugepage에 대한 정보를 포함한다. ```gRPC -// ListPodResourcesResponse는 List 함수가 반환하는 응답이다 +// ListPodResourcesResponse는 List 함수가 반환하는 응답이다. message ListPodResourcesResponse { repeated PodResources pod_resources = 1; } -// PodResources에는 파드에 할당된 노드 리소스에 대한 정보가 포함된다 +// PodResources에는 파드에 할당된 노드 리소스에 대한 정보가 포함된다. message PodResources { string name = 1; string namespace = 2; repeated ContainerResources containers = 3; } -// ContainerResources는 컨테이너에 할당된 리소스에 대한 정보를 포함한다 +// ContainerResources는 컨테이너에 할당된 리소스에 대한 정보를 포함한다. message ContainerResources { string name = 1; repeated ContainerDevices devices = 2; repeated int64 cpu_ids = 3; + repeated ContainerMemory memory = 4; } -// 토폴로지는 리소스의 하드웨어 토폴로지를 설명한다 +// ContainerMemory는 컨테이너에 할당된 메모리와 hugepage에 대한 정보를 포함한다. +message ContainerMemory { + string memory_type = 1; + uint64 size = 2; + TopologyInfo topology = 3; +} + +// 토폴로지는 리소스의 하드웨어 토폴로지를 설명한다. message TopologyInfo { repeated NUMANode nodes = 1; } @@ -231,7 +239,7 @@ message NUMANode { int64 ID = 1; } -// ContainerDevices는 컨테이너에 할당된 장치에 대한 정보를 포함한다 +// ContainerDevices는 컨테이너에 할당된 장치에 대한 정보를 포함한다. message ContainerDevices { string resource_name = 1; repeated string device_ids = 2; @@ -247,6 +255,7 @@ kubelet이 APIServer로 내보내는 것보다 더 많은 정보를 제공한다 message AllocatableResourcesResponse { repeated ContainerDevices devices = 1; repeated int64 cpu_ids = 2; + repeated ContainerMemory memory = 3; } ``` diff --git a/content/ko/docs/concepts/overview/what-is-kubernetes.md b/content/ko/docs/concepts/overview/what-is-kubernetes.md index 5d2ef83d76..2b6809782d 100644 --- a/content/ko/docs/concepts/overview/what-is-kubernetes.md +++ b/content/ko/docs/concepts/overview/what-is-kubernetes.md @@ -45,7 +45,7 @@ sitemap: * 기민한 애플리케이션 생성과 배포: VM 이미지를 사용하는 것에 비해 컨테이너 이미지 생성이 보다 쉽고 효율적임. * 지속적인 개발, 통합 및 배포: 안정적이고 주기적으로 컨테이너 이미지를 빌드해서 배포할 수 있고 (이미지의 불변성 덕에) 빠르고 효율적으로 롤백할 수 있다. * 개발과 운영의 관심사 분리: 배포 시점이 아닌 빌드/릴리스 시점에 애플리케이션 컨테이너 이미지를 만들기 때문에, 애플리케이션이 인프라스트럭처에서 분리된다. -* 가시성은 OS 수준의 정보와 메트릭에 머무르지 않고, 애플리케이션의 헬스와 그 밖의 시그널을 볼 수 있다. +* 가시성(observability): OS 수준의 정보와 메트릭에 머무르지 않고, 애플리케이션의 헬스와 그 밖의 시그널을 볼 수 있다. * 개발, 테스팅 및 운영 환경에 걸친 일관성: 랩탑에서도 클라우드에서와 동일하게 구동된다. * 클라우드 및 OS 배포판 간 이식성: Ubuntu, RHEL, CoreOS, 온-프레미스, 주요 퍼블릭 클라우드와 어디에서든 구동된다. * 애플리케이션 중심 관리: 가상 하드웨어 상에서 OS를 실행하는 수준에서 논리적인 리소스를 사용하는 OS 상에서 애플리케이션을 실행하는 수준으로 추상화 수준이 높아진다. diff --git a/content/ko/docs/concepts/overview/working-with-objects/annotations.md b/content/ko/docs/concepts/overview/working-with-objects/annotations.md index 245da33db3..89334978c0 100644 --- a/content/ko/docs/concepts/overview/working-with-objects/annotations.md +++ b/content/ko/docs/concepts/overview/working-with-objects/annotations.md @@ -30,6 +30,11 @@ weight: 50 } ``` +{{}} +맵의 키와 값은 문자열이어야 한다. 다르게 말해서, 숫자, +불리언(boolean), 리스트 등의 다른 형식을 키나 값에 사용할 수 없다. +{{}} + 다음은 어노테이션에 기록할 수 있는 정보의 예제이다. * 필드는 선언적 구성 계층에 의해 관리된다. 이러한 필드를 어노테이션으로 첨부하는 것은 diff --git a/content/ko/docs/concepts/overview/working-with-objects/labels.md b/content/ko/docs/concepts/overview/working-with-objects/labels.md index 0583ae0fe3..571c62b7db 100644 --- a/content/ko/docs/concepts/overview/working-with-objects/labels.md +++ b/content/ko/docs/concepts/overview/working-with-objects/labels.md @@ -42,7 +42,7 @@ _레이블_ 은 파드와 같은 오브젝트에 첨부된 키와 값의 쌍이 * `"partition" : "customerA"`, `"partition" : "customerB"` * `"track" : "daily"`, `"track" : "weekly"` -이 예시는 일반적으로 사용하는 레이블이며, 사용자는 자신만의 규칙(convention)에 따라 자유롭게 개발할 수 있다. 오브젝트에 붙여진 레이블 키는 고유해야 한다는 것을 기억해야 한다. +이 예시는 [일반적으로 사용하는 레이블](/ko/docs/concepts/overview/working-with-objects/common-labels/)이며, 사용자는 자신만의 규칙(convention)에 따라 자유롭게 개발할 수 있다. 오브젝트에 붙여진 레이블 키는 고유해야 한다는 것을 기억해야 한다. ## 구문과 캐릭터 셋 @@ -50,15 +50,13 @@ _레이블_ 은 키와 값의 쌍이다. 유효한 레이블 키에는 슬래시 접두사를 생략하면 키 레이블은 개인용으로 간주한다. 최종 사용자의 오브젝트에 자동화된 시스템 컴포넌트(예: `kube-scheduler`, `kube-controller-manager`, `kube-apiserver`, `kubectl` 또는 다른 타사의 자동화 구성 요소)의 접두사를 지정해야 한다. -`kubernetes.io/`와 `k8s.io/` 접두사는 쿠버네티스의 핵심 컴포넌트로 예약되어있다. +`kubernetes.io/`와 `k8s.io/` 접두사는 쿠버네티스의 핵심 컴포넌트로 [예약](/ko/docs/reference/labels-annotations-taints/)되어있다. 유효한 레이블 값은 다음과 같다. * 63 자 이하여야 하고 (공백일 수도 있음), * (공백이 아니라면) 시작과 끝은 알파벳과 숫자(`[a-z0-9A-Z]`)이며, * 알파벳과 숫자, 대시(`-`), 밑줄(`_`), 점(`.`)을 중간에 포함할 수 있다. -유효한 레이블 값은 63자 미만 또는 공백이며 시작과 끝은 알파벳과 숫자(`[a-z0-9A-Z]`)이며, 대시(`-`), 밑줄(`_`), 점(`.`)과 함께 사용할 수 있다. - 다음의 예시는 파드에 `environment: production` 과 `app: nginx` 2개의 레이블이 있는 구성 파일이다. ```yaml diff --git a/content/ko/docs/concepts/overview/working-with-objects/names.md b/content/ko/docs/concepts/overview/working-with-objects/names.md index 78b7addd43..69afaa0069 100644 --- a/content/ko/docs/concepts/overview/working-with-objects/names.md +++ b/content/ko/docs/concepts/overview/working-with-objects/names.md @@ -1,4 +1,7 @@ --- + + + title: 오브젝트 이름과 ID content_type: concept weight: 20 @@ -25,7 +28,7 @@ weight: 20 물리적 호스트를 나타내는 노드와 같이 오브젝트가 물리적 엔티티를 나타내는 경우, 노드를 삭제한 후 다시 생성하지 않은 채 동일한 이름으로 호스트를 다시 생성하면, 쿠버네티스는 새 호스트를 불일치로 이어질 수 있는 이전 호스트로 취급한다. {{< /note >}} -다음은 리소스에 일반적으로 사용되는 세 가지 유형의 이름 제한 조건이다. +다음은 리소스에 일반적으로 사용되는 네 가지 유형의 이름 제한 조건이다. ### DNS 서브도메인 이름 @@ -38,7 +41,7 @@ DNS 서브도메인 이름으로 사용할 수 있는 이름이 필요하다. - 영숫자로 시작한다. - 영숫자로 끝난다. -### DNS 레이블 이름 +### RFC 1123 레이블 이름 {#dns-label-names} 일부 리소스 유형은 [RFC 1123](https://tools.ietf.org/html/rfc1123)에 정의된 대로 DNS 레이블 표준을 따라야 한다. @@ -49,6 +52,17 @@ DNS 서브도메인 이름으로 사용할 수 있는 이름이 필요하다. - 영숫자로 시작한다. - 영숫자로 끝난다. +### RFC 1035 레이블 이름 + +몇몇 리소스 타입은 자신의 이름을 [RFC 1035](https://tools.ietf.org/html/rfc1035)에 +정의된 DNS 레이블 표준을 따르도록 요구한다. +이것은 이름이 다음을 만족해야 한다는 의미이다. + +- 최대 63개 문자를 포함 +- 소문자 영숫자 또는 '-'만 포함 +- 알파벳 문자로 시작 +- 영숫자로 끝남 + ### 경로 세그먼트 이름 일부 리소스 유형에서는 이름을 경로 세그먼트로 안전하게 인코딩 할 수 diff --git a/content/ko/docs/concepts/policy/resource-quotas.md b/content/ko/docs/concepts/policy/resource-quotas.md index b5254e4300..b7cb48b23e 100644 --- a/content/ko/docs/concepts/policy/resource-quotas.md +++ b/content/ko/docs/concepts/policy/resource-quotas.md @@ -442,7 +442,7 @@ pods 0 10 ### 네임스페이스 간 파드 어피니티 쿼터 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} 오퍼레이터는 네임스페이스를 교차하는 어피니티가 있는 파드를 가질 수 있는 네임스페이스를 제한하기 위해 `CrossNamespacePodAffinity` 쿼터 범위를 사용할 수 있다. 특히, 파드 어피니티 용어의 @@ -493,9 +493,9 @@ plugins: 해당 필드를 사용하는 파드 수보다 크거나 같은 하드 제한이 있는 경우에만 파드 어피니티에서 `namespaces` 및 `namespaceSelector` 를 사용할 수 있다. -이 기능은 알파이며 기본적으로 비활성화되어 있다. kube-apiserver 및 kube-scheduler 모두에서 +이 기능은 베타이며 기본으로 활성화되어 있다. kube-apiserver 및 kube-scheduler 모두에서 [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/) -`PodAffinityNamespaceSelector` 를 설정하여 활성화할 수 있다. +`PodAffinityNamespaceSelector` 를 사용하여 비활성화할 수 있다. ## 요청과 제한의 비교 {#requests-vs-limits} diff --git a/content/ko/docs/concepts/scheduling-eviction/assign-pod-node.md b/content/ko/docs/concepts/scheduling-eviction/assign-pod-node.md index f46e075b57..1e6a007cdf 100644 --- a/content/ko/docs/concepts/scheduling-eviction/assign-pod-node.md +++ b/content/ko/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -271,16 +271,16 @@ PodSpec에 지정된 NodeAffinity도 적용된다. 연관된 `matchExpressions` 가 모두 충족되어야 한다. #### 네임스페이스 셀렉터 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} 사용자는 네임스페이스 집합에 대한 레이블 쿼리인 `namespaceSelector` 를 사용하여 일치하는 네임스페이스를 선택할 수도 있다. 어피니티 용어는 `namespaceSelector` 에서 선택한 네임스페이스와 `namespaces` 필드에 나열된 네임스페이스의 결합에 적용된다. 빈 `namespaceSelector` ({})는 모든 네임스페이스와 일치하는 반면, null 또는 빈 `namespaces` 목록과 null `namespaceSelector` 는 "이 파드의 네임스페이스"를 의미한다. -이 기능은 알파이며 기본적으로 비활성화되어 있다. kube-apiserver 및 kube-scheduler 모두에서 +이 기능은 베타이며 기본으로 활성화되어 있다. kube-apiserver 및 kube-scheduler 모두에서 [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/) -`PodAffinityNamespaceSelector` 를 설정하여 활성화할 수 있다. +`PodAffinityNamespaceSelector` 를 사용하여 비활성화할 수 있다. #### 더 실용적인 유스케이스 diff --git a/content/ko/docs/concepts/scheduling-eviction/kube-scheduler.md b/content/ko/docs/concepts/scheduling-eviction/kube-scheduler.md index 8c17269a64..5b0a1648c3 100644 --- a/content/ko/docs/concepts/scheduling-eviction/kube-scheduler.md +++ b/content/ko/docs/concepts/scheduling-eviction/kube-scheduler.md @@ -85,7 +85,7 @@ _스코어링_ 단계에서 스케줄러는 목록에 남아있는 노드의 순 * [스케줄러 성능 튜닝](/ko/docs/concepts/scheduling-eviction/scheduler-perf-tuning/)에 대해 읽기 * [파드 토폴로지 분배 제약 조건](/ko/docs/concepts/workloads/pods/pod-topology-spread-constraints/)에 대해 읽기 * kube-scheduler의 [레퍼런스 문서](/docs/reference/command-line-tools-reference/kube-scheduler/) 읽기 -* [kube-scheduler 구성(v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) 레퍼런스 읽기 +* [kube-scheduler 구성(v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) 레퍼런스 읽기 * [멀티 스케줄러 구성하기](/docs/tasks/extend-kubernetes/configure-multiple-schedulers/)에 대해 배우기 * [토폴로지 관리 정책](/docs/tasks/administer-cluster/topology-manager/)에 대해 배우기 * [파드 오버헤드](/ko/docs/concepts/scheduling-eviction/pod-overhead/)에 대해 배우기 diff --git a/content/ko/docs/concepts/scheduling-eviction/pod-priority-preemption.md b/content/ko/docs/concepts/scheduling-eviction/pod-priority-preemption.md index f149290882..6df4ec16b4 100644 --- a/content/ko/docs/concepts/scheduling-eviction/pod-priority-preemption.md +++ b/content/ko/docs/concepts/scheduling-eviction/pod-priority-preemption.md @@ -1,4 +1,7 @@ --- + + + title: 파드 우선순위(priority)와 선점(preemption) content_type: concept weight: 70 @@ -350,21 +353,25 @@ spec: `PodDisruptionBudget` 으로 보호되는 경우에만, 우선순위가 가장 낮은 파드를 축출 대상으로 고려한다. -QoS와 파드 우선순위를 모두 고려하는 유일한 컴포넌트는 -[kubelet 리소스 부족 축출](/docs/concepts/scheduling-eviction/node-pressure-eviction/)이다. -kubelet은 부족한 리소스의 사용이 요청을 초과하는지 여부에 따라, 그런 다음 우선순위에 따라, -파드의 스케줄링 요청에 대한 부족한 컴퓨팅 리소스의 소비에 의해 -먼저 축출 대상 파드의 순위를 매긴다. -더 자세한 내용은 -[엔드유저 파드 축출](/docs/concepts/scheduling-eviction/node-pressure-eviction/#evicting-end-user-pods)을 +kubelet은 우선순위를 사용하여 파드의 [노드-압박(node-pressure) 축출](/ko/docs/concepts/scheduling-eviction/node-pressure-eviction/) 순서를 결정한다. +사용자는 QoS 클래스를 사용하여 어떤 파드가 축출될 것인지 +예상할 수 있다. kubelet은 다음의 요소들을 통해서 파드의 축출 순위를 매긴다. + + 1. 부족한 리소스 사용량이 요청을 초과하는지 여부 + 1. 파드 우선순위 + 1. 요청 대비 리소스 사용량 + +더 자세한 내용은 [kubelet 축출에서 파드 선택](/ko/docs/concepts/scheduling-eviction/node-pressure-eviction/#kubelet-축출을-위한-파드-선택)을 참조한다. -kubelet 리소스 부족 축출은 사용량이 요청을 초과하지 않는 경우 +kubelet 노드-압박 축출은 사용량이 요청을 초과하지 않는 경우 파드를 축출하지 않는다. 우선순위가 낮은 파드가 요청을 초과하지 않으면, 축출되지 않는다. 요청을 초과하는 우선순위가 더 높은 다른 파드가 축출될 수 있다. - ## {{% heading "whatsnext" %}} -* 프라이어리티클래스와 관련하여 리소스쿼터 사용에 대해 [기본적으로 프라이어리티클래스 소비 제한](/ko/docs/concepts/policy/resource-quotas/#기본적으로-우선-순위-클래스-소비-제한)을 읽어보자. +* 프라이어리티클래스와 함께 리소스쿼터 사용에 대해 읽기: [기본으로 프라이어리티 클래스 소비 제한](/ko/docs/concepts/policy/resource-quotas/#기본적으로-우선-순위-클래스-소비-제한) +* [파드 중단(disruption)](/ko/docs/concepts/workloads/pods/disruptions/)에 대해 학습한다. +* [API를 이용한 축출](/ko/docs/concepts/scheduling-eviction/api-eviction/)에 대해 학습한다. +* [노드-압박(node-pressure) 축출](/ko/docs/concepts/scheduling-eviction/node-pressure-eviction/)에 대해 학습한다. diff --git a/content/ko/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md b/content/ko/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md index 6be3e204c8..e2fb1cdcf9 100644 --- a/content/ko/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md +++ b/content/ko/docs/concepts/scheduling-eviction/scheduler-perf-tuning.md @@ -43,7 +43,7 @@ kube-scheduler 의 `percentageOfNodesToScore` 설정을 통해 마치 100을 설정한 것처럼 작동한다. 값을 변경하려면, -[kube-scheduler 구성 파일](/docs/reference/config-api/kube-scheduler-config.v1beta1/)을 +[kube-scheduler 구성 파일](/docs/reference/config-api/kube-scheduler-config.v1beta2/)을 편집한 다음 스케줄러를 재시작한다. 대부분의 경우, 구성 파일은 `/etc/kubernetes/config/kube-scheduler.yaml` 에서 찾을 수 있다. @@ -161,4 +161,4 @@ percentageOfNodesToScore: 50 ## {{% heading "whatsnext" %}} -* [kube-scheduler 구성 레퍼런스(v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) 확인 +* [kube-scheduler 구성 레퍼런스(v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) 확인 diff --git a/content/ko/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/ko/docs/concepts/scheduling-eviction/taint-and-toleration.md index c47f5f995b..f6d8c13a19 100644 --- a/content/ko/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/ko/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -264,10 +264,10 @@ tolerations: 이렇게 하면 이러한 문제로 인해 데몬셋 파드가 축출되지 않는다. -## 컨디션을 기준으로 노드 테인트하기 +## 조건(condition)을 기준으로 노드 테인트하기 컨트롤 플레인은 노드 {{}}를 이용하여 -[노드 조건](/docs/concepts/scheduling-eviction/node-pressure-eviction/)에 대한 `NoSchedule` 효과를 사용하여 자동으로 테인트를 생성한다. +[노드 조건](/ko/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions)에 대한 `NoSchedule` 효과를 사용하여 자동으로 테인트를 생성한다. 스케줄러는 스케줄링 결정을 내릴 때 노드 조건을 확인하는 것이 아니라 테인트를 확인한다. 이렇게 하면 노드 조건이 스케줄링에 직접적인 영향을 주지 않는다. @@ -298,5 +298,5 @@ tolerations: ## {{% heading "whatsnext" %}} -* [리소스 부족 다루기](/docs/concepts/scheduling-eviction/node-pressure-eviction/)와 어떻게 구성하는지에 대해 알아보기 +* [노드-압박(node-pressure) 축출](/ko/docs/concepts/scheduling-eviction/node-pressure-eviction/)과 어떻게 구성하는지에 대해 알아보기 * [파드 우선순위](/ko/docs/concepts/scheduling-eviction/pod-priority-preemption/)에 대해 알아보기 diff --git a/content/ko/docs/concepts/security/overview.md b/content/ko/docs/concepts/security/overview.md index 64ed2675b2..47d0548a7e 100644 --- a/content/ko/docs/concepts/security/overview.md +++ b/content/ko/docs/concepts/security/overview.md @@ -1,17 +1,21 @@ --- + + title: 클라우드 네이티브 보안 개요 +description: > + 클라우드 네이티브 보안 관점에서 쿠버네티스 보안을 생각해보기 위한 모델 content_type: concept -weight: 10 +weight: 1 --- + 이 개요는 클라우드 네이티브 보안의 맥락에서 쿠버네티스 보안에 대한 생각의 모델을 정의한다. {{< warning >}} 이 컨테이너 보안 모델은 입증된 정보 보안 정책이 아닌 제안 사항을 제공한다. {{< /warning >}} - ## 클라우드 네이티브 보안의 4C @@ -83,7 +87,6 @@ etcd 암호화 | 가능한 한 모든 드라이브를 암호화하는 것이 좋 * 설정 가능한 클러스터 컴포넌트의 보안 * 클러스터에서 실행되는 애플리케이션의 보안 - ### 클러스터의 컴포넌트 {#cluster-components} 우발적이거나 악의적인 접근으로부터 클러스터를 보호하고, From ff517544eacfe5ea64a0033abae51f5496e56f73 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 5 Oct 2021 17:11:24 +0100 Subject: [PATCH 128/249] Update to latest CC-BY 4.x license text Copied from https://creativecommons.org/licenses/by/4.0/legalcode.txt --- LICENSE | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/LICENSE b/LICENSE index b6988e7edc..da6ab6cc8f 100644 --- a/LICENSE +++ b/LICENSE @@ -33,7 +33,7 @@ exhaustive, and do not form part of our licenses. material not subject to the license. This includes other CC- licensed material, or material used under an exception or limitation to copyright. More considerations for licensors: - wiki.creativecommons.org/Considerations_for_licensors + wiki.creativecommons.org/Considerations_for_licensors Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the @@ -48,9 +48,9 @@ exhaustive, and do not form part of our licenses. rights in the material. A licensor may make special requests, such as asking that all changes be marked or described. Although not required by our licenses, you are encouraged to - respect those requests where reasonable. More_considerations - for the public: - wiki.creativecommons.org/Considerations_for_licensees + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees ======================================================================= @@ -378,7 +378,7 @@ Section 8 -- Interpretation. Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances -will be considered the "Licensor." The text of the Creative Commons +will be considered the “Licensor.” The text of the Creative Commons public licenses is dedicated to the public domain under the CC0 Public Domain Dedication. Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as @@ -393,3 +393,4 @@ the avoidance of doubt, this paragraph does not form part of the public licenses. Creative Commons may be contacted at creativecommons.org. + From 053fb85c04b3d539d36cf06649b2427c47118149 Mon Sep 17 00:00:00 2001 From: Tim Bannister Date: Tue, 5 Oct 2021 17:14:41 +0100 Subject: [PATCH 129/249] Fix approval rules for GitHub workflows Only SIG Docs leads (chairs + tech leads) should be able to approve changes to workflows. --- .github/workflows/OWNERS | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/OWNERS b/.github/workflows/OWNERS index 404875fc2e..c31fa3a256 100644 --- a/.github/workflows/OWNERS +++ b/.github/workflows/OWNERS @@ -3,6 +3,9 @@ # When modifying this file, consider the security implications of # allowing listed reviewers / approvals to modify or remove any # configured GitHub Actions. +# +options: + no_parent_owners: true reviewers: - sig-docs-leads From 104f241aab3b932f27820bb04fbbbe5e4f9663ae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adolfo=20Garc=C3=ADa=20Veytia=20=28Puerco=29?= Date: Tue, 5 Oct 2021 15:43:34 -0500 Subject: [PATCH 130/249] releng: Bump oct patch release one week later MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As the cherry pick deadline for october '21 lands right during kubecon week, SIG Release decided to move the patch releases one week later. This PR modifies the dates in the schedule data file. Signed-off-by: Adolfo García Veytia (Puerco) --- data/releases/schedule.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/data/releases/schedule.yaml b/data/releases/schedule.yaml index 55d5b12633..2393235733 100644 --- a/data/releases/schedule.yaml +++ b/data/releases/schedule.yaml @@ -1,8 +1,8 @@ schedules: - release: 1.22 next: 1.22.3 - cherryPickDeadline: 2021-10-15 - targetDate: 2021-10-20 + cherryPickDeadline: 2021-10-22 + targetDate: 2021-10-27 endOfLifeDate: 2022-10-28 previousPatches: - release: 1.22.2 @@ -13,8 +13,8 @@ schedules: targetDate: 2021-08-19 - release: 1.21 next: 1.21.6 - cherryPickDeadline: 2021-10-15 - targetDate: 2021-10-20 + cherryPickDeadline: 2021-10-22 + targetDate: 2021-10-27 endOfLifeDate: 2022-06-28 previousPatches: - release: 1.21.5 @@ -35,8 +35,8 @@ schedules: note: Regression https://groups.google.com/g/kubernetes-dev/c/KuF8s2zueFs - release: 1.20 next: 1.20.12 - cherryPickDeadline: 2021-10-15 - targetDate: 2021-10-20 + cherryPickDeadline: 2021-10-22 + targetDate: 2021-10-27 endOfLifeDate: 2022-02-28 previousPatches: - release: 1.20.11 From e5aca814aaf8e120ea39681ca0ab404c339015da Mon Sep 17 00:00:00 2001 From: Charles Pretzer Date: Tue, 5 Oct 2021 14:33:40 -0700 Subject: [PATCH 131/249] Update blog post to include the fix version and fix the link to the PR Signed-off-by: Charles Pretzer --- ...btleties-debugging-an-intermittent-connection-resets.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md b/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md index 12dd2d79a3..bed35dabb5 100644 --- a/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md +++ b/content/en/blog/_posts/2019-03-29-kube-proxy-subtleties-debugging-an-intermittent-connection-resets.md @@ -144,10 +144,9 @@ ways to address it. - Specifically add an iptables rule to drop the packets that are marked as *INVALID*, so it won’t reach to client pod and cause harm. -The fix is drafted (https://github.com/kubernetes/kubernetes/pull/74840), but -unfortunately it didn’t catch the v1.14 release window. However, for the users -that are affected by this bug, there is a way to mitigate the problem by applying -the following rule in your cluster. +The [fix](https://github.com/kubernetes/kubernetes/pull/74840) is available in v1.15+. +However, for the users that are affected by this bug, there is a way to mitigate the +problem by applying the following rule in your cluster. ```yaml apiVersion: extensions/v1beta1 From c8300c0560a4b0ded7c4e7be683279f6d0e35069 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adolfo=20Garc=C3=ADa=20Veytia=20=28Puerco=29?= Date: Tue, 5 Oct 2021 16:55:44 -0500 Subject: [PATCH 132/249] releng: Bump website dates one week later MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit moves the October '21 patch release dates one week later as the original dates for CP and release fell during KubeCon Week. Follow-up to https://github.com/kubernetes/website/pull/29937 Signed-off-by: Adolfo García Veytia (Puerco) --- content/en/releases/patch-releases.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/en/releases/patch-releases.md b/content/en/releases/patch-releases.md index 91bd47bcd0..88725d5b92 100644 --- a/content/en/releases/patch-releases.md +++ b/content/en/releases/patch-releases.md @@ -78,7 +78,7 @@ releases may also occur in between these. | Monthly Patch Release | Cherry Pick Deadline | Target date | | --------------------- | -------------------- | ----------- | -| October 2021 | 2021-10-15 | 2021-10-20 | +| October 2021 | 2021-10-22 | 2021-10-27 | | November 2021 | 2021-11-12 | 2021-11-17 | | December 2021 | 2021-12-10 | 2021-12-15 | | January 2022 | 2021-01-14 | 2021-01-19 | @@ -93,7 +93,7 @@ End of Life for **1.22** is **2022-10-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | |---------------|----------------------|-------------|------| -| 1.22.3 | 2021-10-15 | 2021-10-20 | | +| 1.22.3 | 2021-10-22 | 2021-10-27 | | | 1.22.2 | 2021-09-10 | 2021-09-15 | | | 1.22.1 | 2021-08-16 | 2021-08-19 | | @@ -105,7 +105,7 @@ End of Life for **1.21** is **2022-06-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | | ------------- | -------------------- | ----------- | ---------------------------------------------------------------------- | -| 1.21.6 | 2021-10-15 | 2021-10-20 | | +| 1.21.6 | 2021-10-22 | 2021-10-27 | | | 1.21.5 | 2021-09-10 | 2021-09-15 | | | 1.21.4 | 2021-08-07 | 2021-08-11 | | | 1.21.3 | 2021-07-10 | 2021-07-14 | | @@ -120,7 +120,7 @@ End of Life for **1.20** is **2022-02-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | | ------------- | -------------------- | ----------- | ----------------------------------------------------------------------------------- | -| 1.20.12 | 2021-10-15 | 2021-10-20 | | +| 1.20.12 | 2021-10-22 | 2021-10-27 | | | 1.20.11 | 2021-09-10 | 2021-09-15 | | | 1.20.10 | 2021-08-07 | 2021-08-11 | | | 1.20.9 | 2021-07-10 | 2021-07-14 | | @@ -141,6 +141,7 @@ End of Life for **1.19** is **2021-10-28** | PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE | NOTE | | ------------- | -------------------- | ----------- | ------------------------------------------------------------------------- | +| 1.19.16 | 2021-10-22 | 2021-09-27 | | | 1.19.15 | 2021-09-10 | 2021-09-15 | | | 1.19.14 | 2021-08-07 | 2021-08-11 | | | 1.19.13 | 2021-07-10 | 2021-07-14 | | From 710fa9288dc3ff4deb583fe9348e88807e6535da Mon Sep 17 00:00:00 2001 From: Jihoon Seo Date: Tue, 28 Sep 2021 17:24:20 +0900 Subject: [PATCH 133/249] [ko] Update outdated files in dev-1.22-ko.1 (p6) --- .../services-networking/dns-pod-service.md | 24 ++- .../ingress-controllers.md | 1 + .../concepts/services-networking/ingress.md | 4 +- .../services-networking/network-policies.md | 18 ++- .../concepts/services-networking/service.md | 55 +++++-- .../concepts/storage/persistent-volumes.md | 150 ++++++++++++++---- 6 files changed, 203 insertions(+), 49 deletions(-) diff --git a/content/ko/docs/concepts/services-networking/dns-pod-service.md b/content/ko/docs/concepts/services-networking/dns-pod-service.md index b405617118..dea7c35965 100644 --- a/content/ko/docs/concepts/services-networking/dns-pod-service.md +++ b/content/ko/docs/concepts/services-networking/dns-pod-service.md @@ -197,7 +197,7 @@ A 또는 AAAA 레코드만 생성할 수 있다. (`default-subdomain.my-namespac ### 파드의 setHostnameAsFQDN 필드 {#pod-sethostnameasfqdn-field} -{{< feature-state for_k8s_version="v1.20" state="beta" >}} +{{< feature-state for_k8s_version="v1.22" state="stable" >}} 파드가 전체 주소 도메인 이름(FQDN)을 갖도록 구성된 경우, 해당 호스트네임은 짧은 호스트네임이다. 예를 들어, 전체 주소 도메인 이름이 `busybox-1.default-subdomain.my-namespace.svc.cluster-domain.example` 인 파드가 있는 경우, 기본적으로 해당 파드 내부의 `hostname` 명령어는 `busybox-1` 을 반환하고 `hostname --fqdn` 명령은 FQDN을 반환한다. @@ -313,6 +313,28 @@ search default.svc.cluster-domain.example svc.cluster-domain.example cluster-dom options ndots:5 ``` +#### 확장된 DNS 환경 설정 + +{{< feature-state for_k8s_version="1.22" state="alpha" >}} + +쿠버네티스는 파드의 DNS 환경 설정을 위해 기본적으로 최대 6개의 탐색 도메인과 +최대 256자의 탐색 도메인 목록을 허용한다. + +kube-apiserver와 kubelet에 `ExpandedDNSConfig` 기능 게이트가 활성화되어 있으면, +쿠버네티스는 최대 32개의 탐색 도메인과 +최대 2048자의 탐색 도메인 목록을 허용한다. + +### 기능 가용성 + +파드 DNS 환경 설정 기능과 DNS 정책 "`None`" 기능의 쿠버네티스 버전별 가용성은 다음과 같다. + +| 쿠버네티스 버전 | 기능 지원 | +| :---------: |:-----------:| +| 1.14 | 안정 | +| 1.10 | 베타 (기본값으로 켜져 있음)| +| 1.9 | 알파 | + + ## {{% heading "whatsnext" %}} diff --git a/content/ko/docs/concepts/services-networking/ingress-controllers.md b/content/ko/docs/concepts/services-networking/ingress-controllers.md index 41524039f0..b43467d936 100644 --- a/content/ko/docs/concepts/services-networking/ingress-controllers.md +++ b/content/ko/docs/concepts/services-networking/ingress-controllers.md @@ -32,6 +32,7 @@ weight: 40 Citrix 애플리케이션 딜리버리 컨트롤러에서 작동한다. * [Contour](https://projectcontour.io/)는 [Envoy](https://www.envoyproxy.io/) 기반 인그레스 컨트롤러다. * [EnRoute](https://getenroute.io/)는 인그레스 컨트롤러로 실행할 수 있는 [Envoy](https://www.envoyproxy.io) 기반 API 게이트웨이다. +* [Easegress IngressController](https://github.com/megaease/easegress/blob/main/doc/ingresscontroller.md)는 인그레스 컨트롤러로 실행할 수 있는 [Easegress](https://megaease.com/easegress/) 기반 API 게이트웨이다. * F5 BIG-IP [쿠버네티스 용 컨테이너 인그레스 서비스](https://clouddocs.f5.com/containers/latest/userguide/kubernetes/)를 이용하면 인그레스를 사용하여 F5 BIG-IP 가상 서버를 구성할 수 있다. * [Gloo](https://gloo.solo.io)는 API 게이트웨이 기능을 제공하는 [Envoy](https://www.envoyproxy.io) 기반의 diff --git a/content/ko/docs/concepts/services-networking/ingress.md b/content/ko/docs/concepts/services-networking/ingress.md index 802cc486bf..805260d159 100644 --- a/content/ko/docs/concepts/services-networking/ingress.md +++ b/content/ko/docs/concepts/services-networking/ingress.md @@ -1,4 +1,6 @@ --- + + title: 인그레스(Ingress) content_type: concept weight: 40 @@ -222,7 +224,7 @@ IngressClass 리소스에는 선택적인 파라미터 필드가 있다. 이 클 #### 네임스페이스 범위의 파라미터 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} `Parameters` 필드에는 인그레스 클래스 구성을 위해 네임스페이스 별 리소스를 참조하는 데 사용할 수 있는 `scope` 및 `namespace` 필드가 있다. diff --git a/content/ko/docs/concepts/services-networking/network-policies.md b/content/ko/docs/concepts/services-networking/network-policies.md index 5a9b16a309..d8926be00c 100644 --- a/content/ko/docs/concepts/services-networking/network-policies.md +++ b/content/ko/docs/concepts/services-networking/network-policies.md @@ -223,7 +223,7 @@ SCTP 프로토콜 네트워크폴리시를 지원하는 {{< glossary_tooltip tex ## 포트 범위 지정 -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} 네트워크폴리시를 작성할 때, 단일 포트 대신 포트 범위를 대상으로 지정할 수 있다. @@ -251,17 +251,25 @@ spec: endPort: 32768 ``` -위 규칙은 대상 포트가 32000에서 32768 사이에 있는 경우, 네임스페이스 `default` 에 레이블이 `db` 인 모든 파드가 TCP를 통해 `10.0.0.0/24` 범위 내의 모든 IP와 통신하도록 허용한다. +위 규칙은 대상 포트가 32000에서 32768 사이에 있는 경우, +네임스페이스 `default` 에 레이블이 `db` 인 모든 파드가 +TCP를 통해 `10.0.0.0/24` 범위 내의 모든 IP와 통신하도록 허용한다. 이 필드를 사용할 때 다음의 제한 사항이 적용된다. -* 알파 기능으로, 기본적으로 비활성화되어 있다. 클러스터 수준에서 `endPort` 필드를 활성화하려면, 사용자(또는 클러스터 관리자)가 `--feature-gates=NetworkPolicyEndPort=true,…` 가 있는 API 서버에 대해 `NetworkPolicyEndPort` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 활성화해야 한다. +* 베타 기능으로, 기본적으로 활성화되어 있다. +클러스터 수준에서 `endPort` 필드를 비활성화하려면, 사용자(또는 클러스터 관리자)가 +API 서버에 대해 `--feature-gates=NetworkPolicyEndPort=false,…` 명령을 이용하여 +`NetworkPolicyEndPort` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 비활성화해야 한다. * `endPort` 필드는 `port` 필드보다 크거나 같아야 한다. * `endPort` 는 `port` 도 정의된 경우에만 정의할 수 있다. * 두 포트 모두 숫자여야 한다. {{< note >}} -클러스터는 {{< glossary_tooltip text="CNI" term_id="cni" >}} 플러그인을 사용해야 한다. -네트워크폴리시 명세에서 `endPort` 필드를 지원한다. +클러스터가 네트워크폴리시 명세의 `endPort` 필드를 지원하는 +{{< glossary_tooltip text="CNI" term_id="cni" >}} 플러그인을 사용해야 한다. +만약 [네트워크 플러그인](/ko/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)이 +`endPort` 필드를 지원하지 않는데 네트워크폴리시의 해당 필드에 명시를 하면, +그 정책은 `port` 필드에만 적용될 것이다. {{< /note >}} ## 이름으로 네임스페이스 지정 diff --git a/content/ko/docs/concepts/services-networking/service.md b/content/ko/docs/concepts/services-networking/service.md index 710e462a67..230b716c78 100644 --- a/content/ko/docs/concepts/services-networking/service.md +++ b/content/ko/docs/concepts/services-networking/service.md @@ -72,7 +72,7 @@ _서비스_ 로 들어가보자. 마찬가지로, 서비스 정의를 API 서버에 `POST`하여 새 인스턴스를 생성할 수 있다. 서비스 오브젝트의 이름은 유효한 -[DNS 서브도메인 이름](/ko/docs/concepts/overview/working-with-objects/names/#dns-서브도메인-이름)이어야 한다. +[RFC 1035 레이블 이름](/ko/docs/concepts/overview/working-with-objects/names/#rfc-1035-label-names)이어야 한다. 예를 들어, 각각 TCP 포트 9376에서 수신하고 `app=MyApp` 레이블을 가지고 있는 파드 세트가 있다고 가정해 보자. @@ -188,9 +188,10 @@ DNS명을 대신 사용하는 특수한 상황의 서비스이다. 자세한 내 이 문서 뒷부분의 [ExternalName](#externalname) 섹션을 참조한다. ### 초과 용량 엔드포인트 -엔드포인트 리소스에 1,000개가 넘는 엔드포인트가 있는 경우 쿠버네티스 v1.21(또는 그 이상) -클러스터는 해당 엔드포인트에 `endpoints.kubernetes.io/over-capacity: warning` 어노테이션을 추가한다. -이 어노테이션은 영향을 받는 엔드포인트 오브젝트가 용량을 초과했음을 나타낸다. +엔드포인트 리소스에 1,000개가 넘는 엔드포인트가 있는 경우 쿠버네티스 v1.22(또는 그 이상) +클러스터는 해당 엔드포인트에 `endpoints.kubernetes.io/over-capacity: truncated` 어노테이션을 추가한다. +이 어노테이션은 영향을 받는 엔드포인트 오브젝트가 용량을 초과했으며 +엔드포인트 컨트롤러가 엔드포인트의 수를 1000으로 줄였음을 나타낸다. ### 엔드포인트슬라이스 @@ -384,6 +385,40 @@ CIDR 범위 내의 유효한 IPv4 또는 IPv6 주소여야 한다. 유효하지 않은 clusterIP 주소 값으로 서비스를 생성하려고 하면, API 서버는 422 HTTP 상태 코드를 리턴하여 문제점이 있음을 알린다. +## 트래픽 정책 + +### 외부 트래픽 정책 + +`spec.externalTrafficPolicy` 필드를 설정하여 외부 소스에서 오는 트래픽이 어떻게 라우트될지를 제어할 수 있다. +이 필드는 `Cluster` 또는 `Local`로 설정할 수 있다. 필드를 `Cluster`로 설정하면 외부 트래픽을 준비 상태의 모든 엔드포인트로 라우트하며, +`Local`로 설정하면 준비 상태의 노드-로컬 엔드포인트로만 라우트한다. 만약 트래픽 정책이 `Local`로 설정되어 있는데 노드-로컬 +엔드포인트가 하나도 없는 경우, kube-proxy는 연관된 서비스로의 트래픽을 포워드하지 않는다. + +{{< note >}} +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} +kube-proxy에 대해 `ProxyTerminatingEndpoints` +[기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 +활성화하면, kube-proxy는 노드에 로컬 엔드포인트가 있는지, +그리고 모든 로컬 엔드포인트가 "종료 중(terminating)"으로 표시되어 있는지 여부를 확인한다. +만약 로컬 엔드포인트가 존재하는데 **모두**가 종료 중이면, kube-proxy는 `Local`로 설정된 모든 외부 트래픽 정책을 무시한다. +대신, 모든 노드-로컬 엔드포인트가 "종료 중" 상태를 유지하는 동안, +kube-proxy는 마치 외부 트래픽 정책이 `Cluster`로 설정되어 있는 것처럼 +그 서비스에 대한 트래픽을 정상 상태의 다른 엔드포인트로 포워드한다. +이러한 종료 중인 엔드포인트에 대한 포워딩 정책은 `NodePort` 서비스로 트래픽을 로드밸런싱하던 외부 로드밸런서가 +헬스 체크 노드 포트가 작동하지 않을 때에도 연결들을 비돌발적으로(gracefully) 종료시킬 수 있도록 하기 위해 존재한다. +이러한 정책이 없다면, 노드가 여전히 로드밸런서 노드 풀에 있지만 +파드 종료 과정에서 트래픽이 제거(drop)되는 상황에서 트래픽이 유실될 수 있다. +{{< /note >}} + +### 내부 트래픽 정책 + +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + +`spec.internalTrafficPolicy` 필드를 설정하여 내부 소스에서 오는 트래픽이 어떻게 라우트될지를 제어할 수 있다. +이 필드는 `Cluster` 또는 `Local`로 설정할 수 있다. 필드를 `Cluster`로 설정하면 내부 트래픽을 준비 상태의 모든 엔드포인트로 라우트하며, +`Local`로 설정하면 준비 상태의 노드-로컬 엔드포인트로만 라우트한다. 만약 트래픽 정책이 `Local`로 설정되어 있는데 노드-로컬 +엔드포인트가 하나도 없는 경우, kube-proxy는 트래픽을 포워드하지 않는다. + ## 서비스 디스커버리하기 쿠버네티스는 서비스를 찾는 두 가지 기본 모드를 지원한다. - 환경 @@ -523,6 +558,7 @@ API에서 `엔드포인트` 레코드를 생성하고, DNS 구성을 수정하 [kube-proxy 구성 파일](/docs/reference/config-api/kube-proxy-config.v1alpha1/)의 동등한 `nodePortAddresses` 필드를 특정 IP 블록으로 설정할 수 있다. + 이 플래그는 쉼표로 구분된 IP 블록 목록(예: `10.0.0.0/8`, `192.0.2.0/25`)을 사용하여 kube-proxy가 로컬 노드로 고려해야 하는 IP 주소 범위를 지정한다. 예를 들어, `--nodeport-addresses=127.0.0.0/8` 플래그로 kube-proxy를 시작하면, kube-proxy는 NodePort 서비스에 대하여 루프백(loopback) 인터페이스만 선택한다. `--nodeport-addresses`의 기본 값은 비어있는 목록이다. 이것은 kube-proxy가 NodePort에 대해 사용 가능한 모든 네트워크 인터페이스를 고려해야 한다는 것을 의미한다. (이는 이전 쿠버네티스 릴리스와도 호환된다). @@ -641,12 +677,12 @@ v1.20부터는 `spec.allocateLoadBalancerNodePorts` 필드를 `false`로 설정 #### 로드 밸런서 구현 클래스 지정 {#load-balancer-class} -{{< feature-state for_k8s_version="v1.21" state="alpha" >}} +{{< feature-state for_k8s_version="v1.22" state="beta" >}} -v1.21부터는, `spec.loadBalancerClass` 필드를 설정하여 `LoadBalancer` 서비스 유형에 -대한 로드 밸런서 구현 클래스를 선택적으로 지정할 수 있다. -기본적으로, `spec.loadBalancerClass` 는 `nil` 이고 `LoadBalancer` 유형의 서비스는 -클라우드 공급자의 기본 로드 밸런서 구현을 사용한다. +`spec.loadBalancerClass` 필드를 설정하여 클라우드 제공자가 설정한 기본값 이외의 로드 밸런서 구현을 사용할 수 있다. 이 기능은 v1.21부터 사용할 수 있으며, v1.21에서는 이 필드를 사용하기 위해 `ServiceLoadBalancerClass` 기능 게이트를 활성화해야 하지만, v1.22부터는 해당 기능 게이트가 기본적으로 활성화되어 있다. +기본적으로, `spec.loadBalancerClass` 는 `nil` 이고, +클러스터가 클라우드 제공자의 로드밸런서를 이용하도록 `--cloud-provider` 컴포넌트 플래그를 이용하여 설정되어 있으면 +`LoadBalancer` 유형의 서비스는 클라우드 공급자의 기본 로드 밸런서 구현을 사용한다. `spec.loadBalancerClass` 가 지정되면, 지정된 클래스와 일치하는 로드 밸런서 구현이 서비스를 감시하고 있다고 가정한다. 모든 기본 로드 밸런서 구현(예: 클라우드 공급자가 제공하는 @@ -656,7 +692,6 @@ v1.21부터는, `spec.loadBalancerClass` 필드를 설정하여 `LoadBalancer` `spec.loadBalancerClass` 의 값은 "`internal-vip`" 또는 "`example.com/internal-vip`" 와 같은 선택적 접두사가 있는 레이블 스타일 식별자여야 한다. 접두사가 없는 이름은 최종 사용자를 위해 예약되어 있다. -이 필드를 사용하려면 `ServiceLoadBalancerClass` 기능 게이트를 활성화해야 한다. #### 내부 로드 밸런서 diff --git a/content/ko/docs/concepts/storage/persistent-volumes.md b/content/ko/docs/concepts/storage/persistent-volumes.md index c70b7413ad..87d43ae09e 100644 --- a/content/ko/docs/concepts/storage/persistent-volumes.md +++ b/content/ko/docs/concepts/storage/persistent-volumes.md @@ -314,12 +314,9 @@ EBS 볼륨 확장은 시간이 많이 걸리는 작업이다. 또한 6시간마 * [`azureDisk`](/ko/docs/concepts/storage/volumes/#azuredisk) - Azure Disk * [`azureFile`](/ko/docs/concepts/storage/volumes/#azurefile) - Azure File * [`cephfs`](/ko/docs/concepts/storage/volumes/#cephfs) - CephFS 볼륨 -* [`cinder`](/ko/docs/concepts/storage/volumes/#cinder) - Cinder (오픈스택 블록 스토리지) - (**사용 중단**) * [`csi`](/ko/docs/concepts/storage/volumes/#csi) - 컨테이너 스토리지 인터페이스 (CSI) * [`fc`](/ko/docs/concepts/storage/volumes/#fc) - Fibre Channel (FC) 스토리지 * [`flexVolume`](/ko/docs/concepts/storage/volumes/#flexVolume) - FlexVolume -* [`flocker`](/ko/docs/concepts/storage/volumes/#flocker) - Flocker 스토리지 * [`gcePersistentDisk`](/ko/docs/concepts/storage/volumes/#gcepersistentdisk) - GCE Persistent Disk * [`glusterfs`](/ko/docs/concepts/storage/volumes/#glusterfs) - Glusterfs 볼륨 * [`hostPath`](/ko/docs/concepts/storage/volumes/#hostpath) - HostPath 볼륨 @@ -329,17 +326,28 @@ EBS 볼륨 확장은 시간이 많이 걸리는 작업이다. 또한 6시간마 * [`local`](/ko/docs/concepts/storage/volumes/#local) - 노드에 마운트된 로컬 스토리지 디바이스 * [`nfs`](/ko/docs/concepts/storage/volumes/#nfs) - 네트워크 파일 시스템 (NFS) 스토리지 -* `photonPersistentDisk` - Photon 컨트롤러 퍼시스턴트 디스크. - (이 볼륨 유형은 해당 클라우드 공급자가 없어진 이후 더 이상 - 작동하지 않는다.) * [`portworxVolume`](/ko/docs/concepts/storage/volumes/#portworxvolume) - Portworx 볼륨 -* [`quobyte`](/ko/docs/concepts/storage/volumes/#quobyte) - Quobyte 볼륨 * [`rbd`](/ko/docs/concepts/storage/volumes/#rbd) - Rados Block Device (RBD) 볼륨 -* [`scaleIO`](/ko/docs/concepts/storage/volumes/#scaleio) - ScaleIO 볼륨 - (**사용 중단**) -* [`storageos`](/ko/docs/concepts/storage/volumes/#storageos) - StorageOS 볼륨 * [`vsphereVolume`](/ko/docs/concepts/storage/volumes/#vspherevolume) - vSphere VMDK 볼륨 +아래의 PersistentVolume 타입은 사용 중단되었다. 이 말인 즉슨, 지원은 여전히 제공되지만 추후 쿠버네티스 릴리스에서는 삭제될 예정이라는 것이다. + +* [`cinder`](/ko/docs/concepts/storage/volumes/#cinder) - Cinder (오픈스택 블록 스토리지) + (v1.18에서 **사용 중단**) +* [`flocker`](/ko/docs/concepts/storage/volumes/#flocker) - Flocker 스토리지 + (v1.22에서 **사용 중단**) +* [`quobyte`](/ko/docs/concepts/storage/volumes/#quobyte) - Quobyte 볼륨 + (v1.22에서 **사용 중단**) +* [`storageos`](/ko/docs/concepts/storage/volumes/#storageos) - StorageOS 볼륨 + (v1.22에서 **사용 중단**) + +이전 쿠버네티스 버전은 아래의 인-트리 PersistentVolume 타입도 지원했었다. + +* `photonPersistentDisk` - Photon 컨트롤러 퍼시스턴트 디스크. + (v1.15 이후 **사용 불가**) +* [`scaleIO`](/ko/docs/concepts/storage/volumes/#scaleio) - ScaleIO 볼륨 + (v1.21 이후 **사용 불가**) + ## 퍼시스턴트 볼륨 각 PV에는 스펙과 상태(볼륨의 명세와 상태)가 포함된다. @@ -407,38 +415,40 @@ spec: * ReadWriteOnce -- 하나의 노드에서 볼륨을 읽기-쓰기로 마운트할 수 있다 * ReadOnlyMany -- 여러 노드에서 볼륨을 읽기 전용으로 마운트할 수 있다 * ReadWriteMany -- 여러 노드에서 볼륨을 읽기-쓰기로 마운트할 수 있다 +* ReadWriteOncePod -- 하나의 파드에서 볼륨을 읽기-쓰기로 마운트할 수 있다. + 쿠버네티스 버전 1.22 이상인 경우에 CSI 볼륨에 대해서만 지원된다. CLI에서 접근 모드는 다음과 같이 약어로 표시된다. * RWO - ReadWriteOnce * ROX - ReadOnlyMany * RWX - ReadWriteMany +* RWOP - ReadWriteOncePod > __중요!__ 볼륨이 여러 접근 모드를 지원하더라도 한 번에 하나의 접근 모드를 사용하여 마운트할 수 있다. 예를 들어 GCEPersistentDisk는 하나의 노드가 ReadWriteOnce로 마운트하거나 여러 노드가 ReadOnlyMany로 마운트할 수 있지만 동시에는 불가능하다. -| Volume Plugin | ReadWriteOnce | ReadOnlyMany | ReadWriteMany| -| :--- | :---: | :---: | :---: | -| AWSElasticBlockStore | ✓ | - | - | -| AzureFile | ✓ | ✓ | ✓ | -| AzureDisk | ✓ | - | - | -| CephFS | ✓ | ✓ | ✓ | -| Cinder | ✓ | - | - | -| CSI | 드라이버에 따라 다름 | 드라이버에 따라 다름 | 드라이버에 따라 다름 | -| FC | ✓ | ✓ | - | -| FlexVolume | ✓ | ✓ | 드라이버에 따라 다름 | -| Flocker | ✓ | - | - | -| GCEPersistentDisk | ✓ | ✓ | - | -| Glusterfs | ✓ | ✓ | ✓ | -| HostPath | ✓ | - | - | -| iSCSI | ✓ | ✓ | - | -| Quobyte | ✓ | ✓ | ✓ | -| NFS | ✓ | ✓ | ✓ | -| RBD | ✓ | ✓ | - | -| VsphereVolume | ✓ | - | - (파드가 병치될(collocated) 때 작동) | -| PortworxVolume | ✓ | - | ✓ | -| ScaleIO | ✓ | ✓ | - | -| StorageOS | ✓ | - | - | +| Volume Plugin | ReadWriteOnce | ReadOnlyMany | ReadWriteMany | ReadWriteOncePod | +| :--- | :---: | :---: | :---: | - | +| AWSElasticBlockStore | ✓ | - | - | - | +| AzureFile | ✓ | ✓ | ✓ | - | +| AzureDisk | ✓ | - | - | - | +| CephFS | ✓ | ✓ | ✓ | - | +| Cinder | ✓ | - | - | - | +| CSI | depends on the driver | depends on the driver | depends on the driver | depends on the driver | +| FC | ✓ | ✓ | - | - | +| FlexVolume | ✓ | ✓ | depends on the driver | - | +| Flocker | ✓ | - | - | - | +| GCEPersistentDisk | ✓ | ✓ | - | - | +| Glusterfs | ✓ | ✓ | ✓ | - | +| HostPath | ✓ | - | - | - | +| iSCSI | ✓ | ✓ | - | - | +| Quobyte | ✓ | ✓ | ✓ | - | +| NFS | ✓ | ✓ | ✓ | - | +| RBD | ✓ | ✓ | - | - | +| VsphereVolume | ✓ | - | - (works when Pods are collocated) | - | +| PortworxVolume | ✓ | - | ✓ | - | - | +| StorageOS | ✓ | - | - | - | ### 클래스 @@ -785,6 +795,82 @@ spec: storage: 10Gi ``` +## 볼륨 파퓰레이터(Volume populator)와 데이터 소스 + +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + +{{< note >}} +쿠버네티스는 커스텀 볼륨 파퓰레이터를 지원한다. +이 알파 기능은 쿠버네티스 1.18에서 도입되었으며 +1.22에서는 새로운 메카니즘과 리디자인된 API로 새롭게 구현되었다. +현재 사용 중인 클러스터의 버전에 맞는 쿠버네티스 문서를 읽고 있는지 다시 한번 +확인한다. {{% version-check %}} +커스텀 볼륨 파퓰레이터를 사용하려면, kube-apiserver와 kube-controller-manager에 대해 +`AnyVolumeDataSource` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 활성화해야 한다. +{{< /note >}} + +볼륨 파퓰레이터는 `dataSourceRef`라는 PVC 스펙 필드를 활용한다. +다른 PersistentVolumeClaim 또는 VolumeSnapshot을 가리키는 참조만 명시할 수 있는 +`dataSource` 필드와는 다르게, `dataSourceRef` 필드는 동일 네임스페이스에 있는 +어떠한 오브젝트에 대한 참조도 명시할 수 있다(단, PVC 외의 다른 코어 오브젝트는 제외). +기능 게이트가 활성화된 클러스터에서는 `dataSource`보다 `dataSourceRef`를 사용하는 것을 권장한다. + +## 데이터 소스 참조 + +`dataSourceRef` 필드는 `dataSource` 필드와 거의 동일하게 동작한다. +둘 중 하나만 명시되어 있으면, API 서버는 두 필드에 같은 값을 할당할 것이다. +두 필드 모두 생성 이후에는 변경될 수 없으며, +두 필드에 다른 값을 넣으려고 시도하면 검증 에러가 발생할 것이다. +따라서 두 필드는 항상 같은 값을 갖게 된다. + +`dataSourceRef` 필드와 `dataSource` 필드 사이에는 +사용자가 알고 있어야 할 두 가지 차이점이 있다. +* `dataSource` 필드는 유효하지 않은 값(예를 들면, 빈 값)을 무시하지만, + `dataSourceRef` 필드는 어떠한 값도 무시하지 않으며 유효하지 않은 값이 들어오면 에러를 발생할 것이다. + 유효하지 않은 값은 PVC를 제외한 모든 코어 오브젝트(apiGroup이 없는 오브젝트)이다. +* `dataSourceRef` 필드는 여러 타입의 오브젝트를 포함할 수 있지만, `dataSource` 필드는 + PVC와 VolumeSnapshot만 포함할 수 있다. + +기능 게이트가 활성화된 클러스터에서는 `dataSourceRef`를 사용해야 하고, 그렇지 않은 +클러스터에서는 `dataSource`를 사용해야 한다. 어떤 경우에서든 두 필드 모두를 확인해야 +할 필요는 없다. 이렇게 약간의 차이만 있는 중복된 값은 이전 버전 호환성을 위해서만 +존재하는 것이다. 상세히 설명하면, 이전 버전과 새로운 버전의 컨트롤러가 함께 동작할 +수 있는데, 이는 두 필드가 동일하기 때문이다. + +### 볼륨 파퓰레이터 사용하기 + +볼륨 파퓰레이터는 비어 있지 않은 볼륨(non-empty volume)을 생성할 수 있는 {{< glossary_tooltip text="컨트롤러" term_id="controller" >}}이며, +이 볼륨의 내용물은 커스텀 리소스(Custom Resource)에 의해 결정된다. +파퓰레이티드 볼륨(populated volume)을 생성하려면 `dataSourceRef` 필드에 커스텀 리소스를 기재한다. + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: populated-pvc +spec: + dataSourceRef: + name: example-name + kind: ExampleDataSource + apiGroup: example.storage.k8s.io + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + +볼륨 파퓰레이터는 외부 컴포넌트이기 때문에, +만약 적합한 컴포넌트가 설치되어 있지 않다면 볼륨 파퓰레이터를 사용하는 PVC에 대한 생성 요청이 실패할 수 있다. +외부 컨트롤러는 '컴포넌트가 없어서 PVC를 생성할 수 없음' 경고와 같은 +PVC 생성 상태에 대한 피드백을 제공하기 위해, PVC에 대한 이벤트를 생성해야 한다. + +알파 버전의 [볼륨 데이터 소스 검증기](https://github.com/kubernetes-csi/volume-data-source-validator)를 +클러스터에 설치할 수 있다. +해당 데이터 소스를 다루는 파퓰레이터가 등록되어 있지 않다면 이 컨트롤러가 PVC에 경고 이벤트를 생성한다. +PVC를 위한 적절한 파퓰레이터가 설치되어 있다면, +볼륨 생성과 그 과정에서 발생하는 이슈에 대한 이벤트를 생성하는 것은 파퓰레이터 컨트롤러의 몫이다. + ## 포터블 구성 작성 광범위한 클러스터에서 실행되고 퍼시스턴트 스토리지가 필요한 From 7994d24f5eb0d23b205cbc03586a03c0cff40d38 Mon Sep 17 00:00:00 2001 From: Carlos Panato Date: Wed, 6 Oct 2021 09:55:33 +0200 Subject: [PATCH 134/249] releng: update patch release dates for 1.19 Signed-off-by: Carlos Panato --- data/releases/schedule.yaml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/data/releases/schedule.yaml b/data/releases/schedule.yaml index 2393235733..dbbd69d999 100644 --- a/data/releases/schedule.yaml +++ b/data/releases/schedule.yaml @@ -76,11 +76,14 @@ schedules: targetDate: 2020-12-18 note: "Tagging Issue https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA" - release: 1.19 - next: 1.19.15 - cherryPickDeadline: 2021-09-10 - targetDate: 2021-09-15 + next: 1.19.16 + cherryPickDeadline: 2021-10-22 + targetDate: 2021-10-27 endOfLifeDate: 2021-10-28 previousPatches: + - release: 1.19.15 + cherryPickDeadline: 2021-09-10 + targetDate: 2021-09-15 - release: 1.19.14 cherryPickDeadline: 2021-08-07 targetDate: 2021-08-11 From 98834b63663e08ba6c50f074ee921fc3ff0edab3 Mon Sep 17 00:00:00 2001 From: Maciej Filocha Date: Mon, 4 Oct 2021 10:19:01 +0200 Subject: [PATCH 135/249] Synchronize Polish localization for ver 1.22, part 3 Synchronize Polish localization with upstream up to 08d92f9137924abdf442e7d9fb7372901379b993. Part 3. --- content/pl/docs/reference/_index.md | 52 +++++++++++++++---- content/pl/docs/reference/glossary/index.md | 2 +- .../docs/reference/glossary/kube-scheduler.md | 2 +- 3 files changed, 44 insertions(+), 12 deletions(-) diff --git a/content/pl/docs/reference/_index.md b/content/pl/docs/reference/_index.md index fd4ab2bff3..8f97bfec60 100644 --- a/content/pl/docs/reference/_index.md +++ b/content/pl/docs/reference/_index.md @@ -14,36 +14,68 @@ Tutaj znajdziesz dokumentację źródłową Kubernetesa. ## Dokumentacja API +* [Glossary](/docs/reference/glossary/) - Pełna, zestandaryzowana lista terminologii Kubernetesa + * [Kubernetes API Reference](/docs/reference/kubernetes-api/) * [One-page API Reference for Kubernetes {{< param "version" >}}](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/) -* [Using The Kubernetes API](/docs/reference/using-api/) - ogólne informacje na temat API Kubernetesa. +* [Using The Kubernetes API](/docs/reference/using-api/) - ogólne informacje na temat API Kubernetesa +* [API access control](/docs/reference/access-authn-authz/) - szczegóły dotyczące kontroli dostępu do API Kubernetesa +* [Well-Known Labels, Annotations and Taints](/docs/reference/labels-annotations-taints/) -## Biblioteki klientów API +## Oficjalnie wspierane biblioteki klienckie Aby wywołać Kubernetes API z wybranego języka programowania, możesz skorzystać z [bibliotek klienckich](/docs/reference/using-api/client-libraries/). Oficjalnie wspierane biblioteki to: -* [Kubernetes Go client library](https://github.com/kubernetes/client-go/) * [Kubernetes Python client library](https://github.com/kubernetes-client/python) * [Kubernetes Java client library](https://github.com/kubernetes-client/java) * [Kubernetes JavaScript client library](https://github.com/kubernetes-client/javascript) +* [Kubernetes C# client library](https://github.com/kubernetes-client/csharp) +* [Kubernetes Haskell client library](https://github.com/kubernetes-client/haskell) -## Dokumentacja poleceń tekstowych *(CLI)* +## Polecenia tekstowe *(CLI)* * [kubectl](/docs/reference/kubectl/overview/) - Główne narzędzie tekstowe (linii poleceń) do zarządzania klastrem Kubernetes. * [JSONPath](/docs/reference/kubectl/jsonpath/) - Podręcznik składni [wyrażeń JSONPath](https://goessner.net/articles/JsonPath/) dla kubectl. * [kubeadm](/docs/reference/setup-tools/kubeadm/) - Narzędzie tekstowe do łatwego budowania klastra Kubernetes spełniającego niezbędne wymogi bezpieczeństwa. -## Dokumentacja komponentów +## Komponenty -* [kubelet](/docs/reference/command-line-tools-reference/kubelet/) - Główny agent działający na każdym węźle. Kubelet pobiera zestaw definicji PodSpecs i gwarantuje, że opisane przez nie kontenery poprawnie działają. -* [kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/) - REST API, które sprawdza poprawność i konfiguruje obiekty API, takie jak pody, serwisy czy kontrolery replikacji. +* [kubelet](/docs/reference/command-line-tools-reference/kubelet/) - Główny + agent działający na każdym węźle. Kubelet pobiera zestaw definicji PodSpecs + i gwarantuje, że opisane przez nie kontenery poprawnie działają. +* [kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/) - + REST API, które sprawdza poprawność i konfiguruje obiekty API, takie jak pody, serwisy czy kontrolery replikacji. * [kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/) - Proces wykonujący główne pętle sterowania Kubernetes. -* [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/) - Przekazuje bezpośrednio dane przepływające w transmisji TCP/UDP lub dystrybuuje ruch TCP/UDP zgodnie ze schematem *round-robin* pomiędzy usługi back-endu. +* [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/) - Przekazuje + bezpośrednio dane przepływające w transmisji TCP/UDP lub dystrybuuje ruch TCP/UDP + zgodnie ze schematem *round-robin* pomiędzy usługi back-endu. * [kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/) - Scheduler odpowiada za dostępność, wydajność i zasoby. -* [kube-scheduler Policies](/docs/reference/scheduling/policies) -* [kube-scheduler Profiles](/docs/reference/scheduling/config#profiles) +* [Scheduler Policies](/docs/reference/scheduling/policies) +* [Scheduler Profiles](/docs/reference/scheduling/config#profiles) + +## API konfiguracji + +W tej części zebrano "niepublikowane" API, które służą do konfiguracji komponentów +Kubernetesa lub innych narzędzi. Choć większość tych API nie jest udostępniane przez +serwer API w trybie RESTful, są one niezbędne dla użytkowników i administratorów +w korzystaniu i zarządzaniu klastrem. + +* [kube-apiserver configuration (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/) +* [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) +* [kube-scheduler configuration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) +* [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) +* [kube-scheduler policy reference (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) +* [kube-proxy configuration (v1alpha1)](/docs/reference/config-api/kube-proxy-config.v1alpha1/) +* [`audit.k8s.io/v1` API](/docs/reference/config-api/apiserver-audit.v1/) +* [Client authentication API (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) +* [WebhookAdmission configuration (v1)](/docs/reference/config-api/apiserver-webhookadmission.v1/) + +## API konfiguracji dla kubeadm + +* [v1beta2](/docs/reference/config-api/kubeadm-config.v1beta2/) +* [v1beta3](/docs/reference/config-api/kubeadm-config.v1beta3/) ## Dokumentacja projektowa diff --git a/content/pl/docs/reference/glossary/index.md b/content/pl/docs/reference/glossary/index.md index 1cdf5a1a3e..642bdbdfb2 100644 --- a/content/pl/docs/reference/glossary/index.md +++ b/content/pl/docs/reference/glossary/index.md @@ -1,5 +1,5 @@ --- -title: Ujednolicony słownik +title: Słownik layout: glossary noedit: true default_active_tag: fundamental diff --git a/content/pl/docs/reference/glossary/kube-scheduler.md b/content/pl/docs/reference/glossary/kube-scheduler.md index 4bbcc99a0f..f3f5e02d10 100644 --- a/content/pl/docs/reference/glossary/kube-scheduler.md +++ b/content/pl/docs/reference/glossary/kube-scheduler.md @@ -2,7 +2,7 @@ title: kube-scheduler id: kube-scheduler date: 2018-04-12 -full_link: /docs/reference/generated/kube-scheduler/ +full_link: /docs/reference/command-line-tools-reference/kube-scheduler/ short_description: > Składnik warstwy sterowania, który śledzi tworzenie nowych podów i przypisuje im węzły, na których powinny zostać uruchomione. From f268125675ebe01f1ed441cec3cbd5ef481e4b0d Mon Sep 17 00:00:00 2001 From: fabriziopandini Date: Wed, 6 Oct 2021 23:13:42 +0200 Subject: [PATCH 136/249] align ClusterClass blog to CAPI book tutorial --- ...-08-clusterclass-and-managed-topologies.md | 104 +++++++++--------- 1 file changed, 51 insertions(+), 53 deletions(-) diff --git a/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md index af6349ecb8..cd65d69f8b 100644 --- a/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md +++ b/content/en/blog/_posts/2021-10-08-clusterclass-and-managed-topologies.md @@ -36,52 +36,52 @@ ClusterClass, at its heart, is a collection of Cluster and Machine templates. Yo ```yaml --- apiVersion: cluster.x-k8s.io/v1beta1 - kind: ClusterClass - metadata: - name: my-amazing-cluster-class - namespace: bar - spec: - controlPlane: - ref: - apiVersion: controlplane.cluster.x-k8s.io/v1beta1 - kind: KubeadmControlPlaneTemplate - name: high-availability-kcp - machineInfrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 - kind: VSphereMachineTemplate - name: controlplane-vsphere-machinetemplate - workers: - deployments: - - class: linux-worker - template: - bootstrap: - ref: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfigTemplate - name: linux-bootstrap - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 - kind: VSphereMachineTemplate - name: linux-vsphere-template - - class: windows-worker - template: - bootstrap: - ref: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfigTemplate - name: windows-bootstrap - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 - kind: VSphereMachineTemplate - name: windows-vsphere-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 - kind: VSphereClusterTemplate - name: vsphere-cluster +kind: ClusterClass +metadata: + name: my-amazing-cluster-class +spec: + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlaneTemplate + name: high-availability-control-plane + machineInfrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachineTemplate + name: control-plane-machine + workers: + machineDeployments: + - class: type1-workers + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: type1-bootstrap + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachineTemplate + name: type1-machine + - class: type2-workers + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: type2-bootstrap + infrastructure: + ref: + kind: DockerMachineTemplate + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + name: type2-machine + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerClusterTemplate + name: cluster-infrastructure + ``` The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios. @@ -111,15 +111,12 @@ apiVersion: cluster.x-k8s.io/v1beta1 replicas: 3 workers: machineDeployments: - - class: linux-worker - name: big-pool-of-linux-machines-1 + - class: type1-workers + name: big-pool-of-machines replicas: 5 - - class: linux-worker - name: small-pool-of-linux-machines-1 + - class: type2-workers + name: small-pool-of-machines replicas: 1 - - class: windows-worker - name: pool-of-windows-machines - replicas: 3 ``` But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology. @@ -141,3 +138,4 @@ Stay tuned for what comes next, and if you have any questions, comments or sugge * Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T) * Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents * Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time +* Check out the [ClusterClass tutorial](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-classes.html) in the Cluster API book. \ No newline at end of file From f57f004bd05e58eaa2f7c03dff0981dbb613120d Mon Sep 17 00:00:00 2001 From: Arhell Date: Thu, 7 Oct 2021 00:26:31 +0300 Subject: [PATCH 137/249] [ru] Add seccomp tutorial to index --- content/ru/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/ru/docs/tutorials/_index.md b/content/ru/docs/tutorials/_index.md index e251975329..6cbe48cbe4 100644 --- a/content/ru/docs/tutorials/_index.md +++ b/content/ru/docs/tutorials/_index.md @@ -47,6 +47,8 @@ content_type: concept * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Сервисы * [Использование IP](/docs/tutorials/services/source-ip/) From 78f125c2225e44a732bb943b5fe1dd90a007ac95 Mon Sep 17 00:00:00 2001 From: RA489 Date: Thu, 7 Oct 2021 11:24:04 +0530 Subject: [PATCH 138/249] Improvement for other tools --- content/en/docs/reference/tools/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/tools/_index.md b/content/en/docs/reference/tools/_index.md index f719515af7..389e6e77d2 100644 --- a/content/en/docs/reference/tools/_index.md +++ b/content/en/docs/reference/tools/_index.md @@ -23,8 +23,8 @@ development and testing purposes. [`Dashboard`](/docs/tasks/access-application-cluster/web-ui-dashboard/), the web-based user interface of Kubernetes, allows you to deploy containerized applications to a Kubernetes cluster, troubleshoot them, and manage the cluster and its resources itself. -{{% thirdparty-content %}} ## Helm +{{% thirdparty-content single="true" %}} [Helm](https://helm.sh/) is a tool for managing packages of pre-configured Kubernetes resources. These packages are known as _Helm charts_. From cb9e9352a075e78b720414bd00352ee062c4b4cc Mon Sep 17 00:00:00 2001 From: Pulkit Singh Date: Thu, 7 Oct 2021 12:44:21 +0530 Subject: [PATCH 139/249] Update intro-windows-in-kubernetes.md --- .../windows/intro-windows-in-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md index 7b8e24db60..fa8733bbb7 100644 --- a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md @@ -962,4 +962,4 @@ For a detailed explanation of Windows distribution channels see the [Microsoft d Information on the different Windows Server servicing channels including their support models can be found at -[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). +[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison). From f37e6269ad90141b7c2d91fc1aedacd5f59f3b56 Mon Sep 17 00:00:00 2001 From: Karthikeya Viswanath Date: Wed, 6 Oct 2021 09:54:06 +0100 Subject: [PATCH 140/249] docs: add nlb documentation page reference for aws --- content/en/docs/concepts/services-networking/service.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index ff30adf088..eee679bf8c 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -1072,6 +1072,9 @@ in those modified security groups. {{< /note >}} +Further documentation on annotations for Elastic IPs and other common use-cases may be found +on the [AWS Load Balancer Controller documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/). + #### Other CLB annotations on Tencent Kubernetes Engine (TKE) There are other annotations for managing Cloud Load Balancers on TKE as shown below. From da31405d7b06fbd1905fecb799a4028da103e6ba Mon Sep 17 00:00:00 2001 From: Juhee Kang Date: Tue, 28 Sep 2021 11:09:20 +0900 Subject: [PATCH 141/249] [ko] Update outdated files in dev-1.22-ko.1 (p8) --- .../feature-gates.md | 285 +++++++++++++----- 1 file changed, 217 insertions(+), 68 deletions(-) diff --git a/content/ko/docs/reference/command-line-tools-reference/feature-gates.md b/content/ko/docs/reference/command-line-tools-reference/feature-gates.md index 97262073a1..7b1bdb7c42 100644 --- a/content/ko/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/ko/docs/reference/command-line-tools-reference/feature-gates.md @@ -2,6 +2,9 @@ weight: 10 title: 기능 게이트 content_type: concept +card: + name: reference + weight: 60 --- @@ -25,7 +28,7 @@ content_type: concept kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 쌍 목록에 지정된 `--feature-gates` 플래그를 사용한다. ```shell ---feature-gates="...,DynamicKubeletConfig=true" +--feature-gates="...,GracefulNodeShutdown=true" ``` 다음 표는 다른 쿠버네티스 컴포넌트에서 설정할 수 있는 기능 게이트를 @@ -55,65 +58,60 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `APIResponseCompression` | `false` | 알파 | 1.7 | 1.15 | | `APIResponseCompression` | `true` | 베타 | 1.16 | | | `APIServerIdentity` | `false` | 알파 | 1.20 | | +| `APIServerTracing` | `false` | 알파 | 1.22 | | | `AllowInsecureBackendProxy` | `true` | 베타 | 1.17 | | | `AnyVolumeDataSource` | `false` | 알파 | 1.18 | | | `AppArmor` | `true` | 베타 | 1.4 | | -| `BalanceAttachedNodeVolumes` | `false` | 알파 | 1.11 | | -| `BoundServiceAccountTokenVolume` | `false` | 알파 | 1.13 | 1.20 | -| `BoundServiceAccountTokenVolume` | `true` | 베타 | 1.21 | | | `ControllerManagerLeaderMigration` | `false` | 알파 | 1.21 | | | `CPUManager` | `false` | 알파 | 1.8 | 1.9 | | `CPUManager` | `true` | 베타 | 1.10 | | +| `CPUManagerPolicyOptions` | `false` | 알파 | 1.22 | | | `CSIInlineVolume` | `false` | 알파 | 1.15 | 1.15 | | `CSIInlineVolume` | `true` | 베타 | 1.16 | - | | `CSIMigration` | `false` | 알파 | 1.14 | 1.16 | | `CSIMigration` | `true` | 베타 | 1.17 | | | `CSIMigrationAWS` | `false` | 알파 | 1.14 | | | `CSIMigrationAWS` | `false` | 베타 | 1.17 | | -| `CSIMigrationAWSComplete` | `false` | 알파 | 1.17 | | | `CSIMigrationAzureDisk` | `false` | 알파 | 1.15 | 1.18 | | `CSIMigrationAzureDisk` | `false` | 베타 | 1.19 | | -| `CSIMigrationAzureDiskComplete` | `false` | 알파 | 1.17 | | | `CSIMigrationAzureFile` | `false` | 알파 | 1.15 | 1.19 | | `CSIMigrationAzureFile` | `false` | 베타 | 1.21 | | -| `CSIMigrationAzureFileComplete` | `false` | 알파 | 1.17 | | | `CSIMigrationGCE` | `false` | 알파 | 1.14 | 1.16 | | `CSIMigrationGCE` | `false` | 베타 | 1.17 | | -| `CSIMigrationGCEComplete` | `false` | 알파 | 1.17 | | | `CSIMigrationOpenStack` | `false` | 알파 | 1.14 | 1.17 | | `CSIMigrationOpenStack` | `true` | 베타 | 1.18 | | -| `CSIMigrationOpenStackComplete` | `false` | 알파 | 1.17 | | | `CSIMigrationvSphere` | `false` | 베타 | 1.19 | | -| `CSIMigrationvSphereComplete` | `false` | 베타 | 1.19 | | -| `CSIServiceAccountToken` | `false` | 알파 | 1.20 | 1.20 | -| `CSIServiceAccountToken` | `true` | 베타 | 1.21 | | | `CSIStorageCapacity` | `false` | 알파 | 1.19 | 1.20 | | `CSIStorageCapacity` | `true` | 베타 | 1.21 | | | `CSIVolumeFSGroupPolicy` | `false` | 알파 | 1.19 | 1.19 | | `CSIVolumeFSGroupPolicy` | `true` | 베타 | 1.20 | | | `CSIVolumeHealth` | `false` | 알파 | 1.21 | | +| `CSRDuration` | `true` | 베타 | 1.22 | | | `ConfigurableFSGroupPolicy` | `false` | 알파 | 1.18 | 1.19 | | `ConfigurableFSGroupPolicy` | `true` | 베타 | 1.20 | | -| `CronJobControllerV2` | `false` | 알파 | 1.20 | 1.20 | -| `CronJobControllerV2` | `true` | 베타 | 1.21 | | +| `ControllerManagerLeaderMigration` | `false` | 알파 | 1.21 | 1.21 | +| `ControllerManagerLeaderMigration` | `true` | 베타 | 1.22 | | | `CustomCPUCFSQuotaPeriod` | `false` | 알파 | 1.12 | | +| `DaemonSetUpdateSurge` | `false` | 알파 | 1.21 | 1.21 | +| `DaemonSetUpdateSurge` | `true` | 베타 | 1.22 | | | `DefaultPodTopologySpread` | `false` | 알파 | 1.19 | 1.19 | | `DefaultPodTopologySpread` | `true` | 베타 | 1.20 | | +| `DelegateFSGroupToCSIDriver` | `false` | 알파 | 1.22 | | | `DevicePlugins` | `false` | 알파 | 1.8 | 1.9 | | `DevicePlugins` | `true` | 베타 | 1.10 | | | `DisableAcceleratorUsageMetrics` | `false` | 알파 | 1.19 | 1.19 | | `DisableAcceleratorUsageMetrics` | `true` | 베타 | 1.20 | | +| `DisableCloudProviders` | `false` | 알파 | 1.22 | | | `DownwardAPIHugePages` | `false` | 알파 | 1.20 | 1.20 | | `DownwardAPIHugePages` | `false` | 베타 | 1.21 | | -| `DynamicKubeletConfig` | `false` | 알파 | 1.4 | 1.10 | -| `DynamicKubeletConfig` | `true` | 베타 | 1.11 | | -| `EfficientWatchResumption` | `false` | 알파 | 1.20 | | -| `EndpointSliceProxying` | `false` | 알파 | 1.18 | 1.18 | -| `EndpointSliceProxying` | `true` | 베타 | 1.19 | | -| `EndpointSliceTerminatingCondition` | `false` | 알파 | 1.20 | | +| `EfficientWatchResumption` | `false` | 알파 | 1.20 | 1.20 | +| `EfficientWatchResumption` | `true` | 베타 | 1.21 | | +| `EndpointSliceTerminatingCondition` | `false` | 알파 | 1.20 | 1.21 | +| `EndpointSliceTerminatingCondition` | `true` | 베타 | 1.22 | | | `EphemeralContainers` | `false` | 알파 | 1.16 | | | `ExpandCSIVolumes` | `false` | 알파 | 1.14 | 1.15 | | `ExpandCSIVolumes` | `true` | 베타 | 1.16 | | +| `ExpandedDNSConfig` | `false` | 알파 | 1.22 | | | `ExpandInUsePersistentVolumes` | `false` | 알파 | 1.11 | 1.14 | | `ExpandInUsePersistentVolumes` | `true` | 베타 | 1.15 | | | `ExpandPersistentVolumes` | `false` | 알파 | 1.8 | 1.10 | @@ -125,70 +123,83 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `GracefulNodeShutdown` | `true` | 베타 | 1.21 | | | `HPAContainerMetrics` | `false` | 알파 | 1.20 | | | `HPAScaleToZero` | `false` | 알파 | 1.16 | | -| `HugePageStorageMediumSize` | `false` | 알파 | 1.18 | 1.18 | -| `HugePageStorageMediumSize` | `true` | 베타 | 1.19 | | -| `IndexedJob` | `false` | 알파 | 1.21 | | -| `IngressClassNamespacedParams` | `false` | 알파 | 1.21 | | +| `IndexedJob` | `false` | 알파 | 1.21 | 1.21 | +| `IndexedJob` | `true` | 베타 | 1.22 | | +| `JobTrackingWithFinalizers` | `false` | 알파 | 1.22 | | +| `IngressClassNamespacedParams` | `false` | 알파 | 1.21 | 1.21 | +| `IngressClassNamespacedParams` | `true` | 베타 | 1.22 | | +| `InTreePluginAWSUnregister` | `false` | 알파 | 1.21 | | +| `InTreePluginAzureDiskUnregister` | `false` | 알파 | 1.21 | | +| `InTreePluginAzureFileUnregister` | `false` | 알파 | 1.21 | | +| `InTreePluginGCEUnregister` | `false` | 알파 | 1.21 | | +| `InTreePluginOpenStackUnregister` | `false` | 알파 | 1.21 | | +| `InTreePluginvSphereUnregister` | `false` | 알파 | 1.21 | | | `IPv6DualStack` | `false` | 알파 | 1.15 | 1.20 | | `IPv6DualStack` | `true` | 베타 | 1.21 | | +| `JobTrackingWithFinalizers` | `false` | 알파 | 1.22 | | | `KubeletCredentialProviders` | `false` | 알파 | 1.20 | | -| `LegacyNodeRoleBehavior` | `false` | 알파 | 1.16 | 1.18 | -| `LegacyNodeRoleBehavior` | `true` | 베타 | 1.19 | 1.20 | | `LocalStorageCapacityIsolation` | `false` | 알파 | 1.7 | 1.9 | | `LocalStorageCapacityIsolation` | `true` | 베타 | 1.10 | | | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | 알파 | 1.15 | | -| `LogarithmicScaleDown` | `false` | 알파 | 1.21 | | +| `LogarithmicScaleDown` | `false` | 알파 | 1.21 | 1.21 | +| `LogarithmicScaleDown` | `true` | 베타 | 1.22 | | +| `KubeletInUserNamespace` | `false` | 알파 | 1.22 | | | `KubeletPodResourcesGetAllocatable` | `false` | 알파 | 1.21 | | +| `MemoryManager` | `false` | 알파 | 1.21 | 1.21 | +| `MemoryManager` | `true` | 베타 | 1.22 | | +| `MemoryQoS` | `false` | 알파 | 1.22 | | | `MixedProtocolLBService` | `false` | 알파 | 1.20 | | -| `NamespaceDefaultLabelName` | `true` | 베타 | 1.21 | | -| `NetworkPolicyEndPort` | `false` | 알파 | 1.21 | | -| `NodeDisruptionExclusion` | `false` | 알파 | 1.16 | 1.18 | -| `NodeDisruptionExclusion` | `true` | 베타 | 1.19 | 1.20 | +| `NetworkPolicyEndPort` | `false` | 알파 | 1.21 | 1.21 | +| `NetworkPolicyEndPort` | `true` | 베타 | 1.22 | | +| `NodeSwap` | `false` | 알파 | 1.22 | | | `NonPreemptingPriority` | `false` | 알파 | 1.15 | 1.18 | | `NonPreemptingPriority` | `true` | 베타 | 1.19 | | -| `PodDeletionCost` | `false` | 알파 | 1.21 | | -| `PodAffinityNamespaceSelector` | `false` | 알파 | 1.21 | | +| `PodDeletionCost` | `false` | 알파 | 1.21 | 1.21 | +| `PodDeletionCost` | `true` | 베타 | 1.22 | | +| `PodAffinityNamespaceSelector` | `false` | 알파 | 1.21 | 1.21 | +| `PodAffinityNamespaceSelector` | `true` | 베타 | 1.22 | | | `PodOverhead` | `false` | 알파 | 1.16 | 1.17 | -| `PodOverhead` | `true` | 베타 | 1.18 | | -| `ProbeTerminationGracePeriod` | `false` | 알파 | 1.21 | | +| `PodOverhead` | `true` | 베타 | 1.18 | | +| `PodSecurity` | `false` | 알파 | 1.22 | | +| `PreferNominatedNode` | `false` | 알파 | 1.21 | 1.21 | +| `PreferNominatedNode` | `true` | 베타 | 1.22 | | +| `ProbeTerminationGracePeriod` | `false` | 알파 | 1.21 | 1.21 | +| `ProbeTerminationGracePeriod` | `false` | 베타 | 1.22 | | +| `ProxyTerminatingEndpoints` | `false` | 알파 | 1.22 | | | `ProcMountType` | `false` | 알파 | 1.12 | | | `QOSReserved` | `false` | 알파 | 1.11 | | +| `ReadWriteOncePod` | `false` | 알파 | 1.22 | | | `RemainingItemCount` | `false` | 알파 | 1.15 | 1.15 | | `RemainingItemCount` | `true` | 베타 | 1.16 | | | `RemoveSelfLink` | `false` | 알파 | 1.16 | 1.19 | | `RemoveSelfLink` | `true` | 베타 | 1.20 | | | `RotateKubeletServerCertificate` | `false` | 알파 | 1.7 | 1.11 | | `RotateKubeletServerCertificate` | `true` | 베타 | 1.12 | | -| `RunAsGroup` | `true` | 베타 | 1.14 | | -| `ServerSideApply` | `false` | 알파 | 1.14 | 1.15 | -| `ServerSideApply` | `true` | 베타 | 1.16 | | -| `ServiceInternalTrafficPolicy` | `false` | 알파 | 1.21 | | -| `ServiceLBNodePortControl` | `false` | 알파 | 1.20 | | -| `ServiceLoadBalancerClass` | `false` | 알파 | 1.21 | | -| `ServiceNodeExclusion` | `false` | 알파 | 1.8 | 1.18 | -| `ServiceNodeExclusion` | `true` | 베타 | 1.19 | 1.20 | -| `ServiceTopology` | `false` | 알파 | 1.17 | | -| `SetHostnameAsFQDN` | `false` | 알파 | 1.19 | 1.19 | -| `SetHostnameAsFQDN` | `true` | 베타 | 1.20 | | -| `SizeMemoryBackedVolumes` | `false` | 알파 | 1.20 | | +| `SeccompDefault` | `false` | 알파 | 1.22 | | +| `ServiceInternalTrafficPolicy` | `false` | 알파 | 1.21 | 1.21 | +| `ServiceInternalTrafficPolicy` | `true` | 베타 | 1.22 | | +| `ServiceLBNodePortControl` | `false` | 알파 | 1.20 | 1.21 | +| `ServiceLBNodePortControl` | `true` | 베타 | 1.22 | | +| `ServiceLoadBalancerClass` | `false` | 알파 | 1.21 | 1.21 | +| `ServiceLoadBalancerClass` | `true` | 베타 | 1.22 | | +| `SizeMemoryBackedVolumes` | `false` | 알파 | 1.20 | 1.21 | +| `SizeMemoryBackedVolumes` | `true` | 베타 | 1.22 | | +| `StatefulSetMinReadySeconds` | `false` | 알파 | 1.22 | | | `StorageVersionAPI` | `false` | 알파 | 1.20 | | | `StorageVersionHash` | `false` | 알파 | 1.14 | 1.14 | | `StorageVersionHash` | `true` | 베타 | 1.15 | | -| `SuspendJob` | `false` | 알파 | 1.21 | | +| `SuspendJob` | `false` | 알파 | 1.21 | 1.21 | +| `SuspendJob` | `true` | 베타 | 1.22 | | | `TTLAfterFinished` | `false` | 알파 | 1.12 | 1.20 | | `TTLAfterFinished` | `true` | 베타 | 1.21 | | | `TopologyAwareHints` | `false` | 알파 | 1.21 | | | `TopologyManager` | `false` | 알파 | 1.16 | 1.17 | | `TopologyManager` | `true` | 베타 | 1.18 | | -| `ValidateProxyRedirects` | `false` | 알파 | 1.12 | 1.13 | -| `ValidateProxyRedirects` | `true` | 베타 | 1.14 | | | `VolumeCapacityPriority` | `false` | 알파 | 1.21 | - | -| `WarningHeaders` | `true` | 베타 | 1.19 | | | `WinDSR` | `false` | 알파 | 1.14 | | | `WinOverlay` | `false` | 알파 | 1.14 | 1.19 | | `WinOverlay` | `true` | 베타 | 1.20 | | -| `WindowsEndpointSliceProxying` | `false` | 알파 | 1.19 | 1.20 | -| `WindowsEndpointSliceProxying` | `true` | 베타 | 1.21 | | +| `WindowsHostProcessContainers` | `false` | 알파 | 1.22 | | {{< /table >}} ### GA 또는 사용 중단된 기능을 위한 기능 게이트 @@ -206,9 +217,17 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `AffinityInAnnotations` | - | 사용중단 | 1.8 | - | | `AllowExtTrafficLocalEndpoints` | `false` | 베타 | 1.4 | 1.6 | | `AllowExtTrafficLocalEndpoints` | `true` | GA | 1.7 | - | +| `AttachVolumeLimit` | `false` | 알파 | 1.11 | 1.11 | +| `AttachVolumeLimit` | `true` | 베타 | 1.12 | 1.16 | +| `AttachVolumeLimit` | `true` | GA | 1.17 | - | +| `BalanceAttachedNodeVolumes` | `false` | 알파 | 1.11 | 1.21 | +| `BalanceAttachedNodeVolumes` | `false` | 사용중단 | 1.22 | | | `BlockVolume` | `false` | 알파 | 1.9 | 1.12 | | `BlockVolume` | `true` | 베타 | 1.13 | 1.17 | | `BlockVolume` | `true` | GA | 1.18 | - | +| `BoundServiceAccountTokenVolume` | `false` | 알파 | 1.13 | 1.20 | +| `BoundServiceAccountTokenVolume` | `true` | 베타 | 1.21 | 1.21 | +| `BoundServiceAccountTokenVolume` | `true` | GA | 1.22 | - | | `CRIContainerLogRotation` | `false` | 알파 | 1.10 | 1.10 | | `CRIContainerLogRotation` | `true` | 베타 | 1.11 | 1.20 | | `CRIContainerLogRotation` | `true` | GA | 1.21 | - | @@ -218,15 +237,30 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `CSIDriverRegistry` | `false` | 알파 | 1.12 | 1.13 | | `CSIDriverRegistry` | `true` | 베타 | 1.14 | 1.17 | | `CSIDriverRegistry` | `true` | GA | 1.18 | | +| `CSIMigrationAWSComplete` | `false` | 알파 | 1.17 | 1.20 | +| `CSIMigrationAWSComplete` | - | 사용중단 | 1.21 | - | +| `CSIMigrationAzureDiskComplete` | `false` | 알파 | 1.17 | 1.20 | +| `CSIMigrationAzureDiskComplete` | - | 사용중단 | 1.21 | - | +| `CSIMigrationAzureFileComplete` | `false` | 알파 | 1.17 | 1.20 | +| `CSIMigrationAzureFileComplete` | - | 사용중단 | 1.21 | - | +| `CSIMigrationGCEComplete` | `false` | 알파 | 1.17 | 1.20 | +| `CSIMigrationGCEComplete` | - | 사용중단 | 1.21 | - | +| `CSIMigrationOpenStackComplete` | `false` | 알파 | 1.17 | 1.20 | +| `CSIMigrationOpenStackComplete` | - | 사용중단 | 1.21 | - | +| `CSIMigrationvSphereComplete` | `false` | 베타 | 1.19 | 1.21 | +| `CSIMigrationvSphereComplete` | - | 사용중단 | 1.22 | - | | `CSINodeInfo` | `false` | 알파 | 1.12 | 1.13 | | `CSINodeInfo` | `true` | 베타 | 1.14 | 1.16 | | `CSINodeInfo` | `true` | GA | 1.17 | | -| `AttachVolumeLimit` | `false` | 알파 | 1.11 | 1.11 | -| `AttachVolumeLimit` | `true` | 베타 | 1.12 | 1.16 | -| `AttachVolumeLimit` | `true` | GA | 1.17 | - | | `CSIPersistentVolume` | `false` | 알파 | 1.9 | 1.9 | | `CSIPersistentVolume` | `true` | 베타 | 1.10 | 1.12 | | `CSIPersistentVolume` | `true` | GA | 1.13 | - | +| `CSIServiceAccountToken` | `false` | 알파 | 1.20 | 1.20 | +| `CSIServiceAccountToken` | `true` | 베타 | 1.21 | 1.21 | +| `CSIServiceAccountToken` | `true` | GA | 1.22 | | +| `CronJobControllerV2` | `false` | 알파 | 1.20 | 1.20 | +| `CronJobControllerV2` | `true` | 베타 | 1.21 | 1.21 | +| `CronJobControllerV2` | `true` | GA | 1.22 | - | | `CustomPodDNS` | `false` | 알파 | 1.9 | 1.9 | | `CustomPodDNS` | `true` | 베타| 1.10 | 1.13 | | `CustomPodDNS` | `true` | GA | 1.14 | - | @@ -250,8 +284,14 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `DryRun` | `true` | GA | 1.19 | - | | `DynamicAuditing` | `false` | 알파 | 1.13 | 1.18 | | `DynamicAuditing` | - | 사용중단 | 1.19 | - | +| `DynamicKubeletConfig` | `false` | 알파 | 1.4 | 1.10 | +| `DynamicKubeletConfig` | `true` | 베타 | 1.11 | 1.21 | +| `DynamicKubeletConfig` | `false` | 사용중단 | 1.22 | - | | `DynamicProvisioningScheduling` | `false` | 알파 | 1.11 | 1.11 | | `DynamicProvisioningScheduling` | - | 사용중단| 1.12 | - | +| `DynamicKubeletConfig` | `false` | 알파 | 1.4 | 1.10 | +| `DynamicKubeletConfig` | `true` | 베타 | 1.11 | 1.21 | +| `DynamicKubeletConfig` | `false` | 사용중단 | 1.22 | - | | `DynamicVolumeProvisioning` | `true` | 알파 | 1.3 | 1.7 | | `DynamicVolumeProvisioning` | `true` | GA | 1.8 | - | | `EnableAggregatedDiscoveryTimeout` | `true` | 사용중단 | 1.16 | - | @@ -263,6 +303,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `EndpointSlice` | `true` | GA | 1.21 | - | | `EndpointSliceNodeName` | `false` | 알파 | 1.20 | 1.20 | | `EndpointSliceNodeName` | `true` | GA | 1.21 | - | +| `EndpointSliceProxying` | `false` | 알파 | 1.18 | 1.18 | +| `EndpointSliceProxying` | `true` | 베타 | 1.19 | 1.21 | +| `EndpointSliceProxying` | `true` | GA | 1.22 | - | | `ExperimentalCriticalPodAnnotation` | `false` | 알파 | 1.5 | 1.12 | | `ExperimentalCriticalPodAnnotation` | `false` | 사용중단 | 1.13 | - | | `EvenPodsSpread` | `false` | 알파 | 1.16 | 1.17 | @@ -272,9 +315,15 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `ExternalPolicyForExternalIP` | `true` | GA | 1.18 | - | | `GCERegionalPersistentDisk` | `true` | 베타 | 1.10 | 1.12 | | `GCERegionalPersistentDisk` | `true` | GA | 1.13 | - | +| `HugePageStorageMediumSize` | `false` | 알파 | 1.18 | 1.18 | +| `HugePageStorageMediumSize` | `true` | 베타 | 1.19 | 1.21 | +| `HugePageStorageMediumSize` | `true` | GA | 1.22 | - | | `HugePages` | `false` | 알파 | 1.8 | 1.9 | | `HugePages` | `true` | 베타| 1.10 | 1.13 | | `HugePages` | `true` | GA | 1.14 | - | +| `HugePageStorageMediumSize` | `false` | 알파 | 1.18 | 1.18 | +| `HugePageStorageMediumSize` | `true` | 베타 | 1.19 | 1.21 | +| `HugePageStorageMediumSize` | `true` | GA | 1.22 | - | | `HyperVContainer` | `false` | 알파 | 1.10 | 1.19 | | `HyperVContainer` | `false` | 사용중단 | 1.20 | - | | `ImmutableEphemeralVolumes` | `false` | 알파 | 1.18 | 1.18 | @@ -290,16 +339,22 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `KubeletPodResources` | `false` | 알파 | 1.13 | 1.14 | | `KubeletPodResources` | `true` | 베타 | 1.15 | | | `KubeletPodResources` | `true` | GA | 1.20 | | +| `LegacyNodeRoleBehavior` | `false` | 알파 | 1.16 | 1.18 | +| `LegacyNodeRoleBehavior` | `true` | 베타 | 1.19 | 1.20 | | `LegacyNodeRoleBehavior` | `false` | GA | 1.21 | - | | `MountContainers` | `false` | 알파 | 1.9 | 1.16 | | `MountContainers` | `false` | 사용중단 | 1.17 | - | | `MountPropagation` | `false` | 알파 | 1.8 | 1.9 | | `MountPropagation` | `true` | 베타 | 1.10 | 1.11 | | `MountPropagation` | `true` | GA | 1.12 | - | +| `NodeDisruptionExclusion` | `false` | 알파 | 1.16 | 1.18 | +| `NodeDisruptionExclusion` | `true` | 베타 | 1.19 | 1.20 | | `NodeDisruptionExclusion` | `true` | GA | 1.21 | - | | `NodeLease` | `false` | 알파 | 1.12 | 1.13 | | `NodeLease` | `true` | 베타 | 1.14 | 1.16 | | `NodeLease` | `true` | GA | 1.17 | - | +| `NamespaceDefaultLabelName` | `true` | 베타 | 1.21 | 1.21 | +| `NamespaceDefaultLabelName` | `true` | GA | 1.22 | - | | `PVCProtection` | `false` | 알파 | 1.9 | 1.9 | | `PVCProtection` | - | 사용중단 | 1.10 | - | | `PersistentLocalVolumes` | `false` | 알파 | 1.7 | 1.9 | @@ -329,15 +384,23 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `RootCAConfigMap` | `true` | GA | 1.21 | - | | `RotateKubeletClientCertificate` | `true` | 베타 | 1.8 | 1.18 | | `RotateKubeletClientCertificate` | `true` | GA | 1.19 | - | +| `RunAsGroup` | `true` | 베타 | 1.14 | 1.20 | +| `RunAsGroup` | `true` | GA | 1.21 | - | | `RuntimeClass` | `false` | 알파 | 1.12 | 1.13 | | `RuntimeClass` | `true` | 베타 | 1.14 | 1.19 | | `RuntimeClass` | `true` | GA | 1.20 | - | -| `ScheduleDaemonSetPods` | `false` | 알파 | 1.11 | 1.11 | -| `ScheduleDaemonSetPods` | `true` | 베타 | 1.12 | 1.16 | -| `ScheduleDaemonSetPods` | `true` | GA | 1.17 | - | | `SCTPSupport` | `false` | 알파 | 1.12 | 1.18 | | `SCTPSupport` | `true` | 베타 | 1.19 | 1.19 | | `SCTPSupport` | `true` | GA | 1.20 | - | +| `ScheduleDaemonSetPods` | `false` | 알파 | 1.11 | 1.11 | +| `ScheduleDaemonSetPods` | `true` | 베타 | 1.12 | 1.16 | +| `ScheduleDaemonSetPods` | `true` | GA | 1.17 | - | +| `SelectorIndex` | `false` | 알파 | 1.18 | 1.18 | +| `SelectorIndex` | `true` | 베타 | 1.19 | 1.19 | +| `SelectorIndex` | `true` | GA | 1.20 | - | +| `ServerSideApply` | `false` | 알파 | 1.14 | 1.15 | +| `ServerSideApply` | `true` | 베타 | 1.16 | 1.21 | +| `ServerSideApply` | `true` | GA | 1.22 | - | | `ServiceAccountIssuerDiscovery` | `false` | 알파 | 1.18 | 1.19 | | `ServiceAccountIssuerDiscovery` | `true` | 베타 | 1.20 | 1.20 | | `ServiceAccountIssuerDiscovery` | `true` | GA | 1.21 | - | @@ -347,15 +410,23 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `ServiceLoadBalancerFinalizer` | `false` | 알파 | 1.15 | 1.15 | | `ServiceLoadBalancerFinalizer` | `true` | 베타 | 1.16 | 1.16 | | `ServiceLoadBalancerFinalizer` | `true` | GA | 1.17 | - | +| `ServiceNodeExclusion` | `false` | 알파 | 1.8 | 1.18 | +| `ServiceNodeExclusion` | `true` | 베타 | 1.19 | 1.20 | | `ServiceNodeExclusion` | `true` | GA | 1.21 | - | +| `ServiceTopology` | `false` | 알파 | 1.17 | 1.19 | +| `ServiceTopology` | `false` | 사용중단 | 1.20 | - | +| `SetHostnameAsFQDN` | `false` | 알파 | 1.19 | 1.19 | +| `SetHostnameAsFQDN` | `true` | 베타 | 1.20 | 1.21 | +| `SetHostnameAsFQDN` | `true` | GA | 1.22 | - | | `StartupProbe` | `false` | 알파 | 1.16 | 1.17 | | `StartupProbe` | `true` | 베타 | 1.18 | 1.19 | | `StartupProbe` | `true` | GA | 1.20 | - | | `StorageObjectInUseProtection` | `true` | 베타 | 1.10 | 1.10 | | `StorageObjectInUseProtection` | `true` | GA | 1.11 | - | | `StreamingProxyRedirects` | `false` | 베타 | 1.5 | 1.5 | -| `StreamingProxyRedirects` | `true` | 베타 | 1.6 | 1.18 | -| `StreamingProxyRedirects` | - | GA | 1.19 | - | +| `StreamingProxyRedirects` | `true` | 베타 | 1.6 | 1.17 | +| `StreamingProxyRedirects` | `true` | 사용중단 | 1.18 | 1.21 | +| `StreamingProxyRedirects` | `false` | 사용중단 | 1.22 | - | | `SupportIPVSProxyMode` | `false` | 알파 | 1.8 | 1.8 | | `SupportIPVSProxyMode` | `false` | 베타 | 1.9 | 1.9 | | `SupportIPVSProxyMode` | `true` | 베타 | 1.10 | 1.10 | @@ -380,6 +451,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `TokenRequestProjection` | `false` | 알파 | 1.11 | 1.11 | | `TokenRequestProjection` | `true` | 베타 | 1.12 | 1.19 | | `TokenRequestProjection` | `true` | GA | 1.20 | - | +| `ValidateProxyRedirects` | `false` | 알파 | 1.12 | 1.13 | +| `ValidateProxyRedirects` | `true` | 베타 | 1.14 | 1.21 | +| `ValidateProxyRedirects` | `true` | 사용중단 | 1.22 | - | | `VolumePVCDataSource` | `false` | 알파 | 1.15 | 1.15 | | `VolumePVCDataSource` | `true` | 베타 | 1.16 | 1.17 | | `VolumePVCDataSource` | `true` | GA | 1.18 | - | @@ -393,12 +467,18 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 | `VolumeSubpathEnvExpansion` | `false` | 알파 | 1.14 | 1.14 | | `VolumeSubpathEnvExpansion` | `true` | 베타 | 1.15 | 1.16 | | `VolumeSubpathEnvExpansion` | `true` | GA | 1.17 | - | +| `WarningHeaders` | `true` | 베타 | 1.19 | 1.21 | +| `WarningHeaders` | `true` | GA | 1.22 | - | | `WatchBookmark` | `false` | 알파 | 1.15 | 1.15 | | `WatchBookmark` | `true` | 베타 | 1.16 | 1.16 | | `WatchBookmark` | `true` | GA | 1.17 | - | +| `WindowsEndpointSliceProxying` | `false` | 알파 | 1.19 | 1.20 | +| `WindowsEndpointSliceProxying` | `true` | 베타 | 1.21 | 1.21 | +| `WindowsEndpointSliceProxying` | `true` | GA | 1.22| - | | `WindowsGMSA` | `false` | 알파 | 1.14 | 1.15 | | `WindowsGMSA` | `true` | 베타 | 1.16 | 1.17 | | `WindowsGMSA` | `true` | GA | 1.18 | - | +| `WindowsHostProcessContainers` | `false` | 알파 | 1.22 | | `WindowsRunAsUserName` | `false` | 알파 | 1.16 | 1.16 | | `WindowsRunAsUserName` | `true` | 베타 | 1.17 | 1.17 | | `WindowsRunAsUserName` | `true` | GA | 1.18 | - | @@ -453,6 +533,7 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 관리할 수 있다. (`RequestManagement` 에서 이름이 변경됨) - `APIResponseCompression`: `LIST` 또는 `GET` 요청에 대한 API 응답을 압축한다. - `APIServerIdentity`: 클러스터의 각 API 서버에 ID를 할당한다. +- `APIServerTracing`: API 서버에서 분산 추적(tracing)에 대한 지원을 추가한다. - `Accelerators`: 도커 사용 시 Nvidia GPU 지원 활성화한다. - `AdvancedAuditing`: [고급 감사](/docs/tasks/debug-application-cluster/audit/#advanced-audit) 기능을 활성화한다. - `AffinityInAnnotations`: [파드 어피니티 또는 안티-어피니티](/ko/docs/concepts/scheduling-eviction/assign-pod-node/#어피니티-affinity-와-안티-어피니티-anti-affinity) @@ -486,6 +567,7 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 리더 마이그레이션(Leader Migration)을 활성화한다. - `CPUManager`: 컨테이너 수준의 CPU 어피니티 지원을 활성화한다. [CPU 관리 정책](/docs/tasks/administer-cluster/cpu-management-policies/)을 참고한다. +- `CPUManagerPolicyOptions`: CPUManager 정책의 미세 조정을 허용한다. - `CRIContainerLogRotation`: cri 컨테이너 런타임에 컨테이너 로그 로테이션을 활성화한다. 로그 파일 사이즈 기본값은 10MB이며, 컨테이너 당 최대 로그 파일 수 기본값은 5이다. 이 값은 kubelet 환경설정으로 변경할 수 있다. 더 자세한 내용은 [노드 레벨에서의 로깅](/ko/docs/concepts/cluster-administration/logging/#노드-레벨에서의-로깅)을 참고한다. @@ -506,6 +588,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 인-트리 플러그인에서 EBS CSI 플러그인으로 라우팅할 수 있다. 클러스터의 모든 노드에 CSIMigration과 CSIMigrationAWS 기능 플래그가 활성화되고 EBS CSI 플러그인이 설치 및 구성이 되어 있어야 한다. + 이 플래그는 인-트리 EBS 플러그인의 등록을 막는 `InTreePluginAWSUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSIMigrationAzureDisk`: shim 및 변환 로직을 통해 볼륨 작업을 Azure-Disk 인-트리 플러그인에서 AzureDisk CSI 플러그인으로 라우팅할 수 있다. 노드에 AzureDisk CSI 플러그인이 설치와 구성이 되어 있지 않은 경우 인-트리 @@ -516,7 +600,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 Azure-Disk 인-트리 플러그인에서 AzureDisk CSI 플러그인으로 라우팅할 수 있다. 클러스터의 모든 노드에 CSIMigration과 CSIMigrationAzureDisk 기능 플래그가 활성화되고 AzureDisk CSI 플러그인이 설치 및 구성이 되어 - 있어야 한다. + 있어야 한다. 이 플래그는 인-트리 AzureDisk 플러그인의 등록을 막는 `InTreePluginAzureDiskUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSIMigrationAzureFile`: shim 및 변환 로직을 통해 볼륨 작업을 Azure-File 인-트리 플러그인에서 AzureFile CSI 플러그인으로 라우팅할 수 있다. 노드에 AzureFile CSI 플러그인이 설치 및 구성이 되어 있지 않은 경우 인-트리 @@ -527,7 +612,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 Azure 파일 인-트리 플러그인에서 AzureFile CSI 플러그인으로 라우팅할 수 있다. 클러스터의 모든 노드에 CSIMigration과 CSIMigrationAzureFile 기능 플래그가 활성화되고 AzureFile CSI 플러그인이 설치 및 구성이 되어 - 있어야 한다. + 있어야 한다. 이 플래그는 인-트리 AzureFile 플러그인의 등록을 막는 + `InTreePluginAzureFileUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSIMigrationGCE`: shim 및 변환 로직을 통해 볼륨 작업을 GCE-PD 인-트리 플러그인에서 PD CSI 플러그인으로 라우팅할 수 있다. 노드에 PD CSI 플러그인이 설치 및 구성이 되어 있지 않은 경우 인-트리 GCE 플러그인으로 폴백을 @@ -536,7 +623,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 인-트리 플러그인 등록을 중지하고 shim 및 변환 로직을 통해 볼륨 작업을 GCE-PD 인-트리 플러그인에서 PD CSI 플러그인으로 라우팅할 수 있다. CSIMigration과 CSIMigrationGCE 기능 플래그가 활성화되고 PD CSI - 플러그인이 클러스터의 모든 노드에 설치 및 구성이 되어 있어야 한다. + 플러그인이 클러스터의 모든 노드에 설치 및 구성이 되어 있어야 한다. 이 플래그는 인-트리 GCE PD 플러그인의 등록을 막는 `InTreePluginGCEUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSIMigrationOpenStack`: shim 및 변환 로직을 통해 볼륨 작업을 Cinder 인-트리 플러그인에서 Cinder CSI 플러그인으로 라우팅할 수 있다. 노드에 Cinder CSI 플러그인이 설치 및 구성이 되어 있지 않은 경우 인-트리 @@ -545,7 +633,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 Cinder 인-트리 플러그인 등록을 중지하고 shim 및 변환 로직이 Cinder 인-트리 플러그인에서 Cinder CSI 플러그인으로 볼륨 작업을 라우팅할 수 있도록 한다. 클러스터의 모든 노드에 CSIMigration과 CSIMigrationOpenStack 기능 플래그가 활성화되고 - Cinder CSI 플러그인이 설치 및 구성이 되어 있어야 한다. + Cinder CSI 플러그인이 설치 및 구성이 되어 있어야 한다. 이 플래그는 인-트리 openstack cinder 플러그인의 등록을 막는 `InTreePluginOpenStackUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSIMigrationvSphere`: vSphere 인-트리 플러그인에서 vSphere CSI 플러그인으로 볼륨 작업을 라우팅하는 shim 및 변환 로직을 사용한다. 노드에 vSphere CSI 플러그인이 설치 및 구성이 되어 있지 않은 경우 @@ -554,7 +643,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 플러그인 등록을 중지하고 shim 및 변환 로직을 활성화하여 vSphere 인-트리 플러그인에서 vSphere CSI 플러그인으로 볼륨 작업을 라우팅할 수 있도록 한다. CSIMigration 및 CSIMigrationvSphere 기능 플래그가 활성화되고 vSphere CSI 플러그인이 - 클러스터의 모든 노드에 설치 및 구성이 되어 있어야 한다. + 클러스터의 모든 노드에 설치 및 구성이 되어 있어야 한다. 이 플래그는 인-트리 vsphere 플러그인의 등록을 막는 `InTreePluginvSphereUnregister` 기능 플래그로 인해 + 더 이상 사용되지 않는다. - `CSINodeInfo`: csi.storage.k8s.io에서 CSINodeInfo API 오브젝트와 관련된 모든 로직을 활성화한다. - `CSIPersistentVolume`: [CSI (Container Storage Interface)](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) 호환 볼륨 플러그인을 통해 프로비저닝된 볼륨을 감지하고 @@ -570,14 +660,17 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 이 필드는 CSI드라이버에서 생성된 볼륨이 마운트될 때 볼륨 소유권과 권한 수정을 지원하는지 여부를 제어한다. - `CSIVolumeHealth`: 노드에서의 CSI 볼륨 상태 모니터링 기능을 활성화한다. +- `CSRDuration`: 클라이언트가 쿠버네티스 CSR API를 통해 발급된 인증서의 기간을 + 요청할 수 있다. - `ConfigurableFSGroupPolicy`: 사용자가 파드에 볼륨을 마운트할 때 fsGroups에 대한 볼륨 권한 변경 정책을 구성할 수 있다. 자세한 내용은 [파드의 볼륨 권한 및 소유권 변경 정책 구성](/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods)을 참고한다. +- `ControllerManagerLeaderMigration`: `kube-controller-manager` 및 `cloud-controller-manager`에 + 대한 리더 마이그레이션을 지원한다. - `CronJobControllerV2`: {{< glossary_tooltip text="크론잡(CronJob)" term_id="cronjob" >}} 컨트롤러의 대체 구현을 사용한다. 그렇지 않으면, 동일한 컨트롤러의 버전 1이 선택된다. - 버전 2 컨트롤러는 실험적인 성능 향상을 제공한다. - `CustomCPUCFSQuotaPeriod`: [kubelet config](/docs/tasks/administer-cluster/kubelet-config-file/)에서 `cpuCFSQuotaPeriod` 를 노드가 변경할 수 있도록 한다. - `CustomPodDNS`: `dnsConfig` 속성을 사용하여 파드의 DNS 설정을 사용자 정의할 수 있다. @@ -591,12 +684,20 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 생성된 리소스에서 스키마 기반 유효성 검사를 활성화한다. - `CustomResourceWebhookConversion`: [커스텀리소스데피니션](/ko/docs/concepts/extend-kubernetes/api-extension/custom-resources/)에서 생성된 리소스에 대해 웹 훅 기반의 변환을 활성화한다. +- `DaemonSetUpdateSurge`: 노드당 업데이트 중 가용성을 유지하도록 + 데몬셋 워크로드를 사용하도록 설정한다. - `DefaultPodTopologySpread`: `PodTopologySpread` 스케줄링 플러그인을 사용하여 [기본 분배](/ko/docs/concepts/workloads/pods/pod-topology-spread-constraints/#내부-기본-제약)를 수행한다. +- `DelegateFSGroupToCSIDriver`: CSI 드라이버가 지원할 경우, NodeStageVolume 및 NodePublishVolume CSI 호출을 통해 + `fsGroup`를 전달하여 파드의 `securityContext`에서 + `fsGroup`를 드라이브에 적용하는 역할을 위임한다. - `DevicePlugins`: 노드에서 [장치 플러그인](/ko/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) 기반 리소스 프로비저닝을 활성화한다. - `DisableAcceleratorUsageMetrics`: [kubelet이 수집한 액셀러레이터 지표 비활성화](/ko/docs/concepts/cluster-administration/system-metrics/#액셀러레이터-메트릭-비활성화). +- `DisableCloudProviders`: `kube-apiserver`, `kube-controller-manager`, + `--cloud-provider` 컴포넌트 플래그와 관련된 `kubelet`의 + 모든 기능을 비활성화한다. - `DownwardAPIHugePages`: [다운워드 API](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information)에서 hugepages 사용을 활성화한다. - `DryRun`: 서버 측의 [dry run](/docs/reference/using-api/api-concepts/#dry-run) 요청을 @@ -635,6 +736,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 현재 수정된 결함에 의존하는 경우 존재한다. [준비성 프로브](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes)를 참조한다. - `ExpandCSIVolumes`: CSI 볼륨 확장을 활성화한다. +- `ExpandedDNSConfig`: 더 많은 DNS 검색 경로와 더 긴 DNS 검색 경로 목록을 허용하려면 + kubelet과 kube-apiserver를 사용하도록 설정한다. + [확장된 DNS 구성](/docs/concepts/services-networking/dns-pod-service/#expanded-dns-configuration)을 참고한다. - `ExpandInUsePersistentVolumes`: 사용 중인 PVC를 확장할 수 있다. [사용 중인 퍼시스턴트볼륨클레임 크기 조정](/ko/docs/concepts/storage/persistent-volumes/#사용-중인-퍼시스턴트볼륨클레임-크기-조정)을 참고한다. - `ExpandPersistentVolumes`: 퍼시스턴트 볼륨 확장을 활성화한다. @@ -671,8 +775,24 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 기능을 활성화한다. - `ImmutableEphemeralVolumes`: 안정성과 성능 향상을 위해 개별 시크릿(Secret)과 컨피그맵(ConfigMap)을 변경할 수 없는(immutable) 것으로 표시할 수 있다. +- `InTreePluginAWSUnregister`: kubelet 및 볼륨 컨트롤러에 aws-ebs 인-트리 + 플러그인의 등록을 중지한다. +- `InTreePluginAzureDiskUnregister`: kubelet 및 볼륨 컨트롤러에 azuredisk 인-트리 + 플러그인의 등록을 중지한다. +- `InTreePluginAzureFileUnregister`: kubelet 및 볼륨 컨트롤러에 azurefile 인-트리 + 플러그인의 등록을 중지한다. +- `InTreePluginGCEUnregister`: kubelet 및 볼륨 컨트롤러에 gce-pd 인-트리 + 플러그인의 등록을 중지한다. +- `InTreePluginOpenStackUnregister`: kubelet 및 볼륨 컨트롤러에 오픈스택 cinder 인-트리 + 플러그인의 등록을 중지한다. +- `InTreePluginvSphereUnregister`: kubelet 및 볼륨 컨트롤러에 vSphere 인-트리 + 플러그인의 등록을 중지한다. - `IndexedJob`: [잡](/ko/docs/concepts/workloads/controllers/job/) 컨트롤러가 완료 횟수를 기반으로 파드 완료를 관리할 수 있도록 한다. +- `JobTrackingWithFinalizers`: 클러스터에 무제한으로 남아 있는 파드에 의존하지 않고 + [잡](/ko/docs/concepts/workloads/controllers/job)의 완료를 추적할 수 있다. + 잡 컨트롤러는 완료된 파드를 추적하기 위해 + 완료된 파드의 잡 상태 필드를 사용한다. - `IngressClassNamespacedParams`: `IngressClass` 리소스가 네임스페이스 범위로 한정된 파라미터를 이용할 수 있도록 한다. 이 기능은 `IngressClass.spec.parameters` 에 `Scope` 와 `Namespace` 2개의 필드를 추가한다. @@ -680,11 +800,17 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 비동기 조정을 허용한다. - `IPv6DualStack`: IPv6을 위한 [이중 스택](/ko/docs/concepts/services-networking/dual-stack/) 기능을 활성화한다. +- `JobTrackingWithFinalizers`: 클러스터에 무제한으로 남아있는 파드에 + 의존하지 않고 잡 완료를 추적할 수 있다. + 파드 finalizers는 잡 상태 필드와 + 아직 구성되지 않은 파드를 추적할 수 있다. - `KubeletConfigFile`: 구성 파일을 사용하여 지정된 파일에서 kubelet 구성을 로드할 수 있다. 자세한 내용은 [구성 파일을 통해 kubelet 파라미터 설정](/docs/tasks/administer-cluster/kubelet-config-file/)을 참고한다. - `KubeletCredentialProviders`: 이미지 풀 자격 증명에 대해 kubelet exec 자격 증명 공급자를 활성화한다. +- `KubeletInUserNamespace`: {{}}에서 kubelet 실행을 활성화한다. + [루트가 아닌 유저로 쿠버네티스 노드 컴포넌트 실행](/docs/tasks/administer-cluster/kubelet-in-userns/)을 참고한다. - `KubeletPluginsWatcher`: kubelet이 [CSI 볼륨 드라이버](/ko/docs/concepts/storage/volumes/#csi)와 같은 플러그인을 검색할 수 있도록 프로브 기반 플러그인 감시자(watcher) 유틸리티를 사용한다. - `KubeletPodResources`: kubelet의 파드 리소스 gPRC 엔드포인트를 활성화한다. 자세한 내용은 @@ -709,6 +835,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 향상시킨다. - `LogarithmicScaleDown`: 컨트롤러 스케일 다운 시에 파드 타임스탬프를 로그 스케일로 버켓화하여 축출할 파드를 반-랜덤하게 선택하는 기법을 활성화한다. +- `MemoryManager`: NUMA 토폴로지를 기반으로 컨테이너에 대한 + 메모리 어피니티를 설정할 수 있다. +- `MemoryQoS`: cgroup v2 메모리 컨트롤러를 사용하여 파드/컨테이너에서 메모리 보호 및 사용 제한을 사용하도록 설정한다. - `MixedProtocolLBService`: 동일한 로드밸런서 유형 서비스 인스턴스에서 다른 프로토콜 사용을 활성화한다. - `MountContainers`: 호스트의 유틸리티 컨테이너를 볼륨 마운터로 사용할 수 있다. @@ -720,6 +849,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 - `NodeDisruptionExclusion`: 영역(zone) 장애 시 노드가 제외되지 않도록 노드 레이블 `node.kubernetes.io/exclude-disruption` 사용을 활성화한다. - `NodeLease`: 새로운 리스(Lease) API가 노드 상태 신호로 사용될 수 있는 노드 하트비트(heartbeats)를 보고할 수 있게 한다. +- `NodeSwap`: 노드의 쿠버네티스 워크로드용 스왑 메모리를 할당하려면 kubelet을 활성화한다. + 반드시 `KubeletConfiguration.failSwapOn`를 false로 설정한 후 사용해야 한다. + 더 자세한 정보는 [스왑 메모리](/docs/concepts/architecture/nodes/#swap-memory)를 참고한다. - `NonPreemptingPriority`: 프라이어리티클래스(PriorityClass)와 파드에 `preemptionPolicy` 필드를 활성화한다. - `PVCProtection`: 파드에서 사용 중일 때 퍼시스턴트볼륨클레임(PVC)이 삭제되지 않도록 한다. @@ -737,17 +869,25 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 - `PodReadinessGates`: 파드 준비성 평가를 확장하기 위해 `PodReadinessGate` 필드 설정을 활성화한다. 자세한 내용은 [파드의 준비성 게이트](/ko/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate)를 참고한다. +- `PodSecurity`: `PodSecurity` 어드미션 플러그인을 사용하도록 설정한다. - `PodShareProcessNamespace`: 파드에서 실행되는 컨테이너 간에 단일 프로세스 네임스페이스를 공유하기 위해 파드에서 `shareProcessNamespace` 설정을 활성화한다. 자세한 내용은 [파드의 컨테이너 간 프로세스 네임스페이스 공유](/docs/tasks/configure-pod-container/share-process-namespace/)에서 확인할 수 있다. +- `PreferNominatedNode`: 이 플래그는 클러스터에 존재하는 다른 노드를 반복해서 검사하기 전에 + 지정된 노드를 먼저 검사할지 여부를 + 스케줄러에 알려준다. - `ProbeTerminationGracePeriod`: 파드의 [프로브-수준 `terminationGracePeriodSeconds` 설정하기](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#probe-level-terminationgraceperiodseconds) 기능을 활성화한다. 더 자세한 사항은 [기능개선 제안](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2238-liveness-probe-grace-period)을 참고한다. - `ProcMountType`: SecurityContext의 `procMount` 필드를 설정하여 컨테이너의 proc 타입의 마운트를 제어할 수 있다. +- `ProxyTerminatingEndpoints`: `ExternalTrafficPolicy=Local`일 때 종료 엔드포인트를 처리하도록 + kube-proxy를 활성화한다. - `QOSReserved`: QoS 수준에서 리소스 예약을 허용하여 낮은 QoS 수준의 파드가 더 높은 QoS 수준에서 요청된 리소스로 파열되는 것을 방지한다 (현재 메모리만 해당). +- `ReadWriteOncePod`: `ReadWriteOncePod` 퍼시스턴트 볼륨 엑세스 모드를 + 사용한다. - `RemainingItemCount`: API 서버가 [청크(chunking) 목록 요청](/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks)에 대한 응답에서 남은 항목 수를 표시하도록 허용한다. @@ -779,6 +919,10 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 스케줄링할 수 있다. - `SCTPSupport`: 파드, 서비스, 엔드포인트, 엔드포인트슬라이스 및 네트워크폴리시 정의에서 _SCTP_ `protocol` 값을 활성화한다. +- `SeccompDefault`: 모든 워크로드의 기본 구분 프로파일로 `RuntimeDefault`을 사용한다. + seccomp 프로파일은 파드 및 컨테이너 `securityContext`에 지정되어 있다. +- `SelectorIndex`: API 서버 감시(watch) 캐시의 레이블 및 필드 기반 인덱스를 사용하여 + 목록 작업을 가속화할 수 있다. - `ServerSideApply`: API 서버에서 [SSA(Sever Side Apply)](/docs/reference/using-api/server-side-apply/) 경로를 활성화한다. - `ServiceAccountIssuerDiscovery`: API 서버에서 서비스 어카운트 발행자에 대해 OIDC 디스커버리 엔드포인트(발급자 및 @@ -806,6 +950,8 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 - `StartupProbe`: kubelet에서 [스타트업](/ko/docs/concepts/workloads/pods/pod-lifecycle/#언제-스타트업-프로브를-사용해야-하는가) 프로브를 활성화한다. +- `StatefulSetMinReadySeconds`: 스테이트풀셋 컨트롤러가 `minReadySeconds`를 + 반영할 수 있다. - `StorageObjectInUseProtection`: 퍼시스턴트볼륨 또는 퍼시스턴트볼륨클레임 오브젝트가 여전히 사용 중인 경우 삭제를 연기한다. - `StorageVersionAPI`: [스토리지 버전 API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#storageversion-v1alpha1-internal-apiserver-k8s-io)를 @@ -867,6 +1013,7 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 - `WinDSR`: kube-proxy가 윈도우용 DSR 로드 밸런서를 생성할 수 있다. - `WinOverlay`: kube-proxy가 윈도우용 오버레이 모드에서 실행될 수 있도록 한다. - `WindowsGMSA`: 파드에서 컨테이너 런타임으로 GMSA 자격 증명 스펙을 전달할 수 있다. +- `WindowsHostProcessContainers`: 윈도우 HostProcess 컨테이너에 대한 지원을 사용하도록 설정한다. - `WindowsRunAsUserName` : 기본 사용자가 아닌(non-default) 사용자로 윈도우 컨테이너에서 애플리케이션을 실행할 수 있도록 지원한다. 자세한 내용은 [RunAsUserName 구성](/ko/docs/tasks/configure-pod-container/configure-runasusername/)을 @@ -875,7 +1022,9 @@ kubelet과 같은 컴포넌트의 기능 게이트를 설정하려면, 기능 엔드포인트 대신 엔드포인트슬라이스를 기본 데이터 소스로 사용하여 확장성과 성능을 향상시킨다. [엔드포인트슬라이스 활성화하기](/ko/docs/concepts/services-networking/endpoint-slices/)를 참고한다. - +- `WindowsHostProcessContainers`: 윈도우 노드에서 `HostProcess` + 컨테이너 지원을 활성화한다. + ## {{% heading "whatsnext" %}} From f0192d5c9e36d9536c7722a9030ce33212327799 Mon Sep 17 00:00:00 2001 From: Juhee Kang Date: Mon, 27 Sep 2021 19:48:33 +0900 Subject: [PATCH 142/249] [ko] Update outdated files in dev-1.22-ko.1 (p4) --- .../workloads/controllers/statefulset.md | 49 ++++--- .../ko/docs/concepts/workloads/pods/_index.md | 19 ++- .../workloads/pods/ephemeral-containers.md | 129 ++---------------- .../workloads/pods/init-containers.md | 5 +- .../concepts/workloads/pods/pod-lifecycle.md | 2 +- .../generate-ref-docs/quickstart.md | 13 +- content/ko/docs/reference/_index.md | 3 + .../service-accounts-admin.md | 9 +- 8 files changed, 73 insertions(+), 156 deletions(-) diff --git a/content/ko/docs/concepts/workloads/controllers/statefulset.md b/content/ko/docs/concepts/workloads/controllers/statefulset.md index dbb30150e3..d39559e24c 100644 --- a/content/ko/docs/concepts/workloads/controllers/statefulset.md +++ b/content/ko/docs/concepts/workloads/controllers/statefulset.md @@ -223,27 +223,31 @@ web-0이 실패할 경우 web-1은 web-0이 Running 및 Ready 상태가 ## 업데이트 전략 -쿠버네티스 1.7 및 이후에는 스테이트풀셋의 `.spec.updateStrategy` 필드는 스테이트풀셋의 +스테이트풀셋의 `.spec.updateStrategy` 필드는 스테이트풀셋의 파드에 대한 컨테이너, 레이블, 리소스의 요청/제한 그리고 주석에 대한 자동화된 롤링 업데이트를 -구성하거나 비활성화 할 수 있다. +구성하거나 비활성화할 수 있다. 두 가지 가능한 전략이 있다. -### 삭제 시(On Delete) - -`OnDelete` 업데이트 전략은 레거시(1.6과 이전)의 행위를 구현한다. 이때 스테이트풀셋의 -`.spec.updateStrategy.type` 은 `OnDelete` 를 설정하며, 스테이트풀셋 컨트롤러는 -스테이트풀셋의 파드를 자동으로 업데이트하지 않는다. 사용자는 컨트롤러가 스테이트풀셋의 +`OnDelete`(삭제시) +: 스테이트풀셋의 `.spec.updateStrategy.type` 은 `OnDelete` 를 설정하며, +스테이트풀셋 컨트롤러는 스테이트풀셋의 파드를 자동으로 업데이트하지 않는다. +사용자는 컨트롤러가 스테이트풀셋의 `.spec.template`를 반영하는 수정된 새로운 파드를 생성하도록 수동으로 파드를 삭제해야 한다. -### 롤링 업데이트 +`RollingUpdate`(롤링 업데이트) +: `롤링 업데이트` 의 업데이트 전략은 스테이트풀셋의 파드에 대한 롤링 업데이트를 +구현한다. 롤링 업데이트는 `.spec.updateStrategy` 가 지정되지 않으면 기본 전략이 된다. -`롤링 업데이트` 의 업데이트 전략은 스테이트풀셋의 파드에 대한 롤링 업데이트를 -구현한다. 롤링 업데이트는 `.spec.updateStrategy` 가 지정되지 않으면 기본 전략이 된다. 스테이트풀셋에 `롤링 업데이트` 가 `.spec.updateStrategy.type` 에 설정되면 -스테이트풀셋 컨트롤러는 스테이트풀셋의 각 파드를 삭제 및 재생성을 한다. 이 과정에서 똑같이 -순차적으로 파드가 종료되고(가장 큰 수에서 작은 수까지), -각 파드의 업데이트는 한 번에 하나씩 한다. 이전 버전을 업데이트하기 전까지 업데이트된 파드가 실행 및 준비될 -때까지 기다린다. +## 롤링 업데이트 -#### 파티션(Partition) +스테이트풀셋에 `롤링 업데이트` 가 `.spec.updateStrategy.type` 에 설정되면 +스테이트풀셋 컨트롤러는 스테이트풀셋의 각 파드를 삭제 및 재생성한다. 이 과정에서 똑같이 +순차적으로 파드가 종료되고(가장 큰 순서 색인에서부터에서 작은 순서 색인쪽으로), +각 파드의 업데이트는 한 번에 하나씩 한다. + +쿠버네티스 컨트롤 플레인은 이전 버전을 업데이트 하기 전에, 업데이트된 파드가 실행 및 준비될 때까지 기다린다. +`.spec.minReadySeconds`([최소 준비 시간 초](#minimum-ready-seconds) 참조)를 설정한 경우, 컨트롤 플레인은 파드가 준비 상태로 전환된 후 해당 시간을 추가로 기다린 후 이동한다. + +### 파티션 롤링 업데이트 {#partitions} `롤링 업데이트` 의 업데이트 전략은 `.spec.updateStrategy.rollingUpdate.partition` 를 명시해서 파티션 할 수 있다. 만약 파티션을 명시하면 스테이트풀셋의 `.spec.template` 가 @@ -255,7 +259,7 @@ web-0이 실패할 경우 web-1은 web-0이 Running 및 Ready 상태가 대부분의 케이스는 파티션을 사용할 필요가 없지만 업데이트를 준비하거나, 카나리의 롤 아웃 또는 단계적인 롤 아웃을 행하려는 경우에는 유용하다. -#### 강제 롤백 +### 강제 롤백 기본 [파드 관리 정책](#파드-관리-정책) (`OrderedReady`)과 함께 [롤링 업데이트](#롤링-업데이트)를 사용할 경우 @@ -273,8 +277,19 @@ web-0이 실패할 경우 web-1은 web-0이 Running 및 Ready 상태가 템플릿을 되돌린 이후에는 스테이트풀셋이 이미 잘못된 구성으로 실행하려고 시도한 모든 파드를 삭제해야 한다. -그러면 스테이트풀셋은 되돌린 템플릿을 사용해서 파드를 다시 생성하기 시작 한다. +그러면 스테이트풀셋은 되돌린 템플릿을 사용해서 파드를 다시 생성하기 시작한다. +### 최소 준비 시간 초 {#minimum-ready-seconds} + +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + +`.spec.minReadySeconds`는 새로 생성된 파드가 사용가능하다고 간주되도록 +컨테이너가 충돌되지 않고 준비되는 최소 시간 초를 지정하는 선택적 필드이다. +기본값은 0이다(파드는 준비되는 대로 사용 가능한 것으로 간주된다). +파드가 준비가 되는 시기에 대해 더 자세히 알아보고 싶다면, +[컨테이너 프로브](/ko/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)를 참고한다. + +이 필드는 `StatefulSetMinReadySeconds` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)를 사용하도록 설정한 경우에만 작동한다. ## {{% heading "whatsnext" %}} diff --git a/content/ko/docs/concepts/workloads/pods/_index.md b/content/ko/docs/concepts/workloads/pods/_index.md index 22b98b705c..54fdf4a9a2 100644 --- a/content/ko/docs/concepts/workloads/pods/_index.md +++ b/content/ko/docs/concepts/workloads/pods/_index.md @@ -257,8 +257,12 @@ POSIX 공유 메모리와 같은 표준 프로세스 간 통신을 사용하여 ## 컨테이너에 대한 특권 모드 -파드의 모든 컨테이너는 컨테이너 명세의 [보안 콘텍스트](/docs/tasks/configure-pod-container/security-context/)에 있는 `privileged` 플래그를 사용하여 특권 모드를 활성화할 수 있다. 이는 네트워크 스택 조작이나 하드웨어 장치 접근과 같은 운영 체제 관리 기능을 사용하려는 컨테이너에 유용하다. -특권이 있는 컨테이너 내의 프로세스는 컨테이너 외부의 프로세스가 가지는 거의 동일한 권한을 가진다. +리눅스에서, 파드의 모든 컨테이너는 컨테이너 명세의 [보안 컨텍스트](/docs/tasks/configure-pod-container/security-context/)에 있는 `privileged` (리눅스) 플래그를 사용하여 특권 모드를 활성화할 수 있다. 이는 네트워크 스택 조작이나 하드웨어 장치 접근과 같은 운영 체제 관리 기능을 사용하려는 컨테이너에 유용하다. +클러스터가 `WindowsHostProcessContainers` 기능을 활성화하였다면, 파드 스펙의 보안 컨텍스트의 `windowsOptions.hostProcess` 에 의해 [윈도우 HostProcess 파드](/docs/tasks/configure-pod-container/create-hostprocess-pod)를 생성할 수 있다. 이러한 모든 컨테이너는 윈도우 HostProcess 컨테이너로 실행해야 한다. HostProcess 파드는 직접적으로 호스트에서 실행하는 것으로, 리눅스 특권있는 컨테이너에서 수행되는 관리 태스크 수행에도 사용할 수 있다. + +파드의 모든 컨테이너는 윈도우 HostProcess 컨테이너로 반드시 실행해야 한다. + +HostProcess 파드는 호스트에서 직접 실행되며 리눅스 특권있는 컨테이너에서 수행되는 것과 같은 관리 작업을 수행하는데도 사용할 수 있다. {{< note >}} 이 설정을 사용하려면 사용자의 {{< glossary_tooltip text="컨테이너 런타임" term_id="container-runtime" >}}이 특권이 있는 컨테이너의 개념을 지원해야 한다. @@ -282,6 +286,17 @@ kubelet은 자동으로 각 정적 파드에 대한 쿠버네티스 API 서버 즉, 노드에서 실행되는 파드는 API 서버에서 보이지만, 여기에서 제어할 수는 없다는 의미이다. +## 컨테이너 프로브 + +_프로브_는 컨테이너의 kubelet에 의해 주기적으로 실행되는 진단이다. 진단을 수행하기 위하여 kubelet은 다음과 같은 작업을 호출할 수 있다. + +- `ExecAction` (컨테이너 런타임의 도움을 받아 수행) +- `TCPSocketAction` (kubelet에 의해 직접 검사) +- `HTTPGetAction` (kubelet에 의해 직접 검사) + +[프로브](/ko/docs/concepts/workloads/pods/pod-lifecycle/#컨테이너-프로브-probe)에 대한 자세한 내용은 +파드 라이프사이클 문서를 참고한다. + ## {{% heading "whatsnext" %}} * [파드의 라이프사이클](/ko/docs/concepts/workloads/pods/pod-lifecycle/)에 대해 알아본다. diff --git a/content/ko/docs/concepts/workloads/pods/ephemeral-containers.md b/content/ko/docs/concepts/workloads/pods/ephemeral-containers.md index 9aa9e9bf51..136413b88f 100644 --- a/content/ko/docs/concepts/workloads/pods/ephemeral-containers.md +++ b/content/ko/docs/concepts/workloads/pods/ephemeral-containers.md @@ -6,15 +6,15 @@ weight: 80 -{{< feature-state state="alpha" for_k8s_version="v1.16" >}} +{{< feature-state state="alpha" for_k8s_version="v1.22" >}} -이 페이지는 임시 컨테이너에 대한 개요를 제공한다: 이 특별한 유형의 컨테이너는 -트러블 슈팅과 같은 사용자가 시작한 작업을 완료하기위해 기존 {{< glossary_tooltip text="파드" term_id="pod" >}} 에서 -임시적으로 실행된다. 사용자는 애플리케이션 빌드보다는 서비스를 점검할 때 임시 -컨테이너를 사용한다. +이 페이지는 임시 컨테이너에 대한 개요를 제공한다. +이 특별한 유형의 컨테이너는 트러블슈팅과 같은 사용자가 시작한 작업을 완료하기 위해 +기존 {{< glossary_tooltip text="파드" term_id="pod" >}}에서 임시적으로 실행된다. +임시 컨테이너는 애플리케이션을 빌드하는 경우보다는 서비스 점검과 같은 경우에 더 적합하다. {{< warning >}} -임시 컨테이너는 초기 알파 상태이며, +임시 컨테이너 기능은 알파 상태이며, 프로덕션 클러스터에는 적합하지 않다. [쿠버네티스 사용 중단(deprecation) 정책](/docs/reference/using-api/deprecation-policy/)에 따라 이 알파 기능은 향후 크게 변경되거나, 완전히 제거될 수 있다. @@ -72,119 +72,8 @@ API에서 특별한 `ephemeralcontainers` 핸들러를 사용해서 만들어지 임시 컨테이너 사용 시 [프로세스 네임스페이스 공유](/docs/tasks/configure-pod-container/share-process-namespace/)를 -활성화하면 다른 컨테이너 안의 프로세스를 보는데 도움이 된다. +활성화하면 다른 컨테이너 안의 프로세스를 보는 데 도움이 된다. -임시 컨테이너를 사용해서 문제를 해결하는 예시는 -[임시 디버깅 컨테이너로 디버깅하기] -(/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container)를 참조한다. +## {{% heading "whatsnext" %}} -## 임시 컨테이너 API - -{{< note >}} -이 섹션의 예시는 `EphemeralContainers` [기능 -게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)의 -활성화를 필요로 하고, 쿠버네티스 클라이언트와 서버는 v1.16 또는 이후의 버전이어야 한다. -{{< /note >}} - -이 섹션의 예시는 임시 컨테이너가 어떻게 API에 나타나는지 -보여준다. 일반적으로 `kubectl debug` 또는 -다른 `kubectl` [플러그인](/ko/docs/tasks/extend-kubectl/kubectl-plugins/)을 -사용해서 API를 직접 호출하지 않고 이런 단계들을 자동화 한다. - -임시 컨테이너는 파드의 `ephemeralcontainers` 하위 리소스를 -사용해서 생성되며, `kubectl --raw` 를 사용해서 보여준다. 먼저 -`EphemeralContainers` 목록으로 추가하는 임시 컨테이너를 명시한다. - -```json -{ - "apiVersion": "v1", - "kind": "EphemeralContainers", - "metadata": { - "name": "example-pod" - }, - "ephemeralContainers": [{ - "command": [ - "sh" - ], - "image": "busybox", - "imagePullPolicy": "IfNotPresent", - "name": "debugger", - "stdin": true, - "tty": true, - "terminationMessagePolicy": "File" - }] -} -``` - -이미 실행중인 `example-pod` 에 임시 컨테이너를 업데이트 한다. - -```shell -kubectl replace --raw /api/v1/namespaces/default/pods/example-pod/ephemeralcontainers -f ec.json -``` - -그러면 새로운 임시 컨테이너 목록이 반환된다. - -```json -{ - "kind":"EphemeralContainers", - "apiVersion":"v1", - "metadata":{ - "name":"example-pod", - "namespace":"default", - "selfLink":"/api/v1/namespaces/default/pods/example-pod/ephemeralcontainers", - "uid":"a14a6d9b-62f2-4119-9d8e-e2ed6bc3a47c", - "resourceVersion":"15886", - "creationTimestamp":"2019-08-29T06:41:42Z" - }, - "ephemeralContainers":[ - { - "name":"debugger", - "image":"busybox", - "command":[ - "sh" - ], - "resources":{ - - }, - "terminationMessagePolicy":"File", - "imagePullPolicy":"IfNotPresent", - "stdin":true, - "tty":true - } - ] -} -``` - -사용자는 `kubectl describe` 를 사용해서 새로 만든 임시 컨테이너의 상태를 볼 수 있다. - -```shell -kubectl describe pod example-pod -``` - -``` -... -Ephemeral Containers: - debugger: - Container ID: docker://cf81908f149e7e9213d3c3644eda55c72efaff67652a2685c1146f0ce151e80f - Image: busybox - Image ID: docker-pullable://busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 - Port: - Host Port: - Command: - sh - State: Running - Started: Thu, 29 Aug 2019 06:42:21 +0000 - Ready: False - Restart Count: 0 - Environment: - Mounts: -... -``` - -예시와 같이 `kubectl attach`, `kubectl exec`, 그리고 `kubectl logs` 를 사용해서 -다른 컨테이너와 같은 방식으로 새로운 임시 컨테이너와 -상호작용할 수 있다. - -```shell -kubectl attach -it example-pod -c debugger -``` +* [임시 컨테이너 디버깅하기](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container)에 대해 알아보기. diff --git a/content/ko/docs/concepts/workloads/pods/init-containers.md b/content/ko/docs/concepts/workloads/pods/init-containers.md index c8c7055408..a1c01b17ab 100644 --- a/content/ko/docs/concepts/workloads/pods/init-containers.md +++ b/content/ko/docs/concepts/workloads/pods/init-containers.md @@ -290,8 +290,9 @@ myapp-pod 1/1 Running 0 9m 초기화 컨테이너에게 명령과 실행이 주어진 경우, 리소스 사용에 대한 다음의 규칙이 적용된다. -* 모든 컨테이너에 정의된 특정 리소스 요청량 또는 상한 중 가장 - 높은 것은 *유효한 초기화 요청량/상한* 이다. +* 모든 컨테이너에 정의된 특정 리소스 요청량 또는 상한 중 + 가장 높은 것은 *유효 초기화 요청량/상한* 이다. 리소스 제한이 지정되지 않은 리소스는 + 이 *유효 초기화 요청량/상한*을 가장 높은 요청량/상한으로 간주한다. * 리소스를 위한 파드의 *유효한 초기화 요청량/상한* 은 다음 보다 더 높다. * 모든 앱 컨테이너의 리소스에 대한 요청량/상한의 합계 * 리소스에 대한 유효한 초기화 요청량/상한 diff --git a/content/ko/docs/concepts/workloads/pods/pod-lifecycle.md b/content/ko/docs/concepts/workloads/pods/pod-lifecycle.md index 11aeaa31b0..1e2754ebaf 100644 --- a/content/ko/docs/concepts/workloads/pods/pod-lifecycle.md +++ b/content/ko/docs/concepts/workloads/pods/pod-lifecycle.md @@ -379,7 +379,7 @@ TERM 대신 이 값을 보낸다. 확인하는 즉시(정상적인 종료 기간이 설정됨), kubelet은 로컬 파드의 종료 프로세스를 시작한다. 1. 파드의 컨테이너 중 하나가 `preStop` - [훅](/ko/docs/concepts/containers/container-lifecycle-hooks/#hook-details)을 정의한 경우, kubelet은 + [훅](/ko/docs/concepts/containers/container-lifecycle-hooks)을 정의한 경우, kubelet은 컨테이너 내부에서 해당 훅을 실행한다. 유예 기간이 만료된 후 `preStop` 훅이 계속 실행되면, kubelet은 2초의 작은 일회성 유예 기간 연장을 요청한다. diff --git a/content/ko/docs/contribute/generate-ref-docs/quickstart.md b/content/ko/docs/contribute/generate-ref-docs/quickstart.md index 6855696b9d..531b1ad50b 100644 --- a/content/ko/docs/contribute/generate-ref-docs/quickstart.md +++ b/content/ko/docs/contribute/generate-ref-docs/quickstart.md @@ -6,7 +6,7 @@ weight: 40 -이 문서에서는 `update-imported-docs` 스크립트를 사용하여 +이 문서에서는 `update-imported-docs.py` 스크립트를 사용하여 쿠버네티스 레퍼런스 문서를 생성하는 방법에 대해 설명한다. 이 스크립트는 특정 쿠버네티스 릴리스 버전에 대해 빌드 설정을 자동으로 수행하고 레퍼런스 문서를 생성한다. @@ -39,7 +39,7 @@ git clone git@github.com:/website.git ## `update-imported-docs` 스크립트 개요 {#Overview-of-update-imported-docs} -`update-imported-docs` 스크립트는 `/update-imported-docs/` +`update-imported-docs.py` 스크립트는 `/update-imported-docs/` 디렉터리에 존재한다. 이 스크립트는 다음 레퍼런스를 생성한다. @@ -48,7 +48,7 @@ git clone git@github.com:/website.git * `kubectl` 명령어 레퍼런스 * 쿠버네티스 API 레퍼런스 -`update-imported-docs` 스크립트는 쿠버네티스 소스코드로부터 레퍼런스 문서를 +`update-imported-docs.py` 스크립트는 쿠버네티스 소스코드로부터 레퍼런스 문서를 생성한다. 스크립트가 실행되면 개발 머신의 `/tmp` 디렉터리 아래에 임시 디렉터리를 생성하고, 이 임시 디렉터리 아래에 레퍼런스 문서 생성에 필요한 `kubernetes/kubernetes` 저장소와 `kubernetes-sigs/reference-docs` 저장소를 클론하며, @@ -69,7 +69,7 @@ git clone git@github.com:/website.git `kubernetes-sigs/reference-docs/Makefile` 에 있는 Make 타겟들을 활용하여 빌드하는 일련의 과정이 명시되어 있다. `K8S_RELEASE` 환경 변수는 릴리스 버전을 결정한다. -`update-imported-docs` 스크립트는 다음의 과정을 수행한다. +`update-imported-docs.py` 스크립트는 다음의 과정을 수행한다. 1. 환경설정 파일에 있는 관련 저장소를 클론한다. 레퍼런스 문서 생성을 위해 @@ -152,11 +152,11 @@ repos: ## `update-imported-docs` 도구 실행하기 {#Running-the-update-imported-docs-tool} -다음과 같이 `update-imported-docs` 도구를 실행할 수 있다. +다음과 같이 `update-imported-docs.py` 도구를 실행할 수 있다. ```shell cd /update-imported-docs -./update-imported-docs +./update-imported-docs.py ``` 예를 들면 다음과 같다. @@ -254,4 +254,3 @@ static/docs/reference/generated/kubernetes-api/{{< param "version" >}}/fonts/fon * [kubectl 명령어에 대한 레퍼런스 문서 생성하기](/docs/contribute/generate-ref-docs/kubectl/) * [쿠버네티스 API에 대한 레퍼런스 문서 생성하기](/docs/contribute/generate-ref-docs/kubernetes-api/) - diff --git a/content/ko/docs/reference/_index.md b/content/ko/docs/reference/_index.md index 55401988b4..8bfa706ea4 100644 --- a/content/ko/docs/reference/_index.md +++ b/content/ko/docs/reference/_index.md @@ -71,8 +71,10 @@ TCP/UDP 스트림 포워딩이나 백-엔드 집합에 걸쳐서 라운드-로 사용/관리하는 데에 중요하지만, 이들 API의 대부분은 아직 API 서버가 제공하지 않는다. +* [kube-apiserver 환경설정 (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/) * [kubelet 환경설정 (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) * [kube-scheduler 환경설정 (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) +* [kube-scheduler 환경설정 (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) * [kube-scheduler 정책 레퍼런스 (v1)](/docs/reference/config-api/kube-scheduler-policy-config.v1/) * [kube-proxy 환경설정 (v1alpha1)](/docs/reference/config-api/kube-proxy-config.v1alpha1/) * [`audit.k8s.io/v1` API](/docs/reference/config-api/apiserver-audit.v1/) @@ -82,6 +84,7 @@ TCP/UDP 스트림 포워딩이나 백-엔드 집합에 걸쳐서 라운드-로 ## kubeadm을 위한 API 설정 * [v1beta2](/docs/reference/config-api/kubeadm-config.v1beta2/) +* [v1beta3](/docs/reference/config-api/kubeadm-config.v1beta3/) ## 설계 문서 diff --git a/content/ko/docs/reference/access-authn-authz/service-accounts-admin.md b/content/ko/docs/reference/access-authn-authz/service-accounts-admin.md index ca06783465..55615e55de 100644 --- a/content/ko/docs/reference/access-authn-authz/service-accounts-admin.md +++ b/content/ko/docs/reference/access-authn-authz/service-accounts-admin.md @@ -57,10 +57,9 @@ weight: 50 #### 바인딩된 서비스 어카운트 토큰 볼륨 -{{< feature-state for_k8s_version="v1.21" state="beta" >}} +{{< feature-state for_k8s_version="v1.22" state="stable" >}} -`BoundServiceAccountTokenVolume` [기능 게이트](/ko/docs/reference/command-line-tools-reference/feature-gates/)가 활성화되면, -토큰 컨트롤러에 의해 생성된 무기한 서비스 어카운트 토큰을 위해, 서비스 어카운트 어드미션 컨트롤러가 시크릿 기반 볼륨 대신 다음과 같은 프로젝티드 볼륨을 추가한다. +서비스 어카운트 어드미션 컨트롤러는 토큰 컨트롤러에서 생성한 만료되지 않은 서비스 계정 토큰에 시크릿 기반 볼륨 대신 다음과 같은 프로젝티드 볼륨을 추가한다. ```yaml - name: kube-api-access- @@ -91,10 +90,6 @@ weight: 50 상세 사항은 [프로젝티드 볼륨](/docs/tasks/configure-pod-container/configure-projected-volume-storage/)을 참고한다. -`BoundServiceAccountTokenVolume` 기능 게이트가 활성화되어 있지 않은 경우, -위의 프로젝티드 볼륨을 파드 스펙에 추가하여 -시크릿 기반 서비스 어카운트 볼륨을 프로젝티드 볼륨으로 수동으로 옮길 수 있다. - ### 토큰 컨트롤러 토큰컨트롤러는 `kube-controller-manager` 의 일부로 실행된다. 이것은 비동기적으로 동작한다. 토큰 컨트롤러는, From b1a2be6706bd554607f7afffbef72e7bf98eba90 Mon Sep 17 00:00:00 2001 From: Seokho Son Date: Wed, 6 Oct 2021 02:43:24 +0900 Subject: [PATCH 143/249] Update outdated Korean configuration/secret --- .../ko/docs/concepts/configuration/secret.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/content/ko/docs/concepts/configuration/secret.md b/content/ko/docs/concepts/configuration/secret.md index 06be7d5a54..330e89edac 100644 --- a/content/ko/docs/concepts/configuration/secret.md +++ b/content/ko/docs/concepts/configuration/secret.md @@ -12,26 +12,33 @@ weight: 30 -쿠버네티스 시크릿을 사용하면 비밀번호, OAuth 토큰, ssh 키와 같은 -민감한 정보를 저장하고 관리할 수 ​​있다. 기밀 정보를 시크릿에 저장하는 것이 -{{< glossary_tooltip term_id="pod" >}} 정의나 -{{< glossary_tooltip text="컨테이너 이미지" term_id="image" >}} -내에 그대로 두는 것보다 안전하고 유연하다. -자세한 내용은 [시크릿 디자인 문서](https://git.k8s.io/community/contributors/design-proposals/auth/secrets.md)를 참고한다. - 시크릿은 암호, 토큰 또는 키와 같은 소량의 중요한 데이터를 -포함하는 오브젝트이다. 그렇지 않으면 이러한 정보가 파드 -명세나 이미지에 포함될 수 있다. 사용자는 시크릿을 만들 수 있고 시스템도 -일부 시크릿을 만들 수 있다. +포함하는 오브젝트이다. 이를 사용하지 않으면 중요한 정보가 {{< glossary_tooltip text="파드" term_id="pod" >}} +명세나 {{< glossary_tooltip text="컨테이너 이미지" term_id="image" >}}에 +포함될 수 있다. 시크릿을 사용한다는 것은 사용자의 기밀 데이터를 +애플리케이션 코드에 넣을 필요가 +없음을 뜻한다. + +시크릿은 시크릿을 사용하는 파드와 독립적으로 생성될 수 있기 때문에, +파드를 생성하고, 확인하고, 수정하는 워크플로우 동안 시크릿(그리고 데이터)이 +노출되는 것에 대한 위험을 경감시킬 수 있다. 쿠버네티스 +및 클러스터에서 실행되는 애플리케이션은 기밀 데이터를 비휘발성 +저장소에 쓰는 것을 피하는 것과 같이, 시크릿에 대해 추가 예방 조치를 취할 수도 있다. + +시크릿은 {{< glossary_tooltip text="컨피그맵" term_id="configmap" >}}과 유사하지만 +특별히 기밀 데이터를 보관하기 위한 것이다. {{< caution >}} -쿠버네티스 시크릿은 기본적으로 암호화되지 않은 base64 인코딩 문자열로 저장된다. -기본적으로 API 액세스 권한이 있는 모든 사용자 또는 쿠버네티스의 기본 데이터 저장소 etcd에 -액세스할 수 있는 모든 사용자가 일반 텍스트로 검색할 수 있다. -시크릿을 안전하게 사용하려면 (최소한) 다음과 같이 하는 것이 좋다. +쿠버네티스 시크릿은 기본적으로 API 서버의 기본 데이터 저장소(etcd)에 암호화되지 않은 상태로 저장된다. API 접근(access) 권한이 있는 모든 사용자 또는 etcd에 접근할 수 있는 모든 사용자는 시크릿을 조회하거나 수정할 수 있다. +또한 네임스페이스에서 파드를 생성할 권한이 있는 사람은 누구나 해당 접근을 사용하여 해당 네임스페이스의 모든 시크릿을 읽을 수 있다. 여기에는 디플로이먼트 생성 기능과 같은 간접 접근이 포함된다. + +시크릿을 안전하게 사용하려면 최소한 다음의 단계를 따르는 것이 좋다. 1. 시크릿에 대한 [암호화 활성화](/docs/tasks/administer-cluster/encrypt-data/). -2. 시크릿 읽기 및 쓰기를 제한하는 [RBAC 규칙 활성화 또는 구성](/ko/docs/reference/access-authn-authz/authorization/). 파드를 만들 권한이 있는 모든 사용자는 시크릿을 암묵적으로 얻을 수 있다. +2. 시크릿의 데이터 읽기 및 쓰기(간접적인 방식 포함)를 제한하는 [RBAC 규칙](/ko/docs/reference/access-authn-authz/authorization/) + 활성화 또는 구성. +3. 적절한 경우, RBAC와 같은 메커니즘을 사용하여 새로운 시크릿을 생성하거나 기존 시크릿을 대체할 수 있는 주체(principal)들을 제한한다. + {{< /caution >}} @@ -47,6 +54,10 @@ weight: 30 - [컨테이너 환경 변수](#시크릿을-환경-변수로-사용하기)로써 사용. - 파드의 [이미지를 가져올 때 kubelet](#imagepullsecrets-사용하기)에 의해 사용. +쿠버네티스 컨트롤 플레인 또한 시크릿을 사용한다. 예를 들어, +[부트스트랩 토큰 시크릿](#부트스트랩-토큰-시크릿)은 +노드 등록을 자동화하는 데 도움을 주는 메커니즘이다. + 시크릿 오브젝트의 이름은 유효한 [DNS 서브도메인 이름](/ko/docs/concepts/overview/working-with-objects/names/#dns-서브도메인-이름)이어야 한다. 사용자는 시크릿을 위한 파일을 구성할 때 `data` 및 (또는) `stringData` 필드를 @@ -1236,7 +1247,6 @@ API 서버에서 kubelet으로의 통신은 SSL/TLS로 보호된다. API 서버 정책이 해당 사용자가 시크릿을 읽을 수 있도록 허용하지 않더라도, 사용자는 시크릿을 노출하는 파드를 실행할 수 있다. - ## {{% heading "whatsnext" %}} - [`kubectl` 을 사용한 시크릿 관리](/docs/tasks/configmap-secret/managing-secret-using-kubectl/)하는 방법 배우기 From e47542c90cef8df6dffd3959b1d63f430a18a906 Mon Sep 17 00:00:00 2001 From: Seokho Son Date: Wed, 6 Oct 2021 02:00:16 +0900 Subject: [PATCH 144/249] Update outdated Korean architecture/nodes --- .../ko/docs/concepts/architecture/nodes.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/content/ko/docs/concepts/architecture/nodes.md b/content/ko/docs/concepts/architecture/nodes.md index 3ab89472d8..5daed12077 100644 --- a/content/ko/docs/concepts/architecture/nodes.md +++ b/content/ko/docs/concepts/architecture/nodes.md @@ -392,6 +392,49 @@ Message: Node is shutting, evicting pods 이는 갑작스러운 노드 종료의 경우와 비교했을 때 동작에 차이가 있다. {{< /note >}} +## 스왑(swap) 메모리 관리 {#swap-memory} + +{{< feature-state state="alpha" for_k8s_version="v1.22" >}} + +쿠버네티스 1.22 이전에는 노드가 스왑 메모리를 지원하지 않았다. 그리고 +kubelet은 노드에서 스왑을 발견하지 못한 경우 시작과 동시에 실패하도록 되어 있었다. +1.22부터는 스왑 메모리 지원을 노드 단위로 활성화할 수 있다. + +노드에서 스왑을 활성화하려면, `NodeSwap` 기능 게이트가 kubelet에서 +활성화되어야 하며, 명령줄 플래그 `--fail-swap-on` 또는 +[구성 설정](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)에서 `failSwapOn`가 +false로 지정되어야 한다. + +사용자는 또한 선택적으로 `memorySwap.swapBehavior`를 구성할 수 있으며, +이를 통해 노드가 스왑 메모리를 사용하는 방식을 명시한다. 예를 들면, + +```yaml +memorySwap: + swapBehavior: LimitedSwap +``` + +`swapBehavior`에 가용한 구성 옵션은 다음과 같다. + +- `LimitedSwap`: 쿠버네티스 워크로드는 스왑을 사용할 수 있는 만큼으로 + 제한된다. 쿠버네티스에 의해 관리되지 않는 노드의 워크로드는 여전히 스왑될 수 있다. +- `UnlimitedSwap`: 쿠버네티스 워크로드는 요청한 만큼 스왑 메모리를 사용할 수 있으며, + 시스템의 최대치까지 사용 가능하다. + +만약 `memorySwap` 구성이 명시되지 않았고 기능 게이트가 활성화되어 있다면, +kubelet은 `LimitedSwap` 설정과 같은 행동을 +기본적으로 적용한다. + +`LimitedSwap` 설정에 대한 행동은 노드가 ("cgroups"으로 알려진) +제어 그룹이 v1 또는 v2 중에서 무엇으로 동작하는가에 따라서 결정된다. + +- **cgroupsv1:** 쿠버네티스 워크로드는 메모리와 스왑의 조합을 사용할 수 있다. + 파드의 메모리 제한이 설정되어 있다면 가용 상한이 된다. +- **cgroupsv2:** 쿠버네티스 워크로드는 스왑 메모리를 사용할 수 없다. + +테스트를 지원하고 피드벡을 제공하기 위한 정보는 +[KEP-2400](https://github.com/kubernetes/enhancements/issues/2400) 및 +[디자인 제안](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md)에서 찾을 수 있다. + ## {{% heading "whatsnext" %}} * 노드를 구성하는 [컴포넌트](/ko/docs/concepts/overview/components/#노드-컴포넌트)에 대해 알아본다. From 0a50d3ed5382bfab4d20a003d35d593f47d282e3 Mon Sep 17 00:00:00 2001 From: Arhell Date: Fri, 8 Oct 2021 00:38:30 +0300 Subject: [PATCH 145/249] [uk] Add seccomp tutorial to index --- content/uk/docs/tutorials/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/uk/docs/tutorials/_index.md b/content/uk/docs/tutorials/_index.md index c87a5155a0..a249e4814c 100644 --- a/content/uk/docs/tutorials/_index.md +++ b/content/uk/docs/tutorials/_index.md @@ -63,6 +63,8 @@ Before walking through each tutorial, you may want to bookmark the * [AppArmor](/docs/tutorials/clusters/apparmor/) +* [seccomp](/docs/tutorials/clusters/seccomp/) + ## Сервіси * [Використання Source IP](/docs/tutorials/services/source-ip/) From 2a84b55424c511aec75e7839420490754dbc0fa4 Mon Sep 17 00:00:00 2001 From: Chris Negus Date: Thu, 7 Oct 2021 20:41:50 -0400 Subject: [PATCH 146/249] Add file paths to keys and certificates (#28367) * Adding diagrams to certificates page * Cropped diagrams * Changed diagrams to tree output * Formatting fix * Fixed text block markup and spacing * Changed tree view of files to full-path view * Changed order of two cert files * Broke up links into separate sentences, per review comment * More changes per review comments --- .../docs/setup/best-practices/certificates.md | 50 +++++++++++++++++-- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/content/en/docs/setup/best-practices/certificates.md b/content/en/docs/setup/best-practices/certificates.md index 1648cc4e9e..defa8b59f6 100644 --- a/content/en/docs/setup/best-practices/certificates.md +++ b/content/en/docs/setup/best-practices/certificates.md @@ -38,11 +38,13 @@ etcd also implements mutual TLS to authenticate clients and peers. ## Where certificates are stored -If you install Kubernetes with kubeadm, certificates are stored in `/etc/kubernetes/pki`. All paths in this documentation are relative to that directory. +If you install Kubernetes with kubeadm, most certificates are stored in `/etc/kubernetes/pki`. All paths in this documentation are relative to that directory, with the exception of user account certificates which kubeadm places in `/etc/kubernetes`. ## Configure certificates manually -If you don't want kubeadm to generate the required certificates, you can create them in either of the following ways. +If you don't want kubeadm to generate the required certificates, you can create them using a single root CA or by providing all certificates. See [Certificates](/docs/tasks/administer-cluster/certificates/) for details on creating your own certificate authority. +See [Certificate Management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/) for more on managing certificates. + ### Single root CA @@ -57,7 +59,16 @@ Required CAs: | front-proxy-ca.crt,key | kubernetes-front-proxy-ca | For the [front-end proxy](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) | On top of the above CAs, it is also necessary to get a public/private key pair for service account management, `sa.key` and `sa.pub`. +The following example illustrates the CA key and certificate files shown in the previous table: +``` +/etc/kubernetes/pki/ca.crt +/etc/kubernetes/pki/ca.key +/etc/kubernetes/pki/etcd/ca.crt +/etc/kubernetes/pki/etcd/ca.key +/etc/kubernetes/pki/front-proxy-ca.crt +/etc/kubernetes/pki/front-proxy-ca.key +``` ### All certificates If you don't wish to copy the CA private keys to your cluster, you can generate all certificates yourself. @@ -127,6 +138,32 @@ Same considerations apply for the service account key pair: | sa.key | | kube-controller-manager | --service-account-private-key-file | | | sa.pub | kube-apiserver | --service-account-key-file | +The following example illustrates the file paths [from the previous tables](/docs/setup/best-practices/certificates/#certificate-paths) you need to provide if you are generating all of your own keys and certificates: + +``` +/etc/kubernetes/pki/etcd/ca.key +/etc/kubernetes/pki/etcd/ca.crt +/etc/kubernetes/pki/apiserver-etcd-client.key +/etc/kubernetes/pki/apiserver-etcd-client.crt +/etc/kubernetes/pki/ca.key +/etc/kubernetes/pki/ca.crt +/etc/kubernetes/pki/apiserver.key +/etc/kubernetes/pki/apiserver.crt +/etc/kubernetes/pki/apiserver-kubelet-client.key +/etc/kubernetes/pki/apiserver-kubelet-client.crt +/etc/kubernetes/pki/front-proxy-ca.key +/etc/kubernetes/pki/front-proxy-ca.crt +/etc/kubernetes/pki/front-proxy-client.key +/etc/kubernetes/pki/front-proxy-client.crt +/etc/kubernetes/pki/etcd/server.key +/etc/kubernetes/pki/etcd/server.crt +/etc/kubernetes/pki/etcd/peer.key +/etc/kubernetes/pki/etcd/peer.crt +/etc/kubernetes/pki/etcd/healthcheck-client.key +/etc/kubernetes/pki/etcd/healthcheck-client.crt +/etc/kubernetes/pki/sa.key +/etc/kubernetes/pki/sa.pub +``` ## Configure certificates for user accounts You must manually configure these administrator account and service accounts: @@ -146,7 +183,7 @@ The value of `` for `kubelet.conf` **must** match precisely the value 1. Run `kubectl` as follows for each config: -```shell +``` KUBECONFIG= kubectl config set-cluster default-cluster --server=https://:6443 --certificate-authority --embed-certs KUBECONFIG= kubectl config set-credentials --client-key .pem --client-certificate .pem --embed-certs KUBECONFIG= kubectl config set-context default-system --cluster default-cluster --user @@ -162,4 +199,11 @@ These files are used as follows: | controller-manager.conf | kube-controller-manager | Must be added to manifest in `manifests/kube-controller-manager.yaml` | | scheduler.conf | kube-scheduler | Must be added to manifest in `manifests/kube-scheduler.yaml` | +The following files illustrate full paths to the files listed in the previous table: +``` +/etc/kubernetes/admin.conf +/etc/kubernetes/kubelet.conf +/etc/kubernetes/controller-manager.conf +/etc/kubernetes/scheduler.conf +``` From 583ead2dd6e3796587834102003819f63df528c2 Mon Sep 17 00:00:00 2001 From: Steven Yan Date: Sun, 26 Sep 2021 11:59:26 +0800 Subject: [PATCH 147/249] [zh] translation for admin3 --- .../docs/tasks/administer-cluster/coredns.md | 45 +++---------------- .../tasks/administer-cluster/namespaces.md | 22 ++++----- .../administer-cluster/reconfigure-kubelet.md | 6 ++- 3 files changed, 22 insertions(+), 51 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/coredns.md b/content/zh/docs/tasks/administer-cluster/coredns.md index f68ce54759..22ada21026 100644 --- a/content/zh/docs/tasks/administer-cluster/coredns.md +++ b/content/zh/docs/tasks/administer-cluster/coredns.md @@ -67,12 +67,12 @@ For manual deployment or replacement of kube-dns, see the documentation at the 在 Kubernetes 1.10 及更高版本中，当你使用 `kubeadm` 升级使用 `kube-dns` 的集群时，你还可以迁移到 CoreDNS。 在本例中 `kubeadm` 将生成 CoreDNS 配置（"Corefile"）基于 `kube-dns` ConfigMap， -保存联邦、存根域和上游名称服务器的配置。 +保存存根域和上游名称服务器的配置。 在 Kubernetes 版本 1.13 和更高版本中，`CoreDNS`特性门已经删除，CoreDNS 在默认情况下使用。 -如果你想升级集群以使用 kube-dns，请遵循 -[此处](/zh/docs/reference/setup-tools/kubeadm/kubeadm-init-phase#cmd-phase-addon) 。 -## 使用 kubeadm 安装 kube-dns 而不是 CoreDNS - - -{{< note >}} -在 Kubernetes 1.11 中，CoreDNS 已经升级到通用可用性（GA），并默认安装。 -{{< /note >}} - - -{{< warning >}} -在 Kubernetes 1.18 中，用 kubeadm 来安装 kube-dns 这一做法已经被废弃， -会在将来版本中移除。 -{{< /warning >}} - - -若要在 1.13 之前版本上安装 kube-dns，请将 `CoreDNS` 特性门控设置为 `false`： - -```shell -kubeadm init --feature-gates=CoreDNS=false -``` - - -对于 1.13 版和更高版本，请遵循 -[此处](/zh/docs/reference/setup-tools/kubeadm/kubeadm-init-phase#cmd-phase-addon)概述到指南。 +在 kubernetes 1.21 中，kubeadm 移除了对 `kube-dns` 的支持。 +For more details, see [Namespace](/docs/reference/kubernetes-api/cluster-resources/namespace-v1/) +in the API reference. + --> 名字空间可以处于下列两个阶段中的一个: * `Active` 名字空间正在被使用中 * `Terminating` 名字空间正在被删除，且不能被用于新对象。 -参见[设计文档](https://git.k8s.io/community/contributors/design-proposals/architecture/namespaces.md#phases) 查看更多细节。 +更多细节，参阅 API 参考中的[命名空间](/docs/reference/kubernetes-api/cluster-resources/namespace-v1/)。 ## 创建名字空间 @@ -312,11 +314,11 @@ kubectl delete namespaces ``` - 我们刚刚创建了一个副本个数为 2 的 Deployment，运行名为 `snowflake` 的 - Pod，其中包含一个仅负责提供主机名的基本容器。 + 我们创建了一个副本个数为 2 的 Deployment，运行名为 `snowflake` 的 + Pod，其中包含一个负责提供主机名的基本容器。 ```shell kubectl get deployment -n=development @@ -486,7 +488,7 @@ Use cases include: * 进一步了解[设置名字空间偏好](/zh/docs/concepts/overview/working-with-objects/namespaces/#setting-the-namespace-preference) * 进一步了解[设置请求的名字空间](/zh/docs/concepts/overview/working-with-objects/namespaces/#setting-the-namespace-for-a-request) -* 参阅[名字空间的设计文档](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/architecture/namespaces.md) +* 参阅[名字空间的设计文档](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/namespaces.md) diff --git a/content/zh/docs/tasks/administer-cluster/reconfigure-kubelet.md b/content/zh/docs/tasks/administer-cluster/reconfigure-kubelet.md index 7d34460c24..67d4f90a9d 100644 --- a/content/zh/docs/tasks/administer-cluster/reconfigure-kubelet.md +++ b/content/zh/docs/tasks/administer-cluster/reconfigure-kubelet.md @@ -16,7 +16,11 @@ content_type: task {{< feature-state for_k8s_version="v1.22" state="deprecated" >}} {{< caution >}} [动态 kubelet 配置](https://github.com/kubernetes/enhancements/issues/281) From a2040433f62f9cc61b27d070e80f5bc588b0ced5 Mon Sep 17 00:00:00 2001 From: bang9211 Date: Fri, 8 Oct 2021 03:11:58 +0900 Subject: [PATCH 148/249] Translate tasks/run-application/run-stateless-application-deployment.md in Korean --- .../run-stateless-application-deployment.md | 160 ++++++++++++++++++ .../application/deployment-scale.yaml | 19 +++ .../application/deployment-update.yaml | 19 +++ 3 files changed, 198 insertions(+) create mode 100644 content/ko/docs/tasks/run-application/run-stateless-application-deployment.md create mode 100644 content/ko/examples/application/deployment-scale.yaml create mode 100644 content/ko/examples/application/deployment-update.yaml diff --git a/content/ko/docs/tasks/run-application/run-stateless-application-deployment.md b/content/ko/docs/tasks/run-application/run-stateless-application-deployment.md new file mode 100644 index 0000000000..ea5bfd76af --- /dev/null +++ b/content/ko/docs/tasks/run-application/run-stateless-application-deployment.md @@ -0,0 +1,160 @@ +--- +title: 디플로이먼트(Deployment)로 스테이트리스 애플리케이션 실행하기 +min-kubernetes-server-version: v1.9 +content_type: tutorial +weight: 10 +--- + + + +이 페이지에서는 쿠버네티스 디플로이먼트 오브젝트를 사용하여 애플리케이션을 실행하는 방법을 설명한다. + + + + +## {{% heading "objectives" %}} + + +* nginx 디플로이먼트 생성하기 +* kubectl을 사용하여 디플로이먼트 정보 나열하기 +* 디플로이먼트 업데이트하기 + + + + +## {{% heading "prerequisites" %}} + + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + + + +## nginx 디플로이먼트 생성하고 탐색하기 + +쿠버네티스 디플로이먼트 오브젝트를 생성하여 애플리케이션을 실행할 수 있으며, +디플로이먼트에 대한 명세를 YAML 파일에 기술할 수 있다. 예를 들어 이 YAML 파일은 +nginx:1.14.2 도커 이미지를 실행하는 디플로이먼트에 대한 명세를 담고 있다. + +{{< codenew file="application/deployment.yaml" >}} + + +1. YAML 파일을 기반으로 디플로이먼트를 생성한다. + + kubectl apply -f https://k8s.io/examples/application/deployment.yaml + +1. 디플로이먼트에 대한 정보를 살펴본다. + + kubectl describe deployment nginx-deployment + + 출력은 다음과 유사하다. + + Name: nginx-deployment + Namespace: default + CreationTimestamp: Tue, 30 Aug 2016 18:11:37 -0700 + Labels: app=nginx + Annotations: deployment.kubernetes.io/revision=1 + Selector: app=nginx + Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable + StrategyType: RollingUpdate + MinReadySeconds: 0 + RollingUpdateStrategy: 1 max unavailable, 1 max surge + Pod Template: + Labels: app=nginx + Containers: + nginx: + Image: nginx:1.14.2 + Port: 80/TCP + Environment: + Mounts: + Volumes: + Conditions: + Type Status Reason + ---- ------ ------ + Available True MinimumReplicasAvailable + Progressing True NewReplicaSetAvailable + OldReplicaSets: + NewReplicaSet: nginx-deployment-1771418926 (2/2 replicas created) + No events. + +1. 디플로이먼트에 의해 생성된 파드를 나열한다. + + kubectl get pods -l app=nginx + + 출력은 다음과 유사하다. + + NAME READY STATUS RESTARTS AGE + nginx-deployment-1771418926-7o5ns 1/1 Running 0 16h + nginx-deployment-1771418926-r18az 1/1 Running 0 16h + +1. 파드에 대한 정보를 살펴본다. + + kubectl describe pod + + ``은 파드 중 하나의 이름이다. + +## 디플로이먼트 업데이트하기 + +새 YAML 파일을 적용하여 디플로이먼트를 업데이트할 수 있다. 이 YAML 파일은 +nginx 1.16.1을 사용하도록 디플로이먼트를 업데이트해야 함을 명시하고 있다. + +{{< codenew file="application/deployment-update.yaml" >}} + +1. 새 YAML 파일을 적용한다. + + kubectl apply -f https://k8s.io/examples/application/deployment-update.yaml + +1. 디플로이먼트가 새 이름으로 파드를 생성하고 이전 파드를 삭제하는 것을 확인한다. + + kubectl get pods -l app=nginx + +## 레플리카 수를 늘려 애플리케이션 확장하기 + +새 YAML 파일을 적용하여 디플로이먼트의 파드 수를 늘릴 수 있다. +이 YAML 파일은 `replicas`를 4로 설정하여 디플로이먼트에 +4개의 파드가 있어야 함을 명시하고 있다. + +{{< codenew file="application/deployment-scale.yaml" >}} + +1. 새 YAML 파일을 적용한다. + + kubectl apply -f https://k8s.io/examples/application/deployment-scale.yaml + +1. 디플로이먼트에 4개의 파드가 있는지 확인한다. + + kubectl get pods -l app=nginx + + 출력은 다음과 유사하다. + + NAME READY STATUS RESTARTS AGE + nginx-deployment-148880595-4zdqq 1/1 Running 0 25s + nginx-deployment-148880595-6zgi1 1/1 Running 0 25s + nginx-deployment-148880595-fxcez 1/1 Running 0 2m + nginx-deployment-148880595-rwovn 1/1 Running 0 2m + +## 디플로이먼트 삭제하기 + +이름으로 디플로이먼트를 삭제한다. + + kubectl delete deployment nginx-deployment + +## ReplicationControllers -- 예전 방식 + +애플리케이션을 복제하여 생성하는 기본적인 방법은 내부적으로 레플리카셋(ReplicaSet)을 활용하는 디플로이먼트를 +사용하는 것이다. 쿠버네티스에 디플로이먼트 및 레플리카셋이 도입되기 전에는 +[레플리케이션컨트롤러(ReplicationController)](/ko/docs/concepts/workloads/controllers/replicationcontroller/)를 사용하여 복제 애플리케이션을 +구성했었다. + + + + +## {{% heading "whatsnext" %}} + + +* [디플로이먼트 오브젝트](/ko/docs/concepts/workloads/controllers/deployment/)에 대해 더 배워보기 + + + + diff --git a/content/ko/examples/application/deployment-scale.yaml b/content/ko/examples/application/deployment-scale.yaml new file mode 100644 index 0000000000..01fe96d845 --- /dev/null +++ b/content/ko/examples/application/deployment-scale.yaml @@ -0,0 +1,19 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + selector: + matchLabels: + app: nginx + replicas: 4 # Update the replicas from 2 to 4 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 diff --git a/content/ko/examples/application/deployment-update.yaml b/content/ko/examples/application/deployment-update.yaml new file mode 100644 index 0000000000..1c0b9d1ab8 --- /dev/null +++ b/content/ko/examples/application/deployment-update.yaml @@ -0,0 +1,19 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + selector: + matchLabels: + app: nginx + replicas: 2 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.16.1 # Update the version of nginx from 1.14.2 to 1.16.1 + ports: + - containerPort: 80 From 26898c2495aeaed945f01f3170de4b7bf4a8c2d4 Mon Sep 17 00:00:00 2001 From: Andy Chen <42681629+aychen99@users.noreply.github.com> Date: Thu, 7 Oct 2021 23:41:58 -0700 Subject: [PATCH 149/249] Update zh basic-stateful-set.md with upstream --- .../docs/tutorials/stateful-application/basic-stateful-set.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md b/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md index 07f6302eae..02880bfbc8 100644 --- a/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md +++ b/content/zh/docs/tutorials/stateful-application/basic-stateful-set.md @@ -38,7 +38,7 @@ following Kubernetes concepts. * [Cluster DNS](/zh/docs/concepts/services-networking/dns-pod-service/) * [Headless Services](/zh/docs/concepts/services-networking/service/#headless-services) * [PersistentVolumes](/zh/docs/concepts/storage/persistent-volumes/) -* [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/persistent-volume-provisioning/) +* [PersistentVolume Provisioning](https://github.com/kubernetes/examples/tree/master/staging/persistent-volume-provisioning/) * [StatefulSets](/zh/docs/concepts/workloads/controllers/statefulset/) * [kubectl CLI](/zh/docs/user-guide/kubectl/) From d253d7e5afb76c7ca8cc13b487fe9a2a236699ef Mon Sep 17 00:00:00 2001 From: seokho-son Date: Fri, 8 Oct 2021 15:37:10 +0900 Subject: [PATCH 150/249] Update outdated files in dev-1.22-ko.1 (D1-D2) --- .../kubelet-garbage-collection.md | 12 ++++++++++++ .../workloads/controllers/garbage-collection.md | 12 ++++++++++++ 2 files changed, 24 insertions(+) diff --git a/content/ko/docs/concepts/cluster-administration/kubelet-garbage-collection.md b/content/ko/docs/concepts/cluster-administration/kubelet-garbage-collection.md index c64dd127b3..6db0871e48 100644 --- a/content/ko/docs/concepts/cluster-administration/kubelet-garbage-collection.md +++ b/content/ko/docs/concepts/cluster-administration/kubelet-garbage-collection.md @@ -6,6 +6,18 @@ weight: 70 + +{{< note >}} +이 한글 문서는 더 이상 관리되지 않습니다. + +이 문서의 기반이 된 영어 원문은 삭제되었으며, +[Garbage Collection](/docs/concepts/architecture/garbage-collection/)에 병합되었습니다. + +[Garbage Collection](/docs/concepts/architecture/garbage-collection/)의 한글화가 완료되면, +이 문서는 삭제될 수 있습니다. +{{< /note >}} + + 가비지 수집은 사용되지 않는 [이미지](/ko/docs/concepts/containers/#컨테이너-이미지)들과 [컨테이너](/ko/docs/concepts/containers/)들을 정리하는 kubelet의 유용한 기능이다. Kubelet은 diff --git a/content/ko/docs/concepts/workloads/controllers/garbage-collection.md b/content/ko/docs/concepts/workloads/controllers/garbage-collection.md index a3ed4ea68d..225ff28948 100644 --- a/content/ko/docs/concepts/workloads/controllers/garbage-collection.md +++ b/content/ko/docs/concepts/workloads/controllers/garbage-collection.md @@ -6,6 +6,18 @@ weight: 60 + +{{< note >}} +이 한글 문서는 더 이상 관리되지 않습니다. + +이 문서의 기반이 된 영어 원문은 삭제되었으며, +[Garbage Collection](/docs/concepts/architecture/garbage-collection/)에 병합되었습니다. + +[Garbage Collection](/docs/concepts/architecture/garbage-collection/)의 한글화가 완료되면, +이 문서는 삭제될 수 있습니다. +{{< /note >}} + + 쿠버네티스의 가비지 수집기는 한때 소유자가 있었지만, 더 이상 소유자가 없는 오브젝트들을 삭제하는 역할을 한다. From 704a303a4a31d04908da7e0bdfe12f63a94ad259 Mon Sep 17 00:00:00 2001 From: seokho-son Date: Wed, 6 Oct 2021 15:53:58 +0900 Subject: [PATCH 151/249] Update outdated files in dev-1.22-ko.1 (M52) --- .../tools/kubeadm/control-plane-flags.md | 171 +++++++++++++++--- 1 file changed, 147 insertions(+), 24 deletions(-) diff --git a/content/ko/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md b/content/ko/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md index cae9a85b0a..f1c905c982 100644 --- a/content/ko/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md +++ b/content/ko/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md @@ -1,78 +1,108 @@ --- -reviewers: -title: kubeadm으로 컨트롤 플레인 사용자 정의하기 + + +title: kubeadm API로 컴포넌트 사용자 정의하기 content_type: concept weight: 40 --- +이 페이지는 kubeadm이 배포하는 컴포넌트(component)들을 사용자 정의하는 방법을 다룬다. 컨트롤 플레인 컴포넌트에 +대해서는 `ClusterConfiguration` 구조에서 플래그를 사용하거나 노드당 패치를 사용할 수 있다. kubelet과 +kube-proxy의 경우, `KubeletConfiguration`과 `KubeProxyConfiguration`을 각각 사용할 수 있다. + +이 모든 옵션이 kubeadm 구성 API를 통해 가용하다. +구성의 각 필드 상세 사항은 +[API 참조 페이지](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3)에서 찾아볼 수 있다. + +{{< note >}} +kubeadm의 CoreDNS 디플로이먼트 사용자 정의는 현재 제공되지 않는다. +`kube-system/coredns` {{< glossary_tooltip text="컨피그맵" term_id="configmap" >}}을 수동으로 +패치하고, 그 이후에 CoreDNS {{< glossary_tooltip text="파드" term_id="pod" >}}를 다시 생성해야 한다. 또는, +기본 CoreDNS 디플로이먼트를 생략하고 자체 변형(variant)을 배포할 수 있다. +더 자세한 사항은 [kubeadm에서 초기화 단계 사용하기](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases)을 참고한다. +{{< /note >}} + + + {{< feature-state for_k8s_version="v1.12" state="stable" >}} -kubeadm의 `ClusterConfiguration` 오브젝트는 API 서버, 컨트롤러매니저, 스케줄러와 같은 컨트롤 플레인 구성요소에 전달되는 -기본 플래그 `extraArgs` 필드를 노출한다. 이 구성요소는 다음 필드를 사용하도록 정의되어 있다. +## `ClusterConfiguration`의 플래그로 컨트롤 플레인 사용자 정의하기 + +kubeadm의 `ClusterConfiguration` 오브젝트는 API 서버, 컨트롤러매니저, 스케줄러, Etcd와 같은 컨트롤 플레인 컴포넌트에 전달되는 +기본 플래그를 사용자가 덮어쓸 수 있도록 노출한다. +이 컴포넌트는 다음 구조체를 사용하여 정의된다. - `apiServer` - `controllerManager` - `scheduler` +- `etcd` -`extraArgs` 필드는 `key: value` 쌍으로 구성되어 있다. 컨트롤 플레인 구성요소를 위한 플래그를 대체하려면 다음을 수행한다. +이 구조체들은 공통 필드인 `extraArgs`를 포함하며, 이 필드는 `키: 값` 쌍으로 구성된다. +컨트롤 플레인 컴포넌트를 위한 플래그를 덮어쓰려면 다음을 수행한다. -1. 사용자 구성에서 적절한 필드를 추가한다. -2. 필드에 대체할 플래그를 추가한다. +1. 사용자 구성에 적절한 `extraArgs` 필드를 추가한다. +2. `extraArgs` 필드에 플래그를 추가한다. 3. `kubeadm init`에 `--config ` 파라미터를 추가해서 실행한다. -각 필드의 구성에서 자세한 정보를 보려면, -[API 참고 문서](/docs/reference/config-api/kubeadm-config.v1beta2/)에서 확인해 볼 수 있다. - {{< note >}} -`kubeadm config print init-defaults`를 실행하고 원하는 파일에 출력을 저장하여 기본값인 `ClusterConfiguration` 오브젝트를 생성할 수 있다. +`kubeadm config print init-defaults`를 실행하고 원하는 파일에 출력을 +저장하여 기본값들로 구성된 `ClusterConfiguration` 오브젝트를 생성할 수 있다. {{< /note >}} +{{< note >}} +`ClusterConfiguration` 오브젝트는 현재 kubeadm 클러스터에서 전역(global)으로 사용된다. 즉, 사용자가 추가하는 모든 플래그는 +다른 노드에 있는 동일한 컴포넌트에도 모두 적용될 것이다. 다른 노드에서 +컴포넌트별로 개별 구성을 적용하려면 [패치](#patches)를 사용하면 된다. +{{< /note >}} +{{< note >}} +플래그(키)를 복제하거나 동일한 플래그 `--foo`를 여러 번 전달하는 것은 현재 지원하지 않는다. +이 문제를 해결하려면 [패치](#patches)를 사용해야 한다. +{{< /note >}} - - -## APIServer 플래그 +### APIServer 플래그 자세한 내용은 [kube-apiserver 레퍼런스 문서](/docs/reference/command-line-tools-reference/kube-apiserver/)를 확인한다. -예시: +사용 예시: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 apiServer: extraArgs: - advertise-address: 192.168.0.103 anonymous-auth: "false" enable-admission-plugins: AlwaysPullImages,DefaultStorageClass audit-log-path: /home/johndoe/audit.log ``` -## 컨트롤러매니저 플래그 +### 컨트롤러매니저 플래그 자세한 내용은 [kube-controller-manager 레퍼런스 문서](/docs/reference/command-line-tools-reference/kube-controller-manager/)를 확인한다. -예시: +사용 예시: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 controllerManager: extraArgs: cluster-signing-key-file: /home/johndoe/keys/ca.key - bind-address: 0.0.0.0 deployment-controller-sync-period: "50" ``` -## 스케줄러 플래그 +### 스케줄러 플래그 자세한 내용은 [kube-scheduler 레퍼런스 문서](/docs/reference/command-line-tools-reference/kube-scheduler/)를 확인한다. -예시: +사용 예시: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 scheduler: @@ -85,3 +115,96 @@ scheduler: readOnly: true pathType: "File" ``` + +### Etcd 플래그 + +자세한 사항은 [etcd 서버 문서](https://etcd.io/docs/)를 확인한다. + +사용 예시: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: ClusterConfiguration +etcd: + local: + extraArgs: + election-timeout: 1000 +``` + +## 패치를 통해 컨트롤 플레인 사용자 정의하기 {#patches} + +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + +Kubeadm을 사용하면 패치 파일이 있는 디렉토리를 개별 노드에 대한 `InitConfiguration`과 `JoinConfiguration`에 +전달할 수 있다. 이 패치는 컨트롤 플레인 컴포넌트 메니패스트가 디스크에 기록되기 전에 +최종 사용자 정의 단계로 사용될 수 있다. + +`--config `을 사용하여 이 파일을 `kubeadm init`에 전달할 수 있다. + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: InitConfiguration +nodeRegistration: + patches: + directory: /home/user/somedir +``` + +{{< note >}} +`kubeadm init`의 경우, `---`로 구분된 `ClusterConfiguration`과 `InitConfiguration`을 모두 +포함하는 파일을 전달할 수 있다. +{{< /note >}} + +`--config `을 사용하여 이 파일을 `kubeadm join`에 전달할 수 있다. + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: JoinConfiguration +nodeRegistration: + patches: + directory: /home/user/somedir +``` + +디렉토리는 `target[suffix][+patchtype].extension` 형태의 파일을 포함해야 한다. +예를 들면, `kube-apiserver0+merge.yaml` 또는 단순히 `etcd.json`의 형태이다. + +- `target`은 `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` 그리고 `etcd` 중 하나가 될 수 있다. +- `patchtype`은 `strategic`, `merge` 그리고 `json` 중 하나가 될 수 있으며 +[kubectl에서 지원하는](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch) 패치 형식을 준수해야 한다. +`patchtype`의 기본값은 `strategic`이다. +- `extension`은 `json` 또는 `yaml` 중 하나여야 한다. +- `suffix`는 어떤 패치가 먼저 적용되는지를 결정하는 데 사용할 수 있는 영숫자 형태의 +선택적 문자열이다. + +{{< note >}} +`kubeadm upgrade`를 사용하여 kubeadm 노드를 업그레이드하는 경우, 업그레이드 이후에도 +사용자 정의를 유지하려면 동일한 패치를 다시 제공해야 한다. 이는 동일한 디렉토리로 지정된 `--patches` +플래그를 사용하여 처리할 수 있다. `kubeadm upgrade`는 동일 목적으로 재사용할 수 있는 구성 +API 구조를 현재는 지원하지 않는다. +{{< /note >}} + +## kubelet 사용자 정의하기 + +kubelet을 사용자 정의하려면, `KubeletConfiguration`을 동일한 구성 파일 내에서 `---`로 구분된 `ClusterConfiguration`이나 `InitConfiguration` 다음에 추가하면 +된다. 그런 다음 `kubeadm init`에 해당 파일을 전달한다. + +{{< note >}} +kubeadm은 클러스터의 모든 노드에 동일한 `KubeletConfiguration`을 적용한다. 노드별 설정을 +적용하려면 kubelet 플래그를 덮어쓰기(overrides)로 사용하여, `InitConfiguration` 및 +`JoinConfiguration` 모두에서 지원되는 `nodeRegistration.kubeletExtraArgs`에 전달할 수 있다. +일부 kubelet 플래그는 더 이상 사용되지 않는다(deprecated). 따라서 사용하기 전에 [kubelet 참조 문서](/docs/reference/command-line-tools-reference/kubelet)를 통해 +상태를 확인해야 한다. +{{< /note >}} + +자세한 사항은 [kubeadm을 통해 클러스터의 각 kubelet 구성하기](/docs/setup/production-environment/tools/kubeadm/kubelet-integration)에서 살펴본다. + +## kube-proxy 사용자 정의하기 + +kube-proxy를 사용자 정의하려면, `KubeProxyConfiguration`을 `---`로 구분된 `ClusterConfiguration`이나 `InitConfiguration` +다음에 두고 `kubeadm init`에 전달하면 된다. + +자세한 사항은 [API 참조 페이지](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3)에서 살펴볼 수 있다. + +{{< note >}} +kubeadm은 kube-proxy를 {{< glossary_tooltip text="데몬셋" term_id="daemonset" >}}으로 배포한다. 이것은 +`KubeProxyConfiguration`이 클러스터의 모든 kube-proxy 인스턴스에 적용된다는 것을 의미한다. +{{< /note >}} From 4da4d1bb0a69f28cdad702ae47e7492d561383f0 Mon Sep 17 00:00:00 2001 From: Riita <42636694+riita10069@users.noreply.github.com> Date: Fri, 8 Oct 2021 17:30:40 +0900 Subject: [PATCH 152/249] Update authentication.md --- .../docs/reference/access-authn-authz/authentication.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/ja/docs/reference/access-authn-authz/authentication.md b/content/ja/docs/reference/access-authn-authz/authentication.md index 14875c8229..0aee169573 100644 --- a/content/ja/docs/reference/access-authn-authz/authentication.md +++ b/content/ja/docs/reference/access-authn-authz/authentication.md @@ -63,7 +63,7 @@ openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj "/CN=jbeda/O=app1/O=app これにより、"app1"と"app2"の2つのグループに属するユーザー名"jbeda"の証明書署名要求が作成されます。 -クライアント証明書の生成方法については、[証明書の管理](/docs/concepts/cluster-administration/certificates/)を参照してください。 +クライアント証明書の生成方法については、[証明書の管理](/ja/docs/concepts/cluster-administration/certificates/)を参照してください。 ### 静的なトークンファイル @@ -314,7 +314,7 @@ Webhook認証は、Bearerトークンを検証するためのフックです。 * `--authentication-token-webhook-config-file`: リモートのWebhookサービスへのアクセス方法を記述した設定ファイルです * `--authentication-token-webhook-cache-ttl`: 認証をキャッシュする時間を決定します。デフォルトは2分です -設定ファイルは、[kubeconfig](/docs/concepts/configuration/organize-cluster-access-kubeconfig/)のファイル形式を使用します。 +設定ファイルは、[kubeconfig](/ja/docs/concepts/configuration/organize-cluster-access-kubeconfig/)のファイル形式を使用します。 ファイル内で、`clusters`はリモートサービスを、`users`はAPIサーバーのWebhookを指します。例えば、以下のようになります。 ```yaml @@ -347,7 +347,7 @@ contexts: クライアントが[上記](#putting-a-bearer-token-in-a-request)のようにBearerトークンを使用してAPIサーバーとの認証を試みた場合、認証Webhookはトークンを含むJSONでシリアライズされた`authentication.k8s.io/v1beta1` `TokenReview`オブジェクトをリモートサービスにPOSTします。Kubernetesはそのようなヘッダーが不足しているリクエストを作成しようとはしません。 -Webhook APIオブジェクトは、他のKubernetes APIオブジェクトと同じように、[Versioning Compatibility Rule](/docs/concepts/overview/kubernetes-api/)に従うことに注意してください。実装者は、ベータオブジェクトで保証される互換性が緩いことに注意し、正しいデシリアライゼーションが使用されるようにリクエストの"apiVersion"フィールドを確認する必要があります。さらにAPIサーバーは、API拡張グループ`authentication.k8s.io/v1beta1`を有効にしなければなりません(`--runtime config=authentication.k8s.io/v1beta1=true`)。 +Webhook APIオブジェクトは、他のKubernetes APIオブジェクトと同じように、[Versioning Compatibility Rule](/ja/docs/concepts/overview/kubernetes-api/)に従うことに注意してください。実装者は、ベータオブジェクトで保証される互換性が緩いことに注意し、正しいデシリアライゼーションが使用されるようにリクエストの"apiVersion"フィールドを確認する必要があります。さらにAPIサーバーは、API拡張グループ`authentication.k8s.io/v1beta1`を有効にしなければなりません(`--runtime config=authentication.k8s.io/v1beta1=true`)。 POSTボディは、以下の形式になります。 @@ -600,7 +600,7 @@ rules: ### 設定 -クレデンシャルプラグインの設定は、userフィールドの一部として[kubectlの設定ファイル](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/)で行います。 +クレデンシャルプラグインの設定は、userフィールドの一部として[kubectlの設定ファイル](/ja/docs/tasks/access-application-cluster/configure-access-multiple-clusters/)で行います。 ```yaml apiVersion: v1 From 5919a94075c9e2d42c5358fcecdb805e90c79a8e Mon Sep 17 00:00:00 2001 From: Riita <42636694+riita10069@users.noreply.github.com> Date: Fri, 8 Oct 2021 17:44:20 +0900 Subject: [PATCH 153/249] Update ingress.md --- content/ja/docs/concepts/services-networking/ingress.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/ja/docs/concepts/services-networking/ingress.md b/content/ja/docs/concepts/services-networking/ingress.md index 97d18640b8..be1b51ad84 100644 --- a/content/ja/docs/concepts/services-networking/ingress.md +++ b/content/ja/docs/concepts/services-networking/ingress.md @@ -18,7 +18,7 @@ weight: 40 * ノード: Kubernetes内のワーカーマシンで、クラスターの一部です。 * クラスター: Kubernetesによって管理されているコンテナ化されたアプリケーションを実行させるノードの集合です。この例や、多くのKubernetesによるデプロイでは、クラスター内のノードはインターネットに公開されていません。 * エッジルーター: クラスターでファイアウォールのポリシーを強制するルーターです。クラウドプロバイダーが管理するゲートウェイや、物理的なハードウェアの一部である場合もあります。 -* クラスターネットワーク: 物理的または論理的な繋がりの集合で、Kubernetesの[ネットワークモデル](/docs/concepts/cluster-administration/networking/)によって、クラスター内でのコミュニケーションを司るものです。 +* クラスターネットワーク: 物理的または論理的な繋がりの集合で、Kubernetesの[ネットワークモデル](/ja/docs/concepts/cluster-administration/networking/)によって、クラスター内でのコミュニケーションを司るものです。 * Service: {{< glossary_tooltip text="ラベル" term_id="label" >}}セレクターを使ったPodの集合を特定するKubernetes {{< glossary_tooltip term_id="service" >}}です。特に指定がない限り、Serviceはクラスターネットワーク内でのみ疎通可能な仮想IPを持つものとして扱われます。 ## Ingressとは何か @@ -66,7 +66,7 @@ Ingressリソースの最小構成の例は以下のとおりです。 {{< codenew file="service/networking/minimal-ingress.yaml" >}} -他の全てのKubernetesリソースと同様に、Ingressには`apiVersion`、`kind`や`metadata`フィールドが必要です。Ingressオブジェクトの名前は、有効な[DNSサブドメイン名](/ja/docs/concepts/overview/working-with-objects/names#dns-subdomain-names)である必要があります。設定ファイルに関する一般的な情報は、[アプリケーションのデプロイ](/ja/docs/tasks/run-application/run-stateless-application-deployment/)、[コンテナの設定](/docs/tasks/configure-pod-container/configure-pod-configmap/)、[リソースの管理](/docs/concepts/cluster-administration/manage-deployment/)を参照してください。Ingressでは、Ingressコントローラーに依存しているいくつかのオプションの設定をするためにアノテーションを一般的に使用します。例としては、[rewrite-targetアノテーション](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md)などがあります。[Ingressコントローラー](/ja/docs/concepts/services-networking/ingress-controllers)の種類が異なれば、サポートするアノテーションも異なります。サポートされているアノテーションについて学ぶためには、使用するIngressコントローラーのドキュメントを確認してください。 +他の全てのKubernetesリソースと同様に、Ingressには`apiVersion`、`kind`や`metadata`フィールドが必要です。Ingressオブジェクトの名前は、有効な[DNSサブドメイン名](/ja/docs/concepts/overview/working-with-objects/names#dns-subdomain-names)である必要があります。設定ファイルに関する一般的な情報は、[アプリケーションのデプロイ](/ja/docs/tasks/run-application/run-stateless-application-deployment/)、[コンテナの設定](/ja/docs/tasks/configure-pod-container/configure-pod-configmap/)、[リソースの管理](/ja/docs/concepts/cluster-administration/manage-deployment/)を参照してください。Ingressでは、Ingressコントローラーに依存しているいくつかのオプションの設定をするためにアノテーションを一般的に使用します。例としては、[rewrite-targetアノテーション](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md)などがあります。[Ingressコントローラー](/ja/docs/concepts/services-networking/ingress-controllers)の種類が異なれば、サポートするアノテーションも異なります。サポートされているアノテーションについて学ぶためには、使用するIngressコントローラーのドキュメントを確認してください。 Ingress [Spec](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status)は、ロードバランサーやプロキシーサーバーを設定するために必要な全ての情報を持っています。最も重要なものとして、外部からくる全てのリクエストに対して一致したルールのリストを含みます。IngressリソースはHTTP(S)トラフィックに対してのルールのみサポートしています。 @@ -346,7 +346,7 @@ IngressでこのSecretを参照すると、クライアントとロードバラ Ingressコントローラーは、負荷分散アルゴリズムやバックエンドの重みスキームなど、すべてのIngressに適用されるいくつかの負荷分散ポリシーの設定とともにブートストラップされます。発展した負荷分散のコンセプト(例: セッションの永続化、動的重み付けなど)はIngressによってサポートされていません。代わりに、それらの機能はService用のロードバランサーを介して利用できます。 -Ingressによってヘルスチェックの機能が直接に公開されていない場合でも、Kubernetesにおいて、同等の機能を提供する[Readiness Probe](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)のようなコンセプトが存在することは注目に値します。コントローラーがどのようにヘルスチェックを行うかについては、コントローラーのドキュメントを参照してください(例えば[nginx](https://git.k8s.io/ingress-nginx/README.md)、または[GCE](https://git.k8s.io/ingress-gce/README.md#health-checks))。 +Ingressによってヘルスチェックの機能が直接に公開されていない場合でも、Kubernetesにおいて、同等の機能を提供する[Readiness Probe](/ja/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)のようなコンセプトが存在することは注目に値します。コントローラーがどのようにヘルスチェックを行うかについては、コントローラーのドキュメントを参照してください(例えば[nginx](https://git.k8s.io/ingress-nginx/README.md)、または[GCE](https://git.k8s.io/ingress-gce/README.md#health-checks))。 ## Ingressの更新 @@ -451,4 +451,4 @@ Ingressリソースを直接含まない複数の方法でサービスを公開 ## {{% heading "whatsnext" %}} * [Ingress API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#ingress-v1beta1-networking-k8s-io)について学ぶ * [Ingressコントローラー](/ja/docs/concepts/services-networking/ingress-controllers/)について学ぶ -* [MinikubeとNGINXコントローラーでIngressのセットアップを行う](/docs/tasks/access-application-cluster/ingress-minikube/) +* [MinikubeとNGINXコントローラーでIngressのセットアップを行う](/ja/docs/tasks/access-application-cluster/ingress-minikube/) From 03c87a4746b0fbc393bd27ce660e9cf17ab567ce Mon Sep 17 00:00:00 2001 From: Riita <42636694+riita10069@users.noreply.github.com> Date: Fri, 8 Oct 2021 17:48:40 +0900 Subject: [PATCH 154/249] Update guestbook.md --- .../docs/tutorials/stateless-application/guestbook.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/ja/docs/tutorials/stateless-application/guestbook.md b/content/ja/docs/tutorials/stateless-application/guestbook.md index 82e38a56d2..b9aefecfa9 100644 --- a/content/ja/docs/tutorials/stateless-application/guestbook.md +++ b/content/ja/docs/tutorials/stateless-application/guestbook.md @@ -79,7 +79,7 @@ POD-NAMEの部分を実際のPodの名前に書き換えてください。 ### RedisのマスターのServiceを作成する -ゲストブックアプリケーションは、データを書き込むためにRedisのマスターと通信する必要があります。そのためには、[Service](/docs/concepts/services-networking/service/)を適用して、トラフィックをRedisのマスターのPodへプロキシーしなければなりません。Serviceは、Podにアクセスするためのポリシーを指定します。 +ゲストブックアプリケーションは、データを書き込むためにRedisのマスターと通信する必要があります。そのためには、[Service](/ja/docs/concepts/services-networking/service/)を適用して、トラフィックをRedisのマスターのPodへプロキシーしなければなりません。Serviceは、Podにアクセスするためのポリシーを指定します。 {{< codenew file="application/guestbook/redis-master-service.yaml" >}} @@ -199,7 +199,7 @@ Deploymentはマニフェストファイル内に書かれた設定に基づい ### フロントエンドのServiceを作成する -適用した`redis-slave`および`redis-master` Serviceは、コンテナクラスター内部からのみアクセス可能です。これは、デフォルトのServiceのtypeが[ClusterIP](/docs/concepts/services-networking/service/#publishing-services-service-types)であるためです。`ClusterIP`は、Serviceが指している一連のPodに対して1つのIPアドレスを提供します。このIPアドレスはクラスター内部からのみアクセスできます。 +適用した`redis-slave`および`redis-master` Serviceは、コンテナクラスター内部からのみアクセス可能です。これは、デフォルトのServiceのtypeが[ClusterIP](/ja/docs/concepts/services-networking/service/#publishing-services-service-types)であるためです。`ClusterIP`は、Serviceが指している一連のPodに対して1つのIPアドレスを提供します。このIPアドレスはクラスター内部からのみアクセスできます。 もしゲストの人にゲストブックにアクセスしてほしいのなら、フロントエンドServiceを外部から見えるように設定しなければなりません。そうすれば、クライアントはコンテナクラスターの外部からServiceにリクエストを送れるようになります。Minikubeでは、Serviceを`NodePort`でのみ公開できます。 @@ -363,8 +363,8 @@ DeploymentとServiceを削除すると、実行中のPodも削除されます。 ## {{% heading "whatsnext" %}} -* ゲストブックアプリケーションに対する[ELKによるロギングとモニタリング](/docs/tutorials/stateless-application/guestbook-logs-metrics-with-elk/) +* ゲストブックアプリケーションに対する[ELKによるロギングとモニタリング](/ja/docs/tutorials/stateless-application/guestbook-logs-metrics-with-elk/) * [Kubernetesの基本](/ja/docs/tutorials/kubernetes-basics/)のインタラクティブチュートリアルを終わらせる -* Kubernetesを使って、[MySQLとWordpressのためにPersistent Volume](/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/#visit-your-new-wordpress-blog)を使用したブログを作成する +* Kubernetesを使って、[MySQLとWordpressのためにPersistent Volume](/ja/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/#visit-your-new-wordpress-blog)を使用したブログを作成する * [サービスとアプリケーションの接続](/ja/docs/concepts/services-networking/connect-applications-service/)についてもっと読む -* [リソースの管理](/docs/concepts/cluster-administration/manage-deployment/#using-labels-effectively)についてもっと読む +* [リソースの管理](/ja/docs/concepts/cluster-administration/manage-deployment/#using-labels-effectively)についてもっと読む From 48808351dd68adb87646747049ca062565134945 Mon Sep 17 00:00:00 2001 From: Mauren Berti Date: Fri, 8 Oct 2021 14:13:52 -0400 Subject: [PATCH 155/249] Add explanation on secret encryption. --- content/pt-br/docs/reference/glossary/secret.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/pt-br/docs/reference/glossary/secret.md b/content/pt-br/docs/reference/glossary/secret.md index 9eead5cbf0..abcd1bb421 100644 --- a/content/pt-br/docs/reference/glossary/secret.md +++ b/content/pt-br/docs/reference/glossary/secret.md @@ -15,4 +15,4 @@ tags: -Permite mais controle com relação a como as informações sensíveis são armazenadas e reduz o risco de exposição acidental, incluindo [encriptação](/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted) em disco. Um {{< glossary_tooltip text="Pod" term_id="pod" >}} pode referenciar o Secret como um arquivo em um volume montado, ou o kubelet pode usar este dado quando baixa imagens para um pod. Secrets são úteis para dados confidenciais, enquanto [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) são úteis para dados não-confidenciais. +Permite mais controle com relação a como as informações sensíveis são armazenadas e reduz o risco de exposição acidental. Os valores dos Secrets são codificados como strings no formato base64, mas não encriptados por padrão. É possível configurar o cluster para realizar a [encriptação em disco](/docs/tasks/administer-cluster/encrypt-data/#ensure-all-secrets-are-encrypted) de Secrets. Um {{< glossary_tooltip text="Pod" term_id="pod" >}} pode referenciar o Secret como um arquivo em um volume montado, ou o kubelet pode usar este dado quando baixa imagens para um pod. Secrets são úteis para dados confidenciais, enquanto [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/) são úteis para dados não-confidenciais. From 4d5af741f846320a73f20110178a7367af8deb6a Mon Sep 17 00:00:00 2001 From: Diego Servin Date: Fri, 8 Oct 2021 17:45:22 -0500 Subject: [PATCH 156/249] [es] Add content/es/docs/tasks/tools/wg.md --- content/es/docs/reference/glossary/wg.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 content/es/docs/reference/glossary/wg.md diff --git a/content/es/docs/reference/glossary/wg.md b/content/es/docs/reference/glossary/wg.md new file mode 100644 index 0000000000..4ad7ef451e --- /dev/null +++ b/content/es/docs/reference/glossary/wg.md @@ -0,0 +1,19 @@ +--- +title: WG (working group) +id: wg +date: 2018-04-12 +full_link: https://github.com/kubernetes/community/blob/master/sig-list.md#master-working-group-list +short_description: > + Facilita la discusión y/o la implementación de un proyecto que sea efímero, corto o desacoplado para un comité, {{< glossary_tooltip text="SIG" term_id="sig" >}}, o esfuerzo SIG cruzado. + +aka: +tags: +- community +--- + Facilita la discusión y/o la implementación de un proyecto que sea efímero, corto o desacoplado para un comité, {{< glossary_tooltip text="SIG" term_id="sig" >}}, o trabajo SIG cruzado. + + + +Los grupos de trabajo son una forma de organizar personas para completar una tarea discreta. + +Para más información, consulta el repositorio [kubernetes/community](https://github.com/kubernetes/community) y la lista de los [SIGs y Grupos de Trabajo (WGs).](https://github.com/kubernetes/community/blob/master/sig-list.md). From 21fe964b9da588370d173864ee9d473a802601d2 Mon Sep 17 00:00:00 2001 From: Diego Servin Hamden <54914983+DiegoServinHdn@users.noreply.github.com> Date: Fri, 8 Oct 2021 19:38:47 -0500 Subject: [PATCH 157/249] Update content/es/docs/reference/glossary/wg.md Co-authored-by: Victor Morales --- content/es/docs/reference/glossary/wg.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/es/docs/reference/glossary/wg.md b/content/es/docs/reference/glossary/wg.md index 4ad7ef451e..85c4eb7bd7 100644 --- a/content/es/docs/reference/glossary/wg.md +++ b/content/es/docs/reference/glossary/wg.md @@ -10,7 +10,7 @@ aka: tags: - community --- - Facilita la discusión y/o la implementación de un proyecto que sea efímero, corto o desacoplado para un comité, {{< glossary_tooltip text="SIG" term_id="sig" >}}, o trabajo SIG cruzado. + Facilita la discusión y/o la implementación de un proyecto que sea efímero, corto o desacoplado para un comité, {{< glossary_tooltip text="SIG" term_id="sig" >}}, o un esfuerzo entre varios SIGs. From 95a772fea77f34999870bf8937905d46beae03b5 Mon Sep 17 00:00:00 2001 From: Jizu Sun Date: Sat, 9 Oct 2021 10:47:55 +0800 Subject: [PATCH 158/249] fix link --- .../docs/concepts/overview/working-with-objects/finalizers.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/overview/working-with-objects/finalizers.md b/content/en/docs/concepts/overview/working-with-objects/finalizers.md index 9516b935c9..98c4063ff0 100644 --- a/content/en/docs/concepts/overview/working-with-objects/finalizers.md +++ b/content/en/docs/concepts/overview/working-with-objects/finalizers.md @@ -44,7 +44,8 @@ and the controller deletes the volume. ## Owner references, labels, and finalizers {#owners-labels-finalizers} -Like {{}}, [owner references](/concepts/overview/working-with-objects/owners-dependents/) +Like {{}}, [owner references] +(/docs/concepts/overview/working-with-objects/owners-dependents/) describe the relationships between objects in Kubernetes, but are used for a different purpose. When a {{}} manages objects From 5c0dd0414b2636cf47cbd24f7b1c638c60494df1 Mon Sep 17 00:00:00 2001 From: Jizu Sun Date: Sat, 9 Oct 2021 10:50:09 +0800 Subject: [PATCH 159/249] remove blank --- .../docs/concepts/overview/working-with-objects/finalizers.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/docs/concepts/overview/working-with-objects/finalizers.md b/content/en/docs/concepts/overview/working-with-objects/finalizers.md index 98c4063ff0..c9985cc3ff 100644 --- a/content/en/docs/concepts/overview/working-with-objects/finalizers.md +++ b/content/en/docs/concepts/overview/working-with-objects/finalizers.md @@ -44,8 +44,7 @@ and the controller deletes the volume. ## Owner references, labels, and finalizers {#owners-labels-finalizers} -Like {{}}, [owner references] -(/docs/concepts/overview/working-with-objects/owners-dependents/) +Like {{}}, [owner references](/docs/concepts/overview/working-with-objects/owners-dependents/) describe the relationships between objects in Kubernetes, but are used for a different purpose. When a {{}} manages objects From c599644f693a2120a1ebdba84351455846046caa Mon Sep 17 00:00:00 2001 From: howieyuen Date: Thu, 30 Sep 2021 18:02:45 +0800 Subject: [PATCH 160/249] [zh] sync storage-classes.md & volumes.md in storage --- .../docs/concepts/storage/storage-classes.md | 85 ++++++++- content/zh/docs/concepts/storage/volumes.md | 161 ++++++++---------- 2 files changed, 153 insertions(+), 93 deletions(-) diff --git a/content/zh/docs/concepts/storage/storage-classes.md b/content/zh/docs/concepts/storage/storage-classes.md index 28ca72dd42..539f444168 100644 --- a/content/zh/docs/concepts/storage/storage-classes.md +++ b/content/zh/docs/concepts/storage/storage-classes.md @@ -120,7 +120,7 @@ for provisioning PVs. This field must be specified. | Glusterfs | ✓ | [Glusterfs](#glusterfs) | | iSCSI | - | - | | Quobyte | ✓ | [Quobyte](#quobyte) | -| NFS | - | - | +| NFS | - | [NFS](#nfs) | | RBD | ✓ | [Ceph RBD](#ceph-rbd) | | VsphereVolume | ✓ | [vSphere](#vsphere) | | PortworxVolume | ✓ | [Portworx Volume](#portworx-卷) | @@ -316,6 +316,43 @@ to see its supported topology keys and examples. 动态配置和预先创建的 PV 也支持 [CSI卷](/zh/docs/concepts/storage/volumes/#csi)， 但是你需要查看特定 CSI 驱动程序的文档以查看其支持的拓扑键名和例子。 +{{< note >}} + + 如果你选择使用 `WaitForFirstConsumer`，请不要在 Pod 规约中使用 `nodeName` 来指定节点亲和性。 + 如果在这种情况下使用 `nodeName`，Pod 将会绕过调度程序，PVC 将停留在 `pending` 状态。 + + 相反，在这种情况下，你可以使用节点选择器作为主机名，如下所示 + +{{< /note >}} + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: task-pv-pod +spec: + nodeSelector: + kubernetes.io/hostname: kube-01 + volumes: + - name: task-pv-storage + persistentVolumeClaim: + claimName: task-pv-claim + containers: + - name: task-pv-container + image: nginx + ports: + - containerPort: 80 + name: "http-server" + volumeMounts: + - mountPath: "/usr/share/nginx/html" + name: task-pv-storage +``` + @@ -647,6 +684,41 @@ parameters: 当动态制备持久卷时，Gluster 插件自动创建名为 `gluster-dynamic-` 的端点和无头服务。在 PVC 被删除时动态端点和无头服务会自动被删除。 +### NFS + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: example-nfs +provisioner: example.com/external-nfs +parameters: + server: nfs-server.example.com + path: /share + readOnly: false +``` + + +* `server`：NFS 服务器的主机名或 IP 地址。 +* `path`：NFS 服务器导出的路径。 +* `readOnly`：是否将存储挂载为只读的标志（默认为 false）。 + + +Kubernetes 不包含内部 NFS 驱动。你需要使用外部驱动为 NFS 创建 StorageClass。 +这里有些例子： +* [NFS Ganesha 服务器和外部驱动](https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner) +* [NFS subdir 外部驱动](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) + + ### OpenStack Cinder ```yaml @@ -912,6 +984,17 @@ parameters: ### Quobyte +{{< feature-state for_k8s_version="v1.22" state="deprecated" >}} + + +Quobyte 树内（in-tree）存储插件已弃用， +你可以在 Quobyte CSI 仓库中找到用于树外（out-of-tree）Quobyte 插件的 `StorageClass` +[示例](https://github.com/quobyte/quobyte-csi/blob/master/example/StorageClass.yaml)。 + ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass diff --git a/content/zh/docs/concepts/storage/volumes.md b/content/zh/docs/concepts/storage/volumes.md index 81e4ebf738..e8ec03fdc1 100644 --- a/content/zh/docs/concepts/storage/volumes.md +++ b/content/zh/docs/concepts/storage/volumes.md @@ -219,23 +219,22 @@ beta features must be enabled. -如欲禁止 `awsElasticBlockStore` 存储插件被控制器管理器和 kubelet -组件加载，可将 `CSIMigrationAWSComplete` 特性门控设置为 `true`。此特性要求在 -集群中所有工作节点上安装 `ebs.csi.aws.com` 容器存储接口驱动。 +要禁止控制器管理器和 kubelet 加载 `awsElasticBlockStore` 存储插件， +请将 `InTreePluginAWSUnregister` 标志设置为 `true`。 ### azureDisk {#azuredisk} `azureDisk` 卷类型用来在 Pod 上挂载 Microsoft Azure [数据盘（Data Disk）](https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-about-disks-vhds/) 。 -若需了解更多详情，请参考 [`azureDisk` 卷插件](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/azure_disk/README.md)。 +若需了解更多详情，请参考 [`azureDisk` 卷插件](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_disk/README.md)。 `azureFile` 卷类型用来在 Pod 上挂载 Microsoft Azure 文件卷（File Volume）（SMB 2.1 和 3.0）。 -更多详情请参考 [`azureFile` 卷插件](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/azure_file/README.md)。 +更多详情请参考 [`azureFile` 卷插件](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_file/README.md)。 -更多信息请参考 [CephFS 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/cephfs/)。 +更多信息请参考 [CephFS 示例](https://github.com/kubernetes/examples/tree/master/volumes/cephfs/)。 ### cinder {#cinder} @@ -589,15 +588,15 @@ targetWWNs expect that those WWNs are from multi-path connections. -{{< caution >}} +{{< note >}} 你必须配置 FC SAN Zoning，以便预先向目标 WWN 分配和屏蔽这些 LUN（卷）， 这样 Kubernetes 主机才可以访问它们。 -{{< /caution >}} +{{< /note >}} -更多详情请参考 [FC 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/fibre_channel)。 +更多详情请参考 [FC 示例](https://github.com/kubernetes/examples/tree/master/staging/volumes/fibre_channel)。 -更多详情请参考 [Flocker 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/flocker)。 +更多详情请参考 [Flocker 示例](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker)。 +#### GCE CSI 迁移完成 + +{{< feature-state for_k8s_version="v1.21" state="alpha" >}} + +要禁止控制器管理器和 kubelet 加载 `gcePersistentDisk` 存储插件， +请将 `InTreePluginGCEUnregister` 标志设置为 `true`。 + @@ -883,12 +898,28 @@ You must have your own GlusterFS installation running before you can use it. {{< /note >}} -更多详情请参考 [GlusterFS 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/glusterfs)。 +更多详情请参考 [GlusterFS 示例](https://github.com/kubernetes/examples/tree/master/volumes/glusterfs)。 ### hostPath +{{< warning >}} + +HostPath 卷存在许多安全风险，最佳做法是尽可能避免使用 HostPath。 +当必须使用 HostPath 卷时，它的范围应仅限于所需的文件或目录，并以只读方式挂载。 + +如果通过 AdmissionPolicy 限制 HostPath 对特定目录的访问， +则必须要求 `volumeMounts` 使用 `readOnly` 挂载以使策略生效。 +{{< /warning >}} + 当使用这种类型的卷时要小心，因为： +* HostPath 卷可能会暴露特权系统凭据（例如 Kubelet）或特权 API（例如容器运行时套接字）， + 可用于容器逃逸或攻击集群的其他部分。 * 具有相同配置（例如基于同一 PodTemplate 创建）的多个 Pod 会由于节点上文件的不同 而在不同节点上有不同的行为。 * 下层主机上创建的文件或目录只能由 root 用户写入。你需要在 @@ -1065,9 +1101,9 @@ iSCSI 的一个特点是它可以同时被多个用户以只读方式挂载。 不幸的是，iSCSI 卷只能由单个使用者以读写模式挂载。不允许同时写入。 -更多详情请参考 [iSCSI 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/iscsi)。 +更多详情请参考 [iSCSI 示例](https://github.com/kubernetes/examples/tree/master/volumes/iscsi)。 -要了解更多详情请参考 [NFS 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/nfs)。 +要了解更多详情请参考 [NFS 示例](https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs)。 ### persistentVolumeClaim {#persistentvolumeclaim} @@ -1283,10 +1319,10 @@ before using it in the Pod. {{< /note >}} -更多详情可以参考 [Portworx 卷](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/portworx/README.md)。 +更多详情可以参考 [Portworx 卷](https://github.com/kubernetes/examples/tree/master/staging/volumes/portworx/README.md)。 ### projected @@ -1306,10 +1342,10 @@ Currently, the following types of volume sources can be projected: 所有的卷来源需要和 Pod 处于相同的命名空间。 -更多详情请参考[一体化卷设计文档](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/node/all-in-one-volume.md)。 +更多详情请参考[一体化卷设计文档](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/all-in-one-volume.md)。 RBD 的一个特性是它可以同时被多个用户以只读方式挂载。 这意味着你可以用数据集预先填充卷，然后根据需要在尽可能多的 Pod 中并行地使用卷。 不幸的是，RBD 卷只能由单个使用者以读写模式安装。不允许同时写入。 更多详情请参考 -[RBD 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/volumes/rbd)。 - - -### scaleIO （已弃用） {#scaleio} - - -ScaleIO 是基于软件的存储平台，可以使用现有硬件来创建可伸缩的、共享的而且是网络化的块存储集群。 -`scaleIO` 卷插件允许部署的 Pod 访问现有的 ScaleIO 卷（或者它可以动态地为持久卷申领提供新的卷， -参见 [ScaleIO 持久卷](/zh/docs/concepts/storage/persistent-volumes/#scaleio)）。 - - -{{< note >}} -在使用前，你必须有个安装完毕且运行正常的 ScaleIO 集群，并且创建好了存储卷。 -{{< /note >}} - - -下面是配置了 ScaleIO 的 Pod 示例： - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: pod-0 -spec: - containers: - - image: k8s.gcr.io/test-webserver - name: pod-0 - volumeMounts: - - mountPath: /test-pd - name: vol-0 - volumes: - - name: vol-0 - scaleIO: - gateway: https://localhost:443/api - system: scaleio - protectionDomain: sd0 - storagePool: sp1 - volumeName: vol-0 - secretRef: - name: sio-secret - fsType: xfs -``` - - -更多详情，请参考 [ScaleIO 示例](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/scaleio)。 +[RBD 示例](https://github.com/kubernetes/examples/tree/master/volumes/rbd)。 ### secret @@ -1633,7 +1610,7 @@ For more details, see [Configuring Secrets](/docs/concepts/configuration/secret/ --> 更多详情请参考[配置 Secrets](/zh/docs/concepts/configuration/secret/)。 -### storageOS {#storageos} +### storageOS (已弃用) {#storageos} 为了避免控制器管理器和 kubelet 加载 `vsphereVolume` 插件，你需要将 -`CSIMigrationVSphereComplete` 特性设置为 `true`。你还必须在所有工作节点上安装 +`InTreePluginvSphereUnregister` 特性设置为 `true`。你还必须在所有工作节点上安装 `csi.vsphere.vmware.com` {{< glossary_tooltip text="CSI" term_id="csi" >}} 驱动。 -Pod 的安全性配置一般通过使用 -[安全性上下文（Security Context）](/zh/docs/tasks/configure-pod-container/security-context/) -来保证。安全性上下文允许用户逐个 Pod 地定义特权级及访问控制。 +Pod 安全性标准定义了三种不同的 _策略（Policy）_，以广泛覆盖安全应用场景。 +这些策略是 _渐进式的（Cumulative）_，安全级别从高度宽松至高度受限。 +本指南概述了每个策略的要求。 -以前，对集群的安全性上下文的需求的实施及其基于策略的定义都通过使用 -[Pod 安全性策略](/zh/docs/concepts/policy/pod-security-policy/)来实现。 -_Pod 安全性策略（Pod Security Policy）_ 是一种集群层面的资源，控制 Pod 规约中 -安全性敏感的部分。 - -不过，新的策略实施方式不断涌现，或增强或替换 PodSecurityPolicy 的使用。 -本页的目的是详细介绍建议实施的 Pod 安全框架；这些内容与具体的实现无关。 +| Profile | 描述 | +| ------ | ----------- | +| Privileged | 不受限制的策略，提供最大可能范围的权限许可。此策略允许已知的特权提升。 | +| Baseline | 限制性最弱的策略，禁止已知的策略提升。允许使用默认的（规定最少）Pod 配置。 | +| Restricted | 限制性非常强的策略，遵循当前的保护 Pod 的最佳实践。 | -## 策略类型 {#policy-types} - -在进一步讨论整个策略谱系之前，有必要对基本的策略下个定义。 -策略可以是很严格的也可以是很宽松的： - -- **_Privileged_** - 不受限制的策略，提供最大可能范围的权限许可。这些策略 - 允许已知的特权提升。 -- **_Baseline_** - 限制性最弱的策略，禁止已知的策略提升。 - 允许使用默认的（规定最少）Pod 配置。 -- **_Restricted_** - 限制性非常强的策略，遵循当前的保护 Pod 的最佳实践。 - - -## 策略 {#policies} +## Profile 细节 {#profile-details} ### Privileged -Privileged 策略是有目的地开放且完全无限制的策略。此类策略通常针对由 -特权较高、受信任的用户所管理的系统级或基础设施级负载。 +**_Privileged_ 策略是有目的地开放且完全无限制的策略。** +此类策略通常针对由特权较高、受信任的用户所管理的系统级或基础设施级负载。 Privileged 策略定义中限制较少。对于默认允许（Allow-by-default）实施机制（例如 gatekeeper）， Privileged 框架可能意味着不应用任何约束而不是实施某策略实例。 @@ -93,15 +65,26 @@ Privileged 策略应该默认允许所有控制（即，禁止所有限制）。 ### Baseline -Baseline 策略的目标是便于常见的容器化应用采用，同时禁止已知的特权提升。 +**_Baseline_ 策略的目标是便于常见的容器化应用采用，同时禁止已知的特权提升。** 此策略针对的是应用运维人员和非关键性应用的开发人员。 下面列举的控制应该被实施（禁止）： +{{< note >}} +在下述表格中，通配符（`*`）意味着一个列表中的所有元素。 +例如 `spec.containers[*].securityContext` 表示 _所定义的所有容器_ 的安全性上下文对象。 +如果所列出的任一容器不能满足要求，整个 Pod 将无法通过校验。 +{{< /note >}} + @@ -110,202 +93,429 @@ Baseline 策略的目标是便于常见的容器化应用采用，同时禁止 - - - - + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + + + + + + + + + + + + - + @@ -314,14 +524,26 @@ Baseline 策略的目标是便于常见的容器化应用采用，同时禁止 ### Restricted -Restricted 策略旨在实施当前保护 Pod 的最佳实践，尽管这样作可能会牺牲一些兼容性。 +**_Restricted_ 策略旨在实施当前保护 Pod 的最佳实践，尽管这样作可能会牺牲一些兼容性。** 该类策略主要针对运维人员和安全性很重要的应用的开发人员，以及不太被信任的用户。 下面列举的控制需要被实施（禁止）： +{{< note >}} +在下述表格中，通配符（`*`）意味着一个列表中的所有元素。 +例如 `spec.containers[*].securityContext` 表示 _所定义的所有容器_ 的安全性上下文对象。 +如果所列出的任一容器不能满足要求，整个 Pod 将无法通过校验。 +{{< /note >}} +
    Baseline 策略规范
    控制（Control） 策略（Policy）
    宿主名字空间
    HostProcess - 必须禁止共享宿主名字空间。
    -
    限制的字段：
    - spec.hostNetwork
    - spec.hostPID
    - spec.hostIPC
    -
    允许的值： false
    +

    Windows Pod 提供了运行 + HostProcess 容器 的能力， + 这使得对 Windows 节点的特权访问成为可能。 + 基线策略中对宿主的特权访问是被禁止的。 + HostProcess Pod 是 Kubernetes v1.22 版本的 + alpha 特性。

    +

    限制的字段

    +
      +
    • spec.securityContext.windowsOptions.hostProcess
      • +
    • spec.containers[*].securityContext.windowsOptions.hostProcess
      • +
    • spec.initContainers[*].securityContext.windowsOptions.hostProcess
      • +
    • spec.ephemeralContainers[*].securityContext.windowsOptions.hostProcess
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • false
      • +
    特权容器宿主名字空间 - 特权 Pod 禁用大多数安全性机制，必须被禁止。
    -
    限制的字段：
    - spec.containers[*].securityContext.privileged
    - spec.initContainers[*].securityContext.privileged
    -
    允许的值： false、未定义/nil
    +

    必须禁止共享宿主名字空间。

    +

    限制的字段

    +
      +
    • spec.hostNetwork
      • +
    • spec.hostPID
      • +
    • spec.hostIPC
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • false
      • +
    权能特权容器 - 必须禁止添加默认集合之外的权能。
    -
    限制的字段：
    - spec.containers[*].securityContext.capabilities.add
    - spec.initContainers[*].securityContext.capabilities.add
    -
    允许的值： 空（或限定为一个已知列表）
    +

    特权 Pod 关闭了大多数安全性机制，必须被禁止。

    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.privileged
      • +
    • spec.initContainers[*].securityContext.privileged
      • +
    • spec.ephemeralContainers[*].securityContext.privileged
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • false
      • +
    HostPath 卷权能 - 必须禁止 HostPath 卷。
    -
    限制的字段：
    - spec.volumes[*].hostPath
    -
    允许的值： 未定义/nil
    +

    必须禁止添加除下列字段之外的权能。

    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.capabilities.add
      • +
    • spec.initContainers[*].securityContext.capabilities.add
      • +
    • spec.ephemeralContainers[*].securityContext.capabilities.add
      • +
    +

    允许的值

    +
      +
    • Undefined/nil
      • +
    • AUDIT_WRITE
      • +
    • CHOWN
      • +
    • DAC_OVERRIDE
      • +
    • FOWNER
      • +
    • FSETID
      • +
    • KILL
      • +
    • MKNOD
      • +
    • NET_BIND_SERVICE
      • +
    • SETFCAP
      • +
    • SETGID
      • +
    • SETPCAP
      • +
    • SETUID
      • +
    • SYS_CHROOT
      • +
    宿主端口HostPath 卷 - 应禁止使用宿主端口，或者至少限定为已知列表。
    -
    限制的字段：
    - spec.containers[*].ports[*].hostPort
    - spec.initContainers[*].ports[*].hostPort
    -
    允许的值： 0、未定义（或限定为已知列表）
    +

    必须禁止 HostPath 卷。

    +

    限制的字段

    +
      +
    • spec.volumes[*].hostPath
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    AppArmor宿主端口 - 在被支持的主机上，默认使用 'runtime/default' AppArmor Profile。 - 基线策略应避免覆盖或者禁用默认策略，以及限制覆盖一些 profile 集合的权限。
    -
    限制的字段：
    - metadata.annotations['container.apparmor.security.beta.kubernetes.io/*']
    -
    允许的值： 'runtime/default'、未定义
    +

    应禁止使用宿主端口，或者至少限定为已知列表。

    +

    限制的字段

    +
      +
    • spec.containers[*].ports[*].hostPort
      • +
    • spec.initContainers[*].ports[*].hostPort
      • +
    • spec.ephemeralContainers[*].ports[*].hostPort
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • 已知列表
      • +
    • 0
      • +
    SELinuxAppArmor - 设置 SELinux 类型的操作是被限制的，设置自定义的 SELinux 用户或角色选项是被禁止的。
    -
    限制的字段：
    - spec.securityContext.seLinuxOptions.type
    - spec.containers[*].securityContext.seLinuxOptions.type
    - spec.initContainers[*].securityContext.seLinuxOptions.type
    -
    允许的值：
    - 未定义/空
    - container_t
    - container_init_t
    - container_kvm_t
    -
    被限制的字段：
    - spec.securityContext.seLinuxOptions.user
    - spec.containers[*].securityContext.seLinuxOptions.user
    - spec.initContainers[*].securityContext.seLinuxOptions.user
    - spec.securityContext.seLinuxOptions.role
    - spec.containers[*].securityContext.seLinuxOptions.role
    - spec.initContainers[*].securityContext.seLinuxOptions.role
    -
    允许的值： 未定义或空
    +

    在受支持的主机上，默认使用 runtime/default AppArmor Profile。 + 基线策略应避免覆盖或者禁用默认策略，以及限制覆盖一些 Profile 集合的权限。

    +

    限制的字段

    +
      +
    • metadata.annotations["container.apparmor.security.beta.kubernetes.io/*"]
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • runtime/default
      • +
    • localhost/*
      • +
    /proc 挂载类型SELinux - 要求使用默认的 /proc 掩码以减小攻击面。
    -
    限制的字段：
    - spec.containers[*].securityContext.procMount
    - spec.initContainers[*].securityContext.procMount
    -
    允许的值： 未定义/nil、'Default'
    +

    设置 SELinux 类型的操作是被限制的，设置自定义的 SELinux 用户或角色选项是被禁止的。

    +

    限制的字段

    +
      +
    • spec.securityContext.seLinuxOptions.type
      • +
    • spec.containers[*].securityContext.seLinuxOptions.type
      • +
    • spec.initContainers[*].securityContext.seLinuxOptions.type
      • +
    • spec.ephemeralContainers[*].securityContext.seLinuxOptions.type
      • +
    +

    允许的值

    +
      +
    • 未定义/""
      • +
    • container_t
      • +
    • container_init_t
      • +
    • container_kvm_t
      • +
    +
    +

    限制的字段

    +
      +
    • spec.securityContext.seLinuxOptions.user
      • +
    • spec.containers[*].securityContext.seLinuxOptions.user
      • +
    • spec.initContainers[*].securityContext.seLinuxOptions.user
      • +
    • spec.ephemeralContainers[*].securityContext.seLinuxOptions.user
      • +
    • spec.securityContext.seLinuxOptions.role
      • +
    • spec.containers[*].securityContext.seLinuxOptions.role
      • +
    • spec.initContainers[*].securityContext.seLinuxOptions.role
      • +
    • spec.ephemeralContainers[*].securityContext.seLinuxOptions.role
      • +
    +

    允许的值

    +
      +
    • 未定义/""
      • +
    /proc 挂载类型 +

    要求使用默认的 /proc 掩码以减小攻击面。

    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.procMount
      • +
    • spec.initContainers[*].securityContext.procMount
      • +
    • spec.ephemeralContainers[*].securityContext.procMount
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • Default
      • +
    +
    Seccomp +

    Seccomp Profile 禁止被显式设置为 Unconfined。

    +

    限制的字段

    +
      +
    • spec.securityContext.seccompProfile.type
      • +
    • spec.containers[*].securityContext.seccompProfile.type
      • +
    • spec.initContainers[*].securityContext.seccompProfile.type
      • +
    • spec.ephemeralContainers[*].securityContext.seccompProfile.type
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • RuntimeDefault
      • +
    • Localhost
      • +
    +
    Sysctls - Sysctls 可以禁用安全机制或影响宿主上所有容器，因此除了若干『安全』的子集之外，应该被禁止。 - 如果某 sysctl 是受容器或 Pod 的名字空间限制，且与节点上其他 Pod 或进程相隔离，可认为是安全的。
    -
    限制的字段：
    - spec.securityContext.sysctls
    -
    允许的值：
    - kernel.shm_rmid_forced
    - net.ipv4.ip_local_port_range
    - net.ipv4.tcp_syncookies
    - net.ipv4.ping_group_range
    - 未定义/空值
    +

    Sysctls 可以禁用安全机制或影响宿主上所有容器，因此除了若干“安全”的子集之外，应该被禁止。 + 如果某 sysctl 是受容器或 Pod 的名字空间限制，且与节点上其他 Pod 或进程相隔离，可认为是安全的。

    +

    限制的字段

    +
      +
    • spec.securityContext.sysctls[*].name
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • kernel.shm_rmid_forced
      • +
    • net.ipv4.ip_local_port_range
      • +
    • net.ipv4.ip_unprivileged_port_start
      • +
    • net.ipv4.tcp_syncookies
      • +
    • net.ipv4.ping_group_range
      • +
    @@ -337,90 +559,282 @@ Restricted 策略旨在实施当前保护 Pod 的最佳实践，尽管这样作 - + - - + - 禁止（通过 SetUID 或 SetGID 文件模式）获得特权提升。
    -
    限制的字段：
    - spec.containers[*].securityContext.allowPrivilegeEscalation
    - spec.initContainers[*].securityContext.allowPrivilegeEscalation
    -
    允许的值： false
    +

    Privilege escalation (such as via set-user-ID or set-group-ID file mode) should not be allowed.

    +

    Restricted Fields

    +
      +
    • spec.containers[*].securityContext.allowPrivilegeEscalation
      • +
    • spec.initContainers[*].securityContext.allowPrivilegeEscalation
      • +
    • spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
      • +
    +

    Allowed Values

    +
      +
    • false
      • +
    + --> + + - - + + + - - + + - + + + + + + + @@ -443,11 +857,17 @@ of individual policies are not defined here. 随着相关机制的成熟，这些机制会按策略分别定义在下面。特定策略的实施方法不在这里定义。 +[**Pod 安全性准入控制器**](/zh/docs/concepts/security/pod-security-admission/) + +- {{< example file="security/podsecurity-privileged.yaml" >}}Privileged namespace{{< /example >}} +- {{< example file="security/podsecurity-baseline.yaml" >}}Baseline namespace{{< /example >}} +- {{< example file="security/podsecurity-restricted.yaml" >}}Restricted namespace{{< /example >}} + [**PodSecurityPolicy**](/zh/docs/concepts/policy/pod-security-policy/) -- [Privileged](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/privileged-psp.yaml) -- [Baseline](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/baseline-psp.yaml) -- [Restricted](https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) +- {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}} +- {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}} +- {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}} 安全策略则是控制面用来对安全上下文以及安全性上下文之外的参数实施某种设置的机制。 -在 2020 年 2 月，目前实施这些安全性策略的原生解决方案是 -[Pod 安全性策略](/zh/docs/concepts/policy/pod-security-policy/) - 一种对集群中 -Pod 的安全性策略进行集中控制的机制。 +在 2020 年 7 月， +[Pod 安全性策略](/zh/docs/concepts/policy/pod-security-policy/)已被废弃， +取而代之的是内置的 [Pod 安全性准入控制器](/zh/docs/concepts/security/pod-security-admission/)。 + Kubernetes 生态系统中还在开发一些其他的替代方案，例如 -[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper)。 +- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper)。 +- [Kubewarden](https://github.com/kubewarden)。 +- [Kyverno](https://kyverno.io/policies/pod-security/)。 + +如果你为一个 Windows Pod 应用了 Restricted 策略，**可能会** 对该 Pod 的运行时产生影响。 +Restricted 策略需要强制执行 Linux 特有的限制（如 seccomp Profile，并且禁止特权提升）。 +如果 kubelet 和/或其容器运行时忽略了 Linux 特有的值，那么应该不影响 Windows Pod 正常工作。 +然而，对于使用 Windows 容器的 Pod 来说，缺乏强制执行意味着相比于 Restricted 策略，没有任何额外的限制。 + +你应该只在 Privileged 策略下使用 HostProcess 标志来创建 HostProcess Pod。 +在 Baseline 和 Restricted 策略下，创建 Windows HostProcess Pod 是被禁止的， +因此任何 HostProcess Pod 都应该被认为是有特权的。 + + +Varios objetos clave que representan diferentes tipos o partes de un Workload +incluyen: "DaemonSet", "Deployment", "Job", "ReplicaSet" y "StatefulSet". + +Por ejemplo, un Workload que tiene un servidor web y una base de datos podría ejecutar +la base de datos en un {{< glossary_tooltip term_id="StatefulSet" >}} y el servidor +web en un {{< glossary_tooltip term_id="Deployment" >}}. From b31d7cd87843f306a1030251d668284fad8ee6c9 Mon Sep 17 00:00:00 2001 From: elderdeveloper15 Date: Sat, 9 Oct 2021 19:39:43 -0500 Subject: [PATCH 164/249] Feedback sugerido por Victor Morales aplicado. --- content/es/docs/reference/glossary/workload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/es/docs/reference/glossary/workload.md b/content/es/docs/reference/glossary/workload.md index 4769820401..19e0ef99b8 100644 --- a/content/es/docs/reference/glossary/workload.md +++ b/content/es/docs/reference/glossary/workload.md @@ -15,7 +15,7 @@ tags: Varios objetos clave que representan diferentes tipos o partes de un Workload -incluyen: "DaemonSet", "Deployment", "Job", "ReplicaSet" y "StatefulSet". +incluyen los objetos: DaemonSet, Deployment, Job, ReplicaSet y StatefulSet. Por ejemplo, un Workload que tiene un servidor web y una base de datos podría ejecutar la base de datos en un {{< glossary_tooltip term_id="StatefulSet" >}} y el servidor From d3c842419d326b7709f7fcb77fd454058d3070ba Mon Sep 17 00:00:00 2001 From: Mayo Date: Sun, 10 Oct 2021 11:45:34 +0800 Subject: [PATCH 165/249] [zh] update concept pv --- .../concepts/storage/persistent-volumes.md | 304 ++++++++++++++---- 1 file changed, 239 insertions(+), 65 deletions(-) diff --git a/content/zh/docs/concepts/storage/persistent-volumes.md b/content/zh/docs/concepts/storage/persistent-volumes.md index 607a899f2b..c186202548 100644 --- a/content/zh/docs/concepts/storage/persistent-volumes.md +++ b/content/zh/docs/concepts/storage/persistent-volumes.md @@ -295,13 +295,16 @@ The `Retain` reclaim policy allows for manual reclamation of the resource. When 1. 删除 PersistentVolume 对象。与之相关的、位于外部基础设施中的存储资产 （例如 AWS EBS、GCE PD、Azure Disk 或 Cinder 卷）在 PV 删除之后仍然存在。 1. 根据情况，手动清除所关联的存储资产上的数据。 -1. 手动删除所关联的存储资产；如果你希望重用该存储资产，可以基于存储资产的 - 定义创建新的 PersistentVolume 卷对象。 +1. 手动删除所关联的存储资产。 + +如果你希望重用该存储资产，可以基于存储资产的定义创建新的 PersistentVolume 卷对象。 * [`awsElasticBlockStore`](/zh/docs/concepts/storage/volumes/#awselasticblockstore) - AWS 弹性块存储（EBS） * [`azureDisk`](/zh/docs/concepts/storage/volumes/#azuredisk) - Azure Disk * [`azureFile`](/zh/docs/concepts/storage/volumes/#azurefile) - Azure File * [`cephfs`](/zh/docs/concepts/storage/volumes/#cephfs) - CephFS volume -* [`cinder`](/zh/docs/concepts/storage/volumes/#cinder) - Cinder （OpenStack 块存储） - (**弃用**) * [`csi`](/zh/docs/concepts/storage/volumes/#csi) - 容器存储接口 (CSI) * [`fc`](/zh/docs/concepts/storage/volumes/#fc) - Fibre Channel (FC) 存储 * [`flexVolume`](/zh/docs/concepts/storage/volumes/#flexVolume) - FlexVolume -* [`flocker`](/zh/docs/concepts/storage/volumes/#flocker) - Flocker 存储 * [`gcePersistentDisk`](/zh/docs/concepts/storage/volumes/#gcepersistentdisk) - GCE 持久化盘 * [`glusterfs`](/zh/docs/concepts/storage/volumes/#glusterfs) - Glusterfs 卷 * [`hostPath`](/zh/docs/concepts/storage/volumes/#hostpath) - HostPath 卷 @@ -661,16 +651,45 @@ PV 持久卷是用插件的形式来实现的。Kubernetes 目前支持以下插 * [`iscsi`](/zh/docs/concepts/storage/volumes/#iscsi) - iSCSI (SCSI over IP) 存储 * [`local`](/zh/docs/concepts/storage/volumes/#local) - 节点上挂载的本地存储设备 * [`nfs`](/zh/docs/concepts/storage/volumes/#nfs) - 网络文件系统 (NFS) 存储 -* `photonPersistentDisk` - Photon 控制器持久化盘。 - （这个卷类型已经因对应的云提供商被移除而被弃用）。 * [`portworxVolume`](/zh/docs/concepts/storage/volumes/#portworxvolume) - Portworx 卷 -* [`quobyte`](/zh/docs/concepts/storage/volumes/#quobyte) - Quobyte 卷 * [`rbd`](/zh/docs/concepts/storage/volumes/#rbd) - Rados 块设备 (RBD) 卷 -* [`scaleIO`](/zh/docs/concepts/storage/volumes/#scaleio) - ScaleIO 卷 - (**弃用**) -* [`storageos`](/zh/docs/concepts/storage/volumes/#storageos) - StorageOS 卷 * [`vsphereVolume`](/zh/docs/concepts/storage/volumes/#vspherevolume) - vSphere VMDK 卷 + + +以下的持久卷已被弃用。这意味着当前仍是支持的，但是 Kubernetes 将来的发行版会将其移除。 + +* [`cinder`](/docs/concepts/storage/volumes/#cinder) - Cinder（OpenStack 块存储）（于 v1.18 **弃用**） +* [`flocker`](/docs/concepts/storage/volumes/#flocker) - Flocker 存储（于 v1.22 **弃用**） +* [`quobyte`](/docs/concepts/storage/volumes/#quobyte) - Quobyte 卷 +（于 v1.22 **弃用**） +* [`storageos`](/docs/concepts/storage/volumes/#storageos) - StorageOS 卷（于 v1.22 **弃用**） + + + +旧版本的 Kubernetes 仍支持这些“树内（In-Tree）”持久卷类型： + +* `photonPersistentDisk` - Photon 控制器持久化盘。（v1.15 之后 **不可用**） +* [`scaleIO`](/docs/concepts/storage/volumes/#scaleio) - ScaleIO 卷（v1.21 之后 **不可用**） + 访问模式有： -* ReadWriteOnce -- 卷可以被一个节点以读写方式挂载； -* ReadOnlyMany -- 卷可以被多个节点以只读方式挂载； -* ReadWriteMany -- 卷可以被多个节点以读写方式挂载。 +`ReadWriteOnce` +: 卷可以被一个节点以读写方式挂载。 +ReadWriteOnce 访问模式也允许运行在同一节点上的多个 Pod 访问卷。 + +`ReadOnlyMany` +: 卷可以被多个节点以只读方式挂载。 + +`ReadWriteMany` +: 卷可以被多个节点以读写方式挂载。 + +`ReadWriteOncePod` +: 卷可以被单个 Pod 以读写方式挂载。 +如果你想确保整个集群中只有一个 Pod 可以读取或写入该 PVC， +请使用ReadWriteOncePod 访问模式。这只支持 CSI 卷以及需要 Kubernetes 1.22 以上版本。 + +这篇博客文章 [Introducing Single Pod Access Mode for PersistentVolumes](/blog/2021/09/13/read-write-once-pod-access-mode-alpha/) +描述了更详细的内容。 在命令行接口（CLI）中，访问模式也使用以下缩写形式： * RWO - ReadWriteOnce * ROX - ReadOnlyMany * RWX - ReadWriteMany +* RWOP - ReadWriteOncePod -> __重要提醒！__ 每个卷只能同一时刻只能以一种访问模式挂载，即使该卷能够支持 +> __重要提醒！__ 每个卷同一时刻只能以一种访问模式挂载，即使该卷能够支持 > 多种访问模式。例如，一个 GCEPersistentDisk 卷可以被某节点以 ReadWriteOnce > 模式挂载，或者被多个节点以 ReadOnlyMany 模式挂载，但不可以同时以两种模式 > 挂载。 @@ -826,28 +871,27 @@ In the CLI, the access modes are abbreviated to: | Volume Plugin | ReadWriteOnce | ReadOnlyMany | ReadWriteMany| --> -| 卷插件 | ReadWriteOnce | ReadOnlyMany | ReadWriteMany| -| :--- | :---: | :---: | :---: | -| AWSElasticBlockStore | ✓ | - | - | -| AzureFile | ✓ | ✓ | ✓ | -| AzureDisk | ✓ | - | - | -| CephFS | ✓ | ✓ | ✓ | -| Cinder | ✓ | - | - | -| CSI | 取决于驱动 | 取决于驱动 | 取决于驱动 | -| FC | ✓ | ✓ | - | -| FlexVolume | ✓ | ✓ | 取决于驱动 | -| Flocker | ✓ | - | - | -| GCEPersistentDisk | ✓ | ✓ | - | -| Glusterfs | ✓ | ✓ | ✓ | -| HostPath | ✓ | - | - | -| iSCSI | ✓ | ✓ | - | -| Quobyte | ✓ | ✓ | ✓ | -| NFS | ✓ | ✓ | ✓ | -| RBD | ✓ | ✓ | - | -| VsphereVolume | ✓ | - | - (Pod 运行于同一节点上时可行) | -| PortworxVolume | ✓ | - | ✓ | -| ScaleIO | ✓ | ✓ | - | -| StorageOS | ✓ | - | - | +| 卷插件 | ReadWriteOnce | ReadOnlyMany | ReadWriteMany | ReadWriteOncePod | +| :--- | :---: | :---: | :---: | - | +| AWSElasticBlockStore | ✓ | - | - | - | +| AzureFile | ✓ | ✓ | ✓ | - | +| AzureDisk | ✓ | - | - | - | +| CephFS | ✓ | ✓ | ✓ | - | +| Cinder | ✓ | - | - | - | +| CSI | 取决于驱动 | 取决于驱动 | 取决于驱动 | 取决于驱动 | +| FC | ✓ | ✓ | - | - | +| FlexVolume | ✓ | ✓ | 取决于驱动 | - | +| Flocker | ✓ | - | - | - | +| GCEPersistentDisk | ✓ | ✓ | - | - | +| Glusterfs | ✓ | ✓ | ✓ | - | +| HostPath | ✓ | - | - | - | +| iSCSI | ✓ | ✓ | - | - | +| Quobyte | ✓ | ✓ | ✓ | - | +| NFS | ✓ | ✓ | ✓ | - | +| RBD | ✓ | ✓ | - | - | +| VsphereVolume | ✓ | - | - （Pod 运行于同一节点上时可行） | - | +| PortworxVolume | ✓ | - | ✓ | - | - | +| StorageOS | ✓ | - | - | - | -每个 PV 卷可以通过设置 -[节点亲和性](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volumenodeaffinity-v1-core) -来定义一些约束，进而限制从哪些节点上可以访问此卷。 +每个 PV 卷可以通过设置节点亲和性来定义一些约束，进而限制从哪些节点上可以访问此卷。 使用这些卷的 Pod 只会被调度到节点亲和性规则所选择的节点上执行。 +要设置节点亲和性，配置 PV 卷 `.spec` 中的 `nodeAffinity`。 +[持久卷](/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-v1/#PersistentVolumeSpec) +API 参考关于该字段的更多细节。 + +## 卷填充器（Populator）与数据源 {#volume-populators-and-data-sources} + +{{< feature-state for_k8s_version="v1.22" state="alpha" >}} + +{{< note >}} +Kubernetes 支持自定义的卷填充器；Kubernetes 1.18 版本引入了这个 alpha 特性。 +Kubernetes 1.22 使用重新设计的 API 重新实现了该机制。 +确认你正在阅读与你的集群版本一致的 Kubernetes 文档。{{% version-check %}} +要使用自定义的卷填充器，你必须为 kube-apiserver 和 kube-controller-manager 启用 `AnyVolumeDataSource` +[特性门控](/zh/docs/reference/command-line-tools-reference/feature-gates/)。 +{{< /note >}} + +卷填充器利用了 PVC 规约字段 `dataSourceRef`。 +不像 `dataSource` 字段只能包含对另一个持久卷申领或卷快照的引用， +`dataSourceRef` 字段可以包含对同一命名空间中任何对象的引用（不包含除 PVC 以外的核心资源）。 +对于启用了特性门控的集群，使用 `dataSourceRef` 比 `dataSource` 更好。 + + + +## 数据源引用 {#data-source-references} + +`dataSourceRef` 字段的行为与 `dataSource` 字段几乎相同。 +如果其中一个字段被指定而另一个字段没有被指定，API 服务器将给两个字段相同的值。 +这两个字段都不能在创建后改变，如果试图为这两个字段指定不同的值，将导致验证错误。 +因此，这两个字段将总是有相同的内容。 + + +在 `dataSourceRef` 字段和 `dataSource` 字段之间有两个用户应该注意的区别： +* `dataSource` 字段会忽略无效的值（如同是空值）， + 而 `dataSourceRef` 字段永远不会忽略值，并且若填入一个无效的值，会导致错误。 + 无效值指的是 PVC 之外的核心对象（没有 apiGroup 的对象）。 +* `dataSourceRef` 字段可以包含不同类型的对象，而 `dataSource` 字段只允许 PVC 和卷快照。 + +用户应该始终在启用了特性门控的集群上使用 `dataSourceRef`，而在没有启用特性门控的集群上使用 `dataSource`。 +在任何情况下都没有必要查看这两个字段。 +这两个字段的值看似相同但是语义稍微不一样，是为了向后兼容。 +特别是混用旧版本和新版本的控制器时，它们能够互通。 + + +## 使用卷填充器 {#using-volume-populators} + +卷填充器是能创建非空卷的{{< glossary_tooltip text="控制器" term_id="controller" >}}， +其卷的内容通过一个自定义资源决定。 +用户通过使用 `dataSourceRef` 字段引用自定义资源来创建一个被填充的卷： + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: populated-pvc +spec: + dataSourceRef: + name: example-name + kind: ExampleDataSource + apiGroup: example.storage.k8s.io + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + + +因为卷填充器是外部组件，如果没有安装所有正确的组件，试图创建一个使用卷填充器的 PVC 就会失败。 +外部控制器应该在 PVC 上产生事件，以提供创建状态的反馈，包括在由于缺少某些组件而无法创建 PVC 的情况下发出警告。 + +你可以把 alpha 版本的[卷数据源验证器](https://github.com/kubernetes-csi/volume-data-source-validator) +控制器安装到你的集群中。 +如果没有填充器处理该数据源的情况下，该控制器会在 PVC 上产生警告事件。 +当一个合适的填充器被安装到 PVC 上时，该控制器的职责是上报与卷创建有关的事件，以及在该过程中发生的问题。 + @@ -1506,16 +1680,16 @@ and need persistent storage, it is recommended that you use the following patter * 阅读[持久存储的设计文档](https://git.k8s.io/community/contributors/design-proposals/storage/persistent-storage.md). -### 参考 {#reference} +### API 参考 {#reference} -* [PersistentVolume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolume-v1-core) -* [PersistentVolumeSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumespec-v1-core) -* [PersistentVolumeClaim](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaim-v1-core) -* [PersistentVolumeClaimSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaimspec-v1-core) +阅读以下页面中描述的 API： + +* [`PersistentVolume`](/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-v1/) +* [`PersistentVolumeClaim`](/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-claim-v1/) From 5cce85b5dc074adf894c75819658e5b491a5f6bc Mon Sep 17 00:00:00 2001 From: Mayo Date: Sun, 10 Oct 2021 17:43:42 +0800 Subject: [PATCH 166/249] [zh] update concept workload controller --- .../workloads/controllers/cron-jobs.md | 59 ++++-- .../workloads/controllers/daemonset.md | 117 +++++++---- .../concepts/workloads/controllers/job.md | 183 ++++++++++++------ .../workloads/controllers/replicaset.md | 25 ++- .../workloads/controllers/statefulset.md | 129 +++++++----- 5 files changed, 357 insertions(+), 156 deletions(-) diff --git a/content/zh/docs/concepts/workloads/controllers/cron-jobs.md b/content/zh/docs/concepts/workloads/controllers/cron-jobs.md index 44be9b9a68..265a1fa3cb 100644 --- a/content/zh/docs/concepts/workloads/controllers/cron-jobs.md +++ b/content/zh/docs/concepts/workloads/controllers/cron-jobs.md @@ -19,6 +19,8 @@ A _CronJob_ creates {{< glossary_tooltip term_id="job" text="Jobs" >}} on a repe One CronJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format. + +In addition, the CronJob schedule supports timezone handling, you can specify the timezone by adding "CRON_TZ=
    Restricted 策略规范
    基线策略的所有要求。
    卷类型 - - 除了限制 HostPath 卷之外，此类策略还限制可以通过 PersistentVolumes 定义的非核心卷类型。
    -
    限制的字段：
    - spec.volumes[*].hostPath
    - spec.volumes[*].gcePersistentDisk
    - spec.volumes[*].awsElasticBlockStore
    - spec.volumes[*].gitRepo
    - spec.volumes[*].nfs
    - spec.volumes[*].iscsi
    - spec.volumes[*].glusterfs
    - spec.volumes[*].rbd
    - spec.volumes[*].flexVolume
    - spec.volumes[*].cinder
    - spec.volumes[*].cephFS
    - spec.volumes[*].flocker
    - spec.volumes[*].fc
    - spec.volumes[*].azureFile
    - spec.volumes[*].vsphereVolume
    - spec.volumes[*].quobyte
    - spec.volumes[*].azureDisk
    - spec.volumes[*].portworxVolume
    - spec.volumes[*].scaleIO
    - spec.volumes[*].storageos
    - spec.volumes[*].csi
    -
    允许的值： 未定义/nil
    +

    除了限制 HostPath 卷之外，此类策略还限制可以通过 PersistentVolumes 定义的非核心卷类型。

    +

    限制的字段

    +
      +
    • spec.volumes[*].hostPath
      • +
    • spec.volumes[*].gcePersistentDisk
      • +
    • spec.volumes[*].awsElasticBlockStore
      • +
    • spec.volumes[*].gitRepo
      • +
    • spec.volumes[*].nfs
      • +
    • spec.volumes[*].iscsi
      • +
    • spec.volumes[*].glusterfs
      • +
    • spec.volumes[*].rbd
      • +
    • spec.volumes[*].flexVolume
      • +
    • spec.volumes[*].cinder
      • +
    • spec.volumes[*].cephfs
      • +
    • spec.volumes[*].flocker
      • +
    • spec.volumes[*].fc
      • +
    • spec.volumes[*].azureFile
      • +
    • spec.volumes[*].vsphereVolume
      • +
    • spec.volumes[*].quobyte
      • +
    • spec.volumes[*].azureDisk
      • +
    • spec.volumes[*].portworxVolume
      • +
    • spec.volumes[*].scaleIO
      • +
    • spec.volumes[*].storageos
      • +
    • spec.volumes[*].photonPersistentDisk
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    特权提升特权提升（v1.8+） +

    禁止（通过 SetUID 或 SetGID 文件模式）获得特权提升。

    +
    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.allowPrivilegeEscalation
      • +
    • spec.initContainers[*].securityContext.allowPrivilegeEscalation
      • +
    • spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
      • +
    +

    允许的值

    +
      +
    • false
      • +
    以非 root 账号运行 以非 root 账号运行 - - 必须要求容器以非 root 用户运行。
    -
    限制的字段：
    - spec.securityContext.runAsNonRoot
    - spec.containers[*].securityContext.runAsNonRoot
    - spec.initContainers[*].securityContext.runAsNonRoot
    -
    允许的值： true
    +

    必须要求容器以非 root 用户运行。

    +

    限制的字段

    +
      +
    • spec.securityContext.runAsNonRoot
      • +
    • spec.containers[*].securityContext.runAsNonRoot
      • +
    • spec.initContainers[*].securityContext.runAsNonRoot
      • +
    • spec.ephemeralContainers[*].securityContext.runAsNonRoot
      • +
    +

    允许的值

    +
      +
    • true
      • +
    + + 如果 Pod 级别 spec.securityContext.runAsNonRoot 设置为 + true，则允许容器组的安全上下文字段设置为 未定义/nil。 +
    非 root 组 （可选）非 root 组（可选） - - 禁止容器使用 root 作为主要或辅助 GID 来运行。
    -
    限制的字段：
    - spec.securityContext.runAsGroup
    - spec.securityContext.supplementalGroups[*]
    - spec.securityContext.fsGroup
    - spec.containers[*].securityContext.runAsGroup
    - spec.initContainers[*].securityContext.runAsGroup
    -
    允许的值：
    - 非零值
    - 未定义/nil （*.runAsGroup 除外）
    + +

    禁止容器使用 root 作为主要或辅助 GID 来运行。

    +

    限制的字段

    +
      +
    • spec.securityContext.runAsGroup
      • +
    • spec.securityContext.supplementalGroups[*]
      • +
    • spec.securityContext.fsGroup
      • +
    • spec.containers[*].securityContext.runAsGroup
      • +
    • spec.initContainers[*].securityContext.runAsGroup
      • +
    • spec.ephemeralContainers[*].securityContext.runAsGroup
      • +
    +

    允许的值

    +
      +
    • 未定义/nil（*.runAsGroup 除外）
      • +
    • 非零值
      • +
    SeccompSeccomp (v1.19+) - - 必须要求使用 RuntimeDefault seccomp profile 或者允许使用特定的 profiles。
    -
    限制的字段：
    - spec.securityContext.seccompProfile.type
    - spec.containers[*].securityContext.seccompProfile
    - spec.initContainers[*].securityContext.seccompProfile
    -
    允许的值：
    - 'runtime/default'
    - 未定义/nil
    + +

    Seccomp Profile 必须被显式设置成一个允许的值。禁止使用 Unconfined + Profile 或者指定 不存在的 Profile。

    +

    限制的字段

    +
      +
    • spec.securityContext.seccompProfile.type
      • +
    • spec.containers[*].securityContext.seccompProfile.type
      • +
    • spec.initContainers[*].securityContext.seccompProfile.type
      • +
    • spec.ephemeralContainers[*].securityContext.seccompProfile.type
      • +
    +

    允许的值

    +
      +
    • RuntimeDefault
      • +
    • Localhost
      • +
    + + 如果 Pod 级别的 spec.securityContext.seccompProfile.type + 已设置得当，容器级别的安全上下文字段可以为 未定义/nil。 + 反过来说，如果 _所有的_ 容器级别的安全上下文字段已设置，则 Pod 级别的字段可为 未定义/nil。 + +
    权能（v1.22+） +

    + 容器组必须弃用 ALL 权能，并且只允许添加 NET_BIND_SERVICE 权能。 +

    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.capabilities.drop
      • +
    • spec.initContainers[*].securityContext.capabilities.drop
      • +
    • spec.ephemeralContainers[*].securityContext.capabilities.drop
      • +
    +

    允许的值

    +
      +
    • 包含 ALL 的任何一种权能列表。
      • +
    +
    +

    限制的字段

    +
      +
    • spec.containers[*].securityContext.capabilities.add
      • +
    • spec.initContainers[*].securityContext.capabilities.add
      • +
    • spec.ephemeralContainers[*].securityContext.capabilities.add
      • +
    +

    允许的值

    +
      +
    • 未定义/nil
      • +
    • NET_BIND_SERVICE
      • +
    From c689362c0875b13cc290b552f307df51c210f175 Mon Sep 17 00:00:00 2001 From: RA489 Date: Thu, 24 Jun 2021 15:07:57 +0530 Subject: [PATCH 209/249] Example with data dir for restore --- .../docs/tasks/administer-cluster/configure-upgrade-etcd.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md index ffa688e1b7..3dececd8e0 100644 --- a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md +++ b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md @@ -325,6 +325,10 @@ Here is an example: ```shell ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb ``` +Another example for restoring using etcdutl options: +```shell +ETCDCTL_API=3 etcdutl --data-dir snapshot restore snapshotdb +``` For more information and examples on restoring a cluster from a snapshot file, see [etcd disaster recovery documentation](https://etcd.io/docs/current/op-guide/recovery/#restoring-a-cluster). From 6c38d8755558941f1778fdd4d969aded806cc0bd Mon Sep 17 00:00:00 2001 From: Arhell Date: Tue, 19 Oct 2021 00:33:07 +0300 Subject: [PATCH 210/249] [zh] Update translate-compose-kubernetes.md --- .../configure-pod-container/translate-compose-kubernetes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/zh/docs/tasks/configure-pod-container/translate-compose-kubernetes.md b/content/zh/docs/tasks/configure-pod-container/translate-compose-kubernetes.md index 8c5bb67da5..e0cc2fec77 100644 --- a/content/zh/docs/tasks/configure-pod-container/translate-compose-kubernetes.md +++ b/content/zh/docs/tasks/configure-pod-container/translate-compose-kubernetes.md @@ -52,13 +52,13 @@ Kompose 通过 GitHub 发布，发布周期为三星期。 ```shell # Linux -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-amd64 -o kompose +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-linux-amd64 -o kompose # macOS -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-darwin-amd64 -o kompose +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-darwin-amd64 -o kompose # Windows -curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-windows-amd64.exe -o kompose.exe +curl -L https://github.com/kubernetes/kompose/releases/download/v1.24.0/kompose-windows-amd64.exe -o kompose.exe chmod +x kompose sudo mv ./kompose /usr/local/bin/kompose From a6f27be7e162243a1525ca94df0ae69d5e47b57d Mon Sep 17 00:00:00 2001 From: Jizu Sun Date: Tue, 19 Oct 2021 10:48:54 +0800 Subject: [PATCH 211/249] Update content/en/docs/concepts/overview/working-with-objects/finalizers.md Co-authored-by: Qiming Teng --- .../docs/concepts/overview/working-with-objects/finalizers.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/overview/working-with-objects/finalizers.md b/content/en/docs/concepts/overview/working-with-objects/finalizers.md index c9985cc3ff..fdec67b220 100644 --- a/content/en/docs/concepts/overview/working-with-objects/finalizers.md +++ b/content/en/docs/concepts/overview/working-with-objects/finalizers.md @@ -44,7 +44,8 @@ and the controller deletes the volume. ## Owner references, labels, and finalizers {#owners-labels-finalizers} -Like {{}}, [owner references](/docs/concepts/overview/working-with-objects/owners-dependents/) +Like {{}}, +[owner references](/docs/concepts/overview/working-with-objects/owners-dependents/) describe the relationships between objects in Kubernetes, but are used for a different purpose. When a {{}} manages objects From f4bea11f53f1255d750d1300ed5185510dcece20 Mon Sep 17 00:00:00 2001 From: afro-coder Date: Tue, 19 Oct 2021 18:12:21 +0530 Subject: [PATCH 212/249] Added css to improve user experience with docs --- assets/scss/_custom.scss | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index 20a36b6175..96151865e8 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -378,6 +378,12 @@ body.td-home .deprecation-warning, body.td-blog .deprecation-warning, body.td-do border-radius: 3px; } + +.td-documentation .td-content > .highlight { + max-width: initial; + width: 100%; +} + body.td-home #deprecation-warning { max-width: 1000px; margin-top: 2.5rem; From 33c363a37045ade35e42dcce2a639a71b307ac4f Mon Sep 17 00:00:00 2001 From: ixodie Date: Tue, 19 Oct 2021 09:19:25 -0400 Subject: [PATCH 213/249] Added open contiv repo Adding back repo and removing broken link --- content/en/docs/concepts/cluster-administration/networking.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md index 424b889078..7a9b79cd46 100644 --- a/content/en/docs/concepts/cluster-administration/networking.md +++ b/content/en/docs/concepts/cluster-administration/networking.md @@ -164,6 +164,10 @@ network complexity required to deploy Kubernetes at scale within AWS. [Coil](https://github.com/cybozu-go/coil) is a CNI plugin designed for ease of integration, providing flexible egress networking. Coil operates with a low overhead compared to bare metal, and allows you to define arbitrary egress NAT gateways for external networks. +### Contiv + +[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. + ### Contrail / Tungsten Fabric [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is a truly open, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide different isolation modes for virtual machines, containers/pods and bare metal workloads. From 5ad520a6ab45481dcb42c215ae8a4aa23b5c320c Mon Sep 17 00:00:00 2001 From: robotjellyzone Date: Wed, 20 Oct 2021 11:46:52 +0530 Subject: [PATCH 214/249] Fixing Skew localization doc --- content/en/docs/contribute/localization.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/en/docs/contribute/localization.md b/content/en/docs/contribute/localization.md index c9cd5a5385..6d072fea41 100644 --- a/content/en/docs/contribute/localization.md +++ b/content/en/docs/contribute/localization.md @@ -346,7 +346,7 @@ Teams must merge localized content into the same branch from which the content w For example: - a localization branch sourced from `main` must be merged into `main`. -- a localization branch sourced from `release-{{ skew "prevMinorVersion" }}` must be merged into `release-{{ skew "prevMinorVersion" }}`. +- a localization branch sourced from `release-{{% skew "prevMinorVersion" %}}` must be merged into `release-{{% skew "prevMinorVersion" %}}`. {{< note >}} If your localization branch was created from `main` branch but it is not merged into `main` before new release branch `{{< release-branch >}}` created, merge it into both `main` and new release branch `{{< release-branch >}}`. To merge your localization branch into new release branch `{{< release-branch >}}`, you need to switch upstream branch of your localization branch to `{{< release-branch >}}`. @@ -361,5 +361,3 @@ For more information about working from forks or directly from the repository, s ## Upstream contributions SIG Docs welcomes upstream contributions and corrections to the English source. - - From 5f2dd40003db3b8130c48e8e3ae3e8eb1c4aa83d Mon Sep 17 00:00:00 2001 From: robotjellyzone Date: Wed, 20 Oct 2021 14:16:37 +0530 Subject: [PATCH 215/249] added two spaces at end --- content/en/docs/contribute/localization.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/en/docs/contribute/localization.md b/content/en/docs/contribute/localization.md index 6d072fea41..c840ed06ad 100644 --- a/content/en/docs/contribute/localization.md +++ b/content/en/docs/contribute/localization.md @@ -361,3 +361,5 @@ For more information about working from forks or directly from the repository, s ## Upstream contributions SIG Docs welcomes upstream contributions and corrections to the English source. + + From 4ec23262f4b46f9c155900aad96d7ed66804e09c Mon Sep 17 00:00:00 2001 From: sandipanpanda <87253083+sandipanpanda@users.noreply.github.com> Date: Wed, 20 Oct 2021 16:43:30 +0530 Subject: [PATCH 216/249] Update adding-windows-nodes.md --- .../tasks/administer-cluster/kubeadm/adding-windows-nodes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md b/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md index d70669a93e..e7a1b8fb79 100644 --- a/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md +++ b/content/ko/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md @@ -153,7 +153,7 @@ Install-WindowsFeature -Name containers ``` 도커 설치 -자세한 내용은 [도커 엔진 설치 - 윈도우 서버 엔터프라이즈](https://hub.docker.com/editions/enterprise/docker-ee-server-windows)에서 확인할 수 있다. +자세한 내용은 [도커 엔진 설치 - 윈도우 서버 엔터프라이즈](https://docs.microsoft.com/ko-kr/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server#install-docker)에서 확인할 수 있다. #### wins, kubelet 및 kubeadm 설치 From 2fbb74478311dea60fee52bcbe74aa0d1cdc62fa Mon Sep 17 00:00:00 2001 From: robotjellyzone Date: Wed, 20 Oct 2021 17:48:49 +0530 Subject: [PATCH 217/249] added one space at end --- content/en/docs/contribute/localization.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/docs/contribute/localization.md b/content/en/docs/contribute/localization.md index c840ed06ad..592b59ded9 100644 --- a/content/en/docs/contribute/localization.md +++ b/content/en/docs/contribute/localization.md @@ -361,5 +361,4 @@ For more information about working from forks or directly from the repository, s ## Upstream contributions SIG Docs welcomes upstream contributions and corrections to the English source. - - + From b67a853e93d2153a4b5ab24c8b1f208a7f9dbc6d Mon Sep 17 00:00:00 2001 From: superleo Date: Fri, 8 Oct 2021 23:27:32 +0800 Subject: [PATCH 218/249] Concept files to sync for 1.22 - task13 - k8s Extension --- .../api-extension/apiserver-aggregation.md | 7 ++++--- .../compute-storage-net/device-plugins.md | 15 +++++++++++++-- .../docs/concepts/extend-kubernetes/operator.md | 13 +++++++------ 3 files changed, 24 insertions(+), 11 deletions(-) diff --git a/content/zh/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.md b/content/zh/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.md index 738ba23e06..d2a0c6c188 100644 --- a/content/zh/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.md +++ b/content/zh/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.md @@ -23,12 +23,13 @@ The aggregation layer allows Kubernetes to be extended with additional APIs, bey 而不局限于 Kubernetes 核心 API 提供的功能。 -这里的附加 API 可以是[服务目录](/zh/docs/concepts/extend-kubernetes/service-catalog/) -这类已经成熟的解决方案，也可以是你自己开发的 API。 +这里的附加 API 可以是现成的解决方案比如 +[metrics server](https://github.com/kubernetes-sigs/metrics-server), +或者你自己开发的 API。 聚合层不同于 [定制资源（Custom Resources）](/zh/docs/concepts/extend-kubernetes/api-extension/custom-resources/)。 diff --git a/content/zh/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/zh/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index a25019b3a0..6fdc0a371f 100644 --- a/content/zh/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/zh/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -334,10 +334,12 @@ service PodResourcesLister { 这一 `List` 端点提供运行中 Pods 的资源信息，包括类似独占式分配的 CPU ID、设备插件所报告的设备 ID 以及这些设备分配所处的 NUMA 节点 ID。 +此外，对于基于 NUMA 的机器，它还会包含为容器保留的内存和大页的信息。 ```gRPC // ListPodResourcesResponse 是 List 函数的响应 @@ -357,6 +359,14 @@ message ContainerResources { string name = 1; repeated ContainerDevices devices = 2; repeated int64 cpu_ids = 3; + repeated ContainerMemory memory = 4; +} + +// ContainerMemory 包含分配给容器的内存和大页信息 +message ContainerMemory { + string memory_type = 1; + uint64 size = 2; + TopologyInfo topology = 3; } // Topology 描述资源的硬件拓扑结构 @@ -390,6 +400,7 @@ It provides more information than kubelet exports to APIServer. message AllocatableResourcesResponse { repeated ContainerDevices devices = 1; repeated int64 cpu_ids = 2; + repeated ContainerMemory memory = 3; } ``` @@ -397,7 +408,7 @@ message AllocatableResourcesResponse { `ContainerDevices` 会向外提供各个设备所隶属的 NUMA 单元这类拓扑信息。 NUMA 单元通过一个整数 ID 来标识，其取值与设备插件所报告的一致。 diff --git a/content/zh/docs/concepts/extend-kubernetes/operator.md b/content/zh/docs/concepts/extend-kubernetes/operator.md index 97326b31cd..9ccbc16c1d 100644 --- a/content/zh/docs/concepts/extend-kubernetes/operator.md +++ b/content/zh/docs/concepts/extend-kubernetes/operator.md @@ -98,8 +98,7 @@ Some of the things that you can use an operator to automate include: * 在没有内部成员选举程序的情况下，为分布式应用选择首领角色 -想要更详细的了解 Operator？这儿有一个详细的示例： +想要更详细的了解 Operator？下面是一个示例： 1. 有一个名为 SampleDB 的自定义资源，你可以将其配置到集群中。 2. 一个包含 Operator 控制器部分的 Deployment，用来确保 Pod 处于运行状态。 @@ -216,8 +215,8 @@ Operator. * [Charmed Operator Framework](https://juju.is/) * [kubebuilder](https://book.kubebuilder.io/) * [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator) -* [Metacontroller](https://metacontroller.app/) along with WebHooks that - you implement yourself +* [Metacontroller](https://metacontroller.github.io/metacontroller/intro.html) along with WebHooks that +you implement yourself * [Operator Framework](https://operatorframework.io) * [shell-operator](https://github.com/flant/shell-operator) --> @@ -228,13 +227,14 @@ Operator. * [Charmed Operator Framework](https://juju.is/) * [kubebuilder](https://book.kubebuilder.io/) * [KUDO](https://kudo.dev/) (Kubernetes 通用声明式 Operator) -* [Metacontroller](https://metacontroller.app/)，可与 Webhooks 结合使用，以实现自己的功能。 +* [Metacontroller](https://metacontroller.github.io/metacontroller/intro.html)，可与 Webhooks 结合使用，以实现自己的功能。 * [Operator Framework](https://operatorframework.io) * [shell-operator](https://github.com/flant/shell-operator) ## {{% heading "whatsnext" %}} +* 阅读 {{< glossary_tooltip text="CNCF" term_id="cncf" >}} [Operator 白皮书](https://github.com/cncf/tag-app-delivery/blob/eece8f7307f2970f46f100f51932db106db46968/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md)。 * 详细了解 [定制资源](/zh/docs/concepts/extend-kubernetes/api-extension/custom-resources/) * 在 [OperatorHub.io](https://operatorhub.io/) 上找到现成的、适合你的 Operator * [发布](https://operatorhub.io/)你的 Operator，让别人也可以使用 From c25da5c007fc0eca07f77a5a0e8d32ba1afd9023 Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 16:44:02 +0200 Subject: [PATCH 219/249] fix(homepage): kubecon dates --- content/de/_index.html | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/de/_index.html b/content/de/_index.html index 78d3b5e003..871b7c41d3 100644 --- a/content/de/_index.html +++ b/content/de/_index.html @@ -42,13 +42,14 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or

    - Besuche die KubeCon - 13-16 August 2020 in Amsterdam + Besuche die KubeCon North America vom 11. bis 15. Oktober 2021/a>



    -     Besuche die KubeCon - 17-20 November 2020 in Boston + Besuche die KubeCon Europe vom 17. bis 20. Mai 2022 +
    From 260f0231f576fd7d091a235ee86e2889d3bf4134 Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 17:12:19 +0200 Subject: [PATCH 220/249] de-fix-kubernetes-partners --- content/de/includes/partner-style.css | 201 ++++++++++++++++++++++++++ content/de/partners/_index.html | 118 +++++---------- 2 files changed, 241 insertions(+), 78 deletions(-) create mode 100644 content/de/includes/partner-style.css diff --git a/content/de/includes/partner-style.css b/content/de/includes/partner-style.css new file mode 100644 index 0000000000..dc120872e0 --- /dev/null +++ b/content/de/includes/partner-style.css @@ -0,0 +1,201 @@ +/* SECTIONS */ +.section { + clear: both; + padding: 0px; + margin-bottom: 2em; +} + +.kcsp_section { + clear: both; + padding: 0px; + margin-bottom: 2em; +} + +/* COLUMN SETUP */ +.col { + display: block; + float:left; + margin: 1% 0 1% 1.6%; + background-color: #f9f9f9; +} +.col:first-child { margin-left: 0; } + + +/* GROUPING */ +.group:before, +.group:after { + content:""; + display:table; +} +.group:after { + clear:both; +} +.group { + zoom:1; /* For IE 6/7 */ +} + +/* GRID OF THREE */ +.span_3_of_3 { + width: 35%; + background-color: #f9f9f9; + padding: 20px; +} +.span_2_of_3 { + width: 35%; + background-color: #f9f9f9; + padding: 20px; +} +.span_1_of_3 { + width: 35%; + background-color: #f9f9f9; + padding: 20px; +} + +.col-container { + display: table; /* Make the container element behave like a table */ + width: 100%; /* Set full-width to expand the whole page */ + padding-bottom: 30px; +} + +.col-nav { + display: table-cell; /* Make elements inside the container behave like table cells */ + width: 18%; + background-color: #f9f9f9; + padding: 20px; + border: 5px solid white; +} + +/* GO FULL WIDTH AT LESS THAN 480 PIXELS */ + +@media only screen and (max-width: 480px) { + .col { margin: 1% 0 1% 0%;} + .span_3_of_3, .span_2_of_3, .span_1_of_3 { width: 100%; } +} + +@media only screen and (max-width: 650px) { + .col-nav { + display: block; + width: 100%; + } +} + +.button{ + max-width: 100%; + box-sizing: border-box; + margin: 0; + display: inline-block; + border-radius: 6px; + padding: 0 20px; + line-height: 40px; + color: #ffffff; + font-size: 16px; + background-color: #3371e3; + text-decoration: none; + } + +h5 { + font-size: 16px; + line-height: 1.5em; + margin-bottom: 2em; +} + +#usersGrid a { + display: inline-block; + background-color: #f9f9f9; +} + +#ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer { + position: relative; + width: 100%; + display: flex; + justify-content: space-between; + flex-wrap: wrap; +} + +#isvContainer { + margin-bottom: 80px; +} + +#kcspContainer { + margin-bottom: 80px; +} + +#distContainer { + margin-bottom: 80px; +} + +#ktpContainer { + margin-bottom: 80px; +} + +.partner-box { + position: relative; + width: 47%; + max-width: 48%; + min-width: 48%; + margin-bottom: 20px; + padding: 20px; + flex: 1; + display: flex; + justify-content: left; + align-items: flex-start; +} + +.partner-box img { + background-color: #f9f9f9; +} + +.partner-box > div { + margin-left: 30px; +} + +.partner-box a { + color: #3576E3; +} + +@media screen and (max-width: 1024px) { + .partner-box { + flex-direction: column; + justify-content: flex-start; + } + + .partner-box > div { + margin: 20px 0 0; + } +} + +@media screen and (max-width: 568px) { + #ktpContainer, #distContainter, #kcspContainer, #isvContainer, #servContainer { + justify-content: center; + } + + .partner-box { + flex-direction: column; + justify-content: flex-start; + width: 100%; + max-width: 100%; + min-width: 100%; + } + + .partner-box > div { + margin: 20px 0 0; + } +} + +@media screen and (max-width: 568px) { + #ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer { + justify-content: center; + } + + .partner-box { + flex-direction: column; + justify-content: flex-start; + width: 100%; + max-width: 100%; + min-width: 100%; + } + + .partner-box > div { + margin: 20px 0 0; + } +} diff --git a/content/de/partners/_index.html b/content/de/partners/_index.html index e99b03b62a..fa18115b6a 100644 --- a/content/de/partners/_index.html +++ b/content/de/partners/_index.html @@ -1,91 +1,53 @@ --- title: Partner bigheader: Kubernetes Partner -abstract: Entwicklung des Kubernetes-Ökosystems. +abstract: Erweiterung des Kubernetes-Ökosystems. class: gridPage cid: partners ---
    -
    -
    Kubernetes arbeitet mit Partnern zusammen, um eine starke, dynamische Codebasis zu schaffen, die ein Spektrum von aufeinander abgestimmten Plattformen unterstützt.
    -
    -
    -
    -
    - Kubernetes zertifizierte Service Provider -
    -
    Geprüfte Service Provider mit großer Erfahrung, die Unternehmen bei der erfolgreichen Einführung von Kubernetes unterstützen. -


    - -

    Interessiert daran, ein KCSP zu werden? -
    -
    -
    -
    -
    - Kubernetes-Distributionen, gehostete Plattformen und zertifizierte Installateure -
    Software-Konformität stellt sicher, dass die Kubernetes-Versionen aller Hersteller die erforderlichen APIs unterstützen. -


    - -

    Interessiert daran, Kubernetes zertifiziert zu werden? -
    -
    -
    -
    -
    Kubernetes Training Partner
    -
    Geprüfte Schulungsanbieter, die über umfassende Erfahrung in Cloud Native Technologietrainings verfügen. -



    - -

    Interessiert daran, ein KTP zu werden? -
    -
    -
    - - - -
    - - +
    Kubernetes arbeitet mit Partnern zusammen, um eine starke, lebendige Codebasis zu schaffen, die ein Spektrum von ergänzenden Plattformen unterstützt.
    +
    +
    +
    +
    + Kubernetes-zertifizierte Service-Anbieter +
    +
    Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen. +


    + +

    Interessiert ein + KCSP zu werden? +
    +
    +
    +
    +
    + Zertifizierte Kubernetes-Distributionen, gehostete Plattformen und Installationssysteme +
    Die Softwarekonformität stellt sicher, dass die Kubernetes-Version eines jeden Anbieters die erforderlichen APIs unterstützt. +


    + +

    Interessiert daran, + Kubernetes Zertifiziert zu werden? +
    +
    +
    +
    +
    + Kubernetes Schulungspartner +
    +
    Geprüfte Schulungsanbieter mit umfassender Erfahrung in der Weiterbildung im Bereich Cloud Native Technology. +


    + +

    Interessiert daran, ein + KTP zu werden? +
    +
    - -
    + {{< cncf-landscape helpers=true >}}
    - - + \ No newline at end of file From 897969944d1c39456354adc9e28c193c68676cbd Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 17:13:18 +0200 Subject: [PATCH 221/249] de-fix-kubernetes-partners --- content/de/partners/_index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/de/partners/_index.html b/content/de/partners/_index.html index fa18115b6a..21ce662f25 100644 --- a/content/de/partners/_index.html +++ b/content/de/partners/_index.html @@ -17,7 +17,7 @@ cid: partners
    Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen.


    -

    Interessiert ein +

    Interessiert daran, ein KCSP zu werden?
    From a12ab9693ffc3bb48562ab96c21c0b632065ce6b Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 17:41:21 +0200 Subject: [PATCH 222/249] de-feat-community-20211020 --- content/de/community/_index.html | 297 +++++++++++++++++++++++++------ 1 file changed, 246 insertions(+), 51 deletions(-) diff --git a/content/de/community/_index.html b/content/de/community/_index.html index adff754f03..6ca60f3a8e 100644 --- a/content/de/community/_index.html +++ b/content/de/community/_index.html @@ -4,58 +4,253 @@ layout: basic cid: community --- -
    -
    -
    -

    Die Gewissheit, dass Kubernetes überall und für alle gut funktioniert.

    -

    Verbinden Sie sich mit der Kubernetes-Community in unserem Slack Kanal, Diskussionsforum, oder beteiligen Sie sich an der Kubernetes-dev-Google-Gruppe. Eine wöchentliches Community-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie - diesen Anweisungen für Informationen wie Sie teilnehmen können.

    -

    Sie können Kubernetes auch auf der ganzen Welt über unsere - Kubernetes Meetup Community und der - Kubernetes Cloud Native Meetup Community beitreten.

    -
    -
    -

    Special Interest Groups (SIGs)

    -

    Haben Sie ein besonderes Interesse daran, wie Kubernetes mit einer anderen Technologie arbeitet? Werfen Sie einen Blick auf unsere kontinuierlich wachsende - Listen von SIGs, von AWS und Openstack bis hin zu Big Data und Skalierbarkeit, es gibt einen Platz für Sie, an dem Sie mitwirken können, und Anweisungen zur Gründung einer neuen SIG finden, wenn Ihr besonderes Interesse (noch) nicht abgedeckt ist. -

    +
    +
    + Kubernetes Conference Gallery + Kubernetes Conference Gallery +
    -

    Als Mitglied der Kubernetes-Community sind Sie herzlich eingeladen, an allen SIG-Treffen teilzunehmen, die Sie interessieren. Eine Registrierung ist nicht erforderlich.

    +
    +
    +

    Die Kubernetes-Community - Nutzer, Mitwirkende und die Kultur, die wir gemeinsam aufgebaut haben - ist einer der Hauptgründe für den kometenhaften Aufstieg dieses Open-Source-Projekts. Unsere Kultur und unsere Werte wachsen und entwickeln sich mit dem Wachstum und der Veränderung des Projekts selbst. Wir alle arbeiten gemeinsam an der ständigen Verbesserung des Projekts und der Art und Weise, wie wir daran arbeiten. +

    Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen, Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, kubectl get pods ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.

    +
    +
    -
    +
    -
    -

    Verhaltensregeln

    -

    Die Kubernetes-Community schätzt Respekt und Inklusivität und setzt einen Verhaltenskodex - in allen Interaktionen durch. Wenn Sie einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder Sitzung, - in Slack oder in einem anderen Kommunikationsmechanismus feststellen, wenden Sie sich - bitte an das Kubernetes Code of Conduct Committee conduct@kubernetes.io. Ihre Anonymität wird geschützt. -

    -
    -
    -
    +Gemeinschaftswerte      +Verhaltenskodex       +Videos      +Diskussionen      +Events and meetups      +Neuigkeiten      +Releases -
    -
    -

    Talk to Us!

    -

    Wir würden uns freuen, von Ihnen zu hören, wie Sie Kubernetes verwenden
    und was wir tun können, um es besser zu machen.

    -
    -
    - @kubernetesio -

    Erhalten Sie die neuesten Nachrichten und Updates.

    -
    -
    - Github Project -

    Informieren Sie sich über das Projekt und erwägen Sie, einen Beitrag zu leisten.

    -
    -
    - #kubernetes-users -

    Unser Slack-Kanal ist der beste Weg, um unsere Ingenieure zu kontaktieren und Ihre Ideen mit ihnen zu teilen.

    -
    -
    - Stack Overflow -

    Unser Benutzerforum ist ein großartiger Ort, um Community-Support zu erhalten.

    -
    -
    -
    -
    + +

    +
    +
    +
    + Kubernetes Conference Gallery +
    + +
    + Kubernetes Conference Gallery +
    + +
    + Kubernetes Conference Gallery +
    + Kubernetes Conference Gallery + +
    + +
    +
    +
    +

    +

    +

    Gemeinschaftswerte

    +Die Werte der Kubernetes-Community sind der Grundstein für den anhaltenden Erfolg des Projekts.
    +Diese Prinzipien leiten jeden Aspekt des Kubernetes-Projekts. +
    + +

    + + MEHR ERFAHREN + +     +
    +
    +
    + + + +
    +
    +

    +

    +

    Verhaltenskodex

    +Die Kubernetes-Gemeinschaft legt Wert auf Respekt und Inklusivität und setzt bei allen Interaktionen einen Verhaltenskodex durch. Wenn Du einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder einem Treffen, in Slack oder in einem anderen Kommunikationsmechanismus bemerkst, wende dich an das Kubernetes Code of Conduct Committee unter conduct@kubernetes.io. Alle Berichte werden vertraulich behandelt. Du kannst hier mehr über den Ausschuss erfahren. +
    + +

    + + +MEHR ERFAHREN + +     +
    +
    +
    + + + +
    +

    +

    +

    Videos

    + +
    Wir sind auf YouTube, und zwar oft. Abonniere uns für eine Vielzahl von Themen.
    + +
    + +
    + + + + +
    +Monatliche Bürozeiten ansehen ▶
    +     +
    + +
    + + +
    +Wöchentliche Treffen der Gemeinschaft ansehen ▶ +
    +     +
    + +
    + + + + +
    +Vortrag eines Mitglieds der Gemeinschaft ansehen ▶ +
    + +     + +
    +
    +
    + + +
    +

    +

    +

    Diskussionen

    + +
    Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige Dich an den Diskussionen.
    + +
    + +
    +Forum" + +forum ▶ + +
    +Themenbezogene technische Diskussionen, die eine Brücke zu Docs, StackOverflow und vielem mehr schlagen. +
    +
    + +
    +Twitter + +twitter ▶ + +
    Echtzeit-Ankündigungen von Blogeinträgen, Veranstaltungen, Neuigkeiten und Ideen +
    +
    + +
    +GitHub + +github ▶ + +
    +Die gesamte Projekt- und Problemverfolgung und natürlich der Code +
    +
    + +
    +Stack Overflow + +stack overflow ▶ + +
    + Technische Problemlösung für jeden Anwendungsfall + +
    +
    + + + +
    +
    +
    +

    +

    +
    +

    Bevorstehende Veranstaltungen

    + {{< upcoming-events >}} +
    +
    + +
    +
    +
    +

    Globale Gemeinschaft

    +Mit mehr als 150 Treffen auf der ganzen Welt, Tendenz steigend, solltest du deine lokalen Kube-Leute finden. Wenn keins in der Nähe ist, nimm die Sache in die Hand und gründe dein eigenes. +
    + +
    +EIN MEETUP FINDEN +
    +     + +
    +
    + + + + +
    +

    +

    +

    Aktuelle Neuigkeiten

    + +
    +
    + +
    + +
    +



    +
    + + From af2f72f6032236f5c567881ec1b9829c2782a4a4 Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 17:51:03 +0200 Subject: [PATCH 223/249] de-fix-kubecon-dates-20211020 --- content/de/_index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/de/_index.html b/content/de/_index.html index 871b7c41d3..a148cad7d4 100644 --- a/content/de/_index.html +++ b/content/de/_index.html @@ -42,7 +42,7 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or

    - Besuche die KubeCon North America vom 11. bis 15. Oktober 2021/a> + Besuche die KubeCon North America vom 11. bis 15. Oktober 2021


    From 64e4933cf6f5988209c326b6d3fe44576c39a1bc Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 20:11:08 +0200 Subject: [PATCH 224/249] Update content/de/_index.html --- content/de/_index.html | 1 - 1 file changed, 1 deletion(-) diff --git a/content/de/_index.html b/content/de/_index.html index a148cad7d4..fc7926ec93 100644 --- a/content/de/_index.html +++ b/content/de/_index.html @@ -49,7 +49,6 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or
    Besuche die KubeCon Europe vom 17. bis 20. Mai 2022 -
    From d31a7ebcfcc24d72c6f08f959656d2bce0f713db Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 20:33:57 +0200 Subject: [PATCH 225/249] fix typo --- content/de/partners/_index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/de/partners/_index.html b/content/de/partners/_index.html index 21ce662f25..ecf3e2542b 100644 --- a/content/de/partners/_index.html +++ b/content/de/partners/_index.html @@ -16,7 +16,7 @@ cid: partners
    Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen.


    - +

    Interessiert daran, ein KCSP zu werden? From 569b037a130a99b9bbba8f80444ee7aea6534e3a Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Wed, 20 Oct 2021 20:36:02 +0200 Subject: [PATCH 226/249] fix typo --- content/de/community/_index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/de/community/_index.html b/content/de/community/_index.html index 6ca60f3a8e..62797e5d06 100644 --- a/content/de/community/_index.html +++ b/content/de/community/_index.html @@ -136,7 +136,7 @@ Vortrag eines Mitglieds der Gemeinschaft ansehen ▶

    Diskussionen

    -
    Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige Dich an den Diskussionen.
    +
    Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige dich an den Diskussionen.
    From d33605578f88879a08bda7c8bb0a1d29a6f01dc5 Mon Sep 17 00:00:00 2001 From: sandipanpanda <87253083+sandipanpanda@users.noreply.github.com> Date: Thu, 21 Oct 2021 05:26:08 +0530 Subject: [PATCH 227/249] Fix broken link in k8s.io/docs/concepts/overview/working-with-objects/finalizers/ (#30159) * Fix broken link in finalizers.md * Update finalizers.md * Update finalizers.md --- .../docs/concepts/overview/working-with-objects/finalizers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/overview/working-with-objects/finalizers.md b/content/en/docs/concepts/overview/working-with-objects/finalizers.md index 9516b935c9..9c241210db 100644 --- a/content/en/docs/concepts/overview/working-with-objects/finalizers.md +++ b/content/en/docs/concepts/overview/working-with-objects/finalizers.md @@ -44,7 +44,7 @@ and the controller deletes the volume. ## Owner references, labels, and finalizers {#owners-labels-finalizers} -Like {{}}, [owner references](/concepts/overview/working-with-objects/owners-dependents/) +Like {{}}, [owner references](/docs/concepts/overview/working-with-objects/owners-dependents) describe the relationships between objects in Kubernetes, but are used for a different purpose. When a {{}} manages objects From d9db5520afd401e1b167de69a47ebdbdb5d972d7 Mon Sep 17 00:00:00 2001 From: Benedikt Rollik Date: Thu, 21 Oct 2021 15:07:18 +0200 Subject: [PATCH 228/249] fixed some translations --- content/de/community/_index.html | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/content/de/community/_index.html b/content/de/community/_index.html index 62797e5d06..77d6005a37 100644 --- a/content/de/community/_index.html +++ b/content/de/community/_index.html @@ -6,14 +6,14 @@ cid: community
    - Kubernetes Conference Gallery - Kubernetes Conference Gallery + Kubernetes-Konferenz Galerie + Kubernetes-Konferenz Galerie

    Die Kubernetes-Community - Nutzer, Mitwirkende und die Kultur, die wir gemeinsam aufgebaut haben - ist einer der Hauptgründe für den kometenhaften Aufstieg dieses Open-Source-Projekts. Unsere Kultur und unsere Werte wachsen und entwickeln sich mit dem Wachstum und der Veränderung des Projekts selbst. Wir alle arbeiten gemeinsam an der ständigen Verbesserung des Projekts und der Art und Weise, wie wir daran arbeiten. -

    Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen, Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, kubectl get pods ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.

    +

    Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen (Special Interest Groups), Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, kubectl get pods ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.


    @@ -23,7 +23,7 @@ cid: community Verhaltenskodex       Videos      Diskussionen      -Events and meetups      +Veranstaltungen und meetups      Neuigkeiten      Releases @@ -32,17 +32,17 @@ cid: community

    - Kubernetes Conference Gallery + Kubernetes-Konferenz Galerie
    - Kubernetes Conference Gallery + Kubernetes-Konferenz Galerie
    - Kubernetes Conference Gallery + Kubernetes-Konferenz Galerie
    - Kubernetes Conference Gallery + Kubernetes-Konferenz Galerie
    From c9847dafe284f2f9f5e88f75c5b0bca73dc5a4de Mon Sep 17 00:00:00 2001 From: Lucas Basquerotto Date: Thu, 21 Oct 2021 10:57:42 -0300 Subject: [PATCH 229/249] Fix typo at network-policies.md --- .../en/docs/concepts/services-networking/network-policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/services-networking/network-policies.md b/content/en/docs/concepts/services-networking/network-policies.md index e3f3a203b7..108c2be59b 100644 --- a/content/en/docs/concepts/services-networking/network-policies.md +++ b/content/en/docs/concepts/services-networking/network-policies.md @@ -261,7 +261,7 @@ The following restrictions apply when using this field: at a cluster level, you (or your cluster administrator) need to disable the `NetworkPolicyEndPort` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the API server with `--feature-gates=NetworkPolicyEndPort=false,…`. -* The `endPort` field must be equal than or greater to the `port` field. +* The `endPort` field must be equal to or greater than the `port` field. * `endPort` can only be defined if `port` is also defined. * Both ports must be numeric. From e4177d31ee9c26311cb1bf08788c9a60dd6cd34a Mon Sep 17 00:00:00 2001 From: Arhell Date: Sat, 23 Oct 2021 00:35:38 +0300 Subject: [PATCH 230/249] [id] Including Oracle Cloud Infrastructure --- content/id/docs/concepts/security/overview.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/id/docs/concepts/security/overview.md b/content/id/docs/concepts/security/overview.md index bc271e0645..0346314f3a 100644 --- a/content/id/docs/concepts/security/overview.md +++ b/content/id/docs/concepts/security/overview.md @@ -36,6 +36,7 @@ Amazon Web Services | https://aws.amazon.com/security/ | Google Cloud Platform | https://cloud.google.com/security/ | IBM Cloud | https://www.ibm.com/cloud/security | Microsoft Azure | https://docs.microsoft.com/en-us/azure/security/azure-security | +Oracle Cloud Infrastructure | https://www.oracle.com/security/ | VMWare VSphere | https://www.vmware.com/security/hardening-guides.html | Jika kamu mengoperasikan perangkat keras kamu sendiri, atau penyedia layanan cloud yang berbeda, kamu perlu merujuk pada dokumentasi penyedia layanan cloud yang kamu pakai untuk praktik keamanan terbaik. From 64b5b16bac9abb0de3d22fc5489134a5cb8ec5a8 Mon Sep 17 00:00:00 2001 From: Richie McG Date: Sat, 23 Oct 2021 17:29:03 +0900 Subject: [PATCH 231/249] Correct Typo Correct typo form etcdutl to etcdctl on line 333 --- .../en/docs/tasks/administer-cluster/configure-upgrade-etcd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md index 3dececd8e0..9e0b49480b 100644 --- a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md +++ b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md @@ -327,7 +327,7 @@ ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb ``` Another example for restoring using etcdutl options: ```shell -ETCDCTL_API=3 etcdutl --data-dir snapshot restore snapshotdb +ETCDCTL_API=3 etcdctl --data-dir snapshot restore snapshotdb ``` For more information and examples on restoring a cluster from a snapshot file, see From 032f1bc7fd3ec1540ab62c669e9a28f76b50bbcc Mon Sep 17 00:00:00 2001 From: Mauren Berti Date: Sat, 23 Oct 2021 14:14:16 -0400 Subject: [PATCH 232/249] Update translation for pt-br strings. --- data/i18n/pt/pt-br.toml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/data/i18n/pt/pt-br.toml b/data/i18n/pt/pt-br.toml index 3f55a665b3..3301cd3295 100644 --- a/data/i18n/pt/pt-br.toml +++ b/data/i18n/pt/pt-br.toml @@ -32,7 +32,7 @@ other = "Descontinuado" other = "Você está vendo a documentação do Kubernetes versão:" [deprecation_warning] -other = " a documentação não é mais mantida ativamente. A versão que você está visualizando no momento é uma captura instantânea estática. Para obter documentação atualizada, consulte " +other = " a documentação não é mais mantida ativamente. A versão que você está visualizando no momento é uma cópia estática. Para obter a documentação atualizada, consulte " [docs_label_browse] other = "Procurar documentos" @@ -98,7 +98,7 @@ other = "indicadores abaixo para uma maior explicação sobre um termo em partic other = "Clique nos" [layouts_docs_glossary_description] -other = "Este glossário pretende ser uma lista padronizada e abrangente da terminologia do Kubernetes. Inclui termos técnicos específicos dos K8s, além de termos mais gerais que fornecem um contexto útil." +other = "Este glossário tem por objetivo ser uma lista padronizada e abrangente da terminologia do Kubernetes. Inclui termos técnicos específicos do K8s, além de termos mais gerais que fornecem um contexto útil." [layouts_docs_glossary_deselect_all] other = "Desmarcar tudo" @@ -119,7 +119,7 @@ other = "Abra um bug no repositório do GitHub se você deseja " other = "ou" [layouts_docs_partials_feedback_problem] -other = "reportar um problema" +other = "relatar um problema" [layouts_docs_partials_feedback_thanks] other = "Obrigado pelo feedback. Se você tiver uma pergunta específica sobre como utilizar o Kubernetes, faça em" @@ -133,7 +133,7 @@ other = "Buscando resultados.." other = "por" [main_cncf_project] -other = """Nós somos uma CNCF projeto graduado

    """ +other = """Nós somos um projeto graduado da CNCF

    """ [main_community_explore] other = "Explore a comunidade" @@ -210,10 +210,10 @@ other = """Esta seção tem links para projetos de terceiros que fornecem a func other = "Procurar" [version_check_mustbeorlater] -other = "O seu servidor Kubernetes deve estar em ou depois da versão " +other = "O seu servidor Kubernetes deve estar numa versão igual ou superior a " [version_check_mustbe] -other = "Seu servidor Kubernetes deve ser versão" +other = "O seu servidor Kubernetes deve estar na versão " [version_check_tocheck] other = "Para verificar a versão, digite " @@ -225,7 +225,7 @@ other = "Versões" other = "Aviso:" [whatsnext_heading] -other = "Qual é o próximo" +other = "Próximos passos" [print_printable_section] other = "Essa é a versão completa de impressão dessa seção" From b0904d232824ecb6a9e4d105cccbdd74e8ff3ef2 Mon Sep 17 00:00:00 2001 From: Zilmar de Souza Junior Date: Sat, 23 Oct 2021 17:08:21 -0300 Subject: [PATCH 233/249] [l10n] PT-BR - docs/concepts/scheduling-eviction/taint-and-toleration.md (#30049) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [l10n] starting pt-br l10n [l10n] Localização para pt-br finalizada * Apply suggestions from code review Co-authored-by: Ricardo Katz Co-authored-by: Ricardo Katz --- .../taint-and-toleration.md | 210 ++++++++++++++++++ .../examples/pods/pod-with-toleration.yaml | 15 ++ 2 files changed, 225 insertions(+) create mode 100644 content/pt-br/docs/concepts/scheduling-eviction/taint-and-toleration.md create mode 100644 content/pt-br/examples/pods/pod-with-toleration.yaml diff --git a/content/pt-br/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/pt-br/docs/concepts/scheduling-eviction/taint-and-toleration.md new file mode 100644 index 0000000000..dc4e362f7d --- /dev/null +++ b/content/pt-br/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -0,0 +1,210 @@ +--- +title: Taints e Tolerâncias +content_type: concept +weight: 40 +--- + + + +[_Afinidade de nó_](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) +é uma propriedade dos {{< glossary_tooltip text="Pods" term_id="pod" >}} que os *associa* a um conjunto de {{< glossary_tooltip text="nós" term_id="node" >}} (seja como uma preferência ou uma exigência). _Taints_ são o oposto -- eles permitem que um nó repudie um conjunto de pods. + +_Tolerâncias_ são aplicadas em pods e permitem, mas não exigem, que os pods sejam alocados em nós com _taints_ correspondentes. + +Taints e tolerâncias trabalham juntos para garantir que pods não sejam alocados em nós inapropriados. Um ou mais taints são aplicados em um nó; isso define que o nó não deve aceitar nenhum pod que não tolera essas taints. + + + + +## Conceitos + +Você adiciona um taint a um nó utilizando [kubectl taint](/docs/reference/generated/kubectl/kubectl-commands#taint). +Por exemplo, + +```shell +kubectl taint nodes node1 key1=value1:NoSchedule +``` + +define um taint no nó `node1`. O taint tem a chave `key1`, valor `value1` e o efeito `NoSchedule`. +Isso significa que nenhum pod conseguirá ser executado no nó `node1` a menos que possua uma tolerância correspondente. + +Para remover o taint adicionado pelo comando acima, você pode executar: +```shell +kubectl taint nodes node1 key1=value1:NoSchedule- +``` + +Você especifica uma tolerância para um pod na [especificação do Pod](/docs/reference/kubernetes-api/workload-resources/pod-v1#PodSpec). Ambas as seguintes tolerâncias "correspondem" ao taint criado pelo `kubectl taint` acima, e assim um pod com qualquer uma delas poderia ser executado no `node1`: + +```yaml +tolerations: +- key: "key1" + operator: "Equal" + value: "value1" + effect: "NoSchedule" +``` + +```yaml +tolerations: +- key: "key1" + operator: "Exists" + effect: "NoSchedule" +``` + +Aqui está um exemplo de um pod que utiliza tolerâncias: + +{{< codenew file="pods/pod-with-toleration.yaml" >}} + +O valor padrão de `operator` é `Equal`. + +Uma tolerância "casa" um taint se as chaves e efeitos são os mesmos, e: + +* o valor de `operator` é `Exists` (no caso nenhum `value` deve ser especificado), ou +* o valor de `operator` é `Equal` e os valores de `value` são iguais. + +{{< note >}} + +Existem dois casos especiais: + +Uma `key` vazia com o operador `Exists` "casa" todas as chaves, valores e efeitos, o que significa que o pod irá tolerar tudo. + +Um `effect` vazio "casa" todos os efeitos com a chave `key1`. + +{{< /note >}} + +O exemplo acima usou `effect` de `NoSchedule`. De forma alternativa, você pode usar `effect` de `PreferNoSchedule`. +Nesse efeito, o sistema *tentará* evitar que o pod seja alocado ao nó caso ele não tolere os taints definidos, contudo a alocação não será evitada de forma obrigatória. Pode-se dizer que o `PreferNoSchedule` é uma versão permissiva do `NoSchedule`. O terceiro tipo de `effect` é o `NoExecute` que será descrito posteriormente. + +Você pode colocar múltiplos taints no mesmo nó e múltiplas tolerâncias no mesmo pod. +O jeito que o Kubernetes processa múltiplos taints e tolerâncias é como um filtro: começa com todos os taints de um nó, em seguida ignora aqueles para os quais o pod tem uma tolerância relacionada; os taints restantes que não foram ignorados indicam o efeito no pod. Mais especificamente, + +* se existe pelo menos um taint não tolerado com o efeito `NoSchedule`, o Kubernetes não alocará o pod naquele nó +* se existe um taint não tolerado com o efeito `NoSchedule`, mas existe pelo menos um taint não tolerado com o efeito `PreferNoSchedule`, o Kubernetes *tentará* não alocar o pod no nó +* se existe pelo menos um taint não tolerado com o efeito `NoExecute`, o pod será expulso do nó (caso já esteja em execução) e não será alocado ao nó (caso ainda não esteja em execução). + +Por exemplo, imagine que você tem um nó com os seguintes taints + +```shell +kubectl taint nodes node1 key1=value1:NoSchedule +kubectl taint nodes node1 key1=value1:NoExecute +kubectl taint nodes node1 key2=value2:NoSchedule +``` + +E um pod com duas tolerâncias: + +```yaml +tolerations: +- key: "key1" + operator: "Equal" + value: "value1" + effect: "NoSchedule" +- key: "key1" + operator: "Equal" + value: "value1" + effect: "NoExecute" +``` + +Nesse caso, o pod não será alocado ao nó porque não possui uma tolerância para o terceiro taint. Porém, se ele já estiver rodando no nó quando o taint foi adicionado, não será afetado e continuará rodando, tendo em vista que o terceiro taint é o único não tolerado pelo pod. + +Normalmente, se um taint com o efeito `NoExecute` é adicionado a um nó, qualquer pod que não o tolere será expulso imediatamente e pods que o toleram nunca serão expulsos. Contudo, uma tolerância com efeito `NoExecute` pode especificar de forma opcional o campo `tolerationSeconds`, que determina quanto tempo o pod continuará alocado ao nó depois que o taint é adicionado. Por exemplo, + +```yaml +tolerations: +- key: "key1" + operator: "Equal" + value: "value1" + effect: "NoExecute" + tolerationSeconds: 3600 +``` + +significa que se esse pod está sendo executado e um taint correspondente é adicionado ao nó, o pod irá continuar rodando neste nó por 3600 segundos e depois será expulso. Se o taint for removido antes desse tempo acabar, o pod não será expulso. + +## Exemplos de Casos de Uso + +Taints e tolerâncias são um modo flexível de conduzir pods para *fora* dos nós ou expulsar pods que não deveriam estar sendo executados. Alguns casos de uso são + +* **Nós Dedicados**: Se você quiser dedicar um conjunto de nós para uso exclusivo de um conjunto específico de usuários, poderá adicionar um taint nesses nós. (digamos, `kubectl taint nodes nodename dedicated=groupName:NoSchedule`) e em seguida adicionar uma tolerância correspondente para seus pods (isso seria feito mais facilmente com a escrita de um [controlador de admissão](/docs/reference/access-authn-authz/admission-controllers/) customizado). +Os pods com tolerância terão sua execução permitida nos nós com taints (dedicados), assim como em qualquer outro nó no cluster. Se você quiser dedicar nós a esses pods *e garantir* que eles usem *apenas* os nós dedicados, precisará adicionar uma label similar ao taint para o mesmo conjunto de nós (por exemplo, `dedicated=groupName`), e o controle de admissão deverá adicionar uma afinidade de nó para exigir que os pods podem ser executados apenas nos nós definidos com a label `dedicated=groupName`. + +* **Nós com hardware especial**: Em um cluster no qual um pequeno grupo de nós possui hardware especializado (por exemplo, GPUs), é desejável manter pods que não necessitem desse tipo de hardware fora desses nós, dessa forma o recurso estará disponível para pods que precisem do hardware especializado. Isso pode ser feito aplicando taints nos nós com o hardware especializado (por exemplo, `kubectl taint nodes nodename special=true:NoSchedule` or `kubectl taint nodes nodename special=true:PreferNoSchedule`) e aplicando uma tolerância correspondente nos pods que usam o hardware especial. Assim como no caso de uso de nós dedicados, é provavelmente mais fácil aplicar as tolerâncias utilizando um [controlador de admissão](/docs/reference/access-authn-authz/admission-controllers/). +Por exemplo, é recomendado usar [Extended Resources](/docs/concepts/configuration/manage-resources-containers/#extended-resources) para representar hardware especial, adicione um taint ao seus nós de hardware especializado com o nome do recurso estendido e execute o controle de admissão [ExtendedResourceToleration](/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration). Agora, tendo em vista que os nós estão marcados com um taint, nenhum pod sem a tolerância será executado neles. Porém, quando você submete um pod que requisita o recurso estendido, o controlador de admissão `ExtendedResourceToleration` irá adicionar automaticamente as tolerâncias necessárias ao pod que irá, por sua vez, ser alocado no nó com hardware especial. Isso garantirá que esses nós de hardware especial serão dedicados para os pods que requisitarem tal recurso e você não precisará adicionar manualmente as tolerâncias aos seus pods. + +* **Expulsões baseadas em Taint**: Um comportamento de expulsão configurada por pod quando problemas existem em um nó, o qual será descrito na próxima seção. + +## Expulsões baseadas em Taint + +{{< feature-state for_k8s_version="v1.18" state="stable" >}} + +O efeito de taint `NoExecute`, mencionado acima, afeta pods que já estão rodando no nó da seguinte forma + + * pods que não toleram o taint são expulsos imediatamente + * pods que toleram o taint sem especificar `tolerationSeconds` em sua especificação de tolerância, ficam alocados para sempre + * pods que toleram o taint com um `tolerationSeconds` especificado, permanecem alocados pela quantidade de tempo definida + +O controlador de nó automaticamente adiciona um taint ao Nó quando certas condições se tornam verdadeiras. Os seguintes taints são embutidos: + + * `node.kubernetes.io/not-ready`: Nó não está pronto. Isso corresponde ao NodeCondition `Ready` com o valor "`False`". + * `node.kubernetes.io/unreachable`: Nó é inalcançável a partir do controlador de nó. Isso corresponde ao NodeCondition `Ready` com o valor "`Unknown`". + * `node.kubernetes.io/memory-pressure`: Nó possui pressão de memória. + * `node.kubernetes.io/disk-pressure`: Nó possui pressão de disco. + * `node.kubernetes.io/pid-pressure`: Nó possui pressão de PID. + * `node.kubernetes.io/network-unavailable`: A rede do nó está indisponível. + * `node.kubernetes.io/unschedulable`: Nó não é alocável. + * `node.cloudprovider.kubernetes.io/uninitialized`: Quando o kubelet é iniciado com um provedor de nuvem "externo", esse taint é adicionado ao nó para que ele seja marcado como não utilizável. Após o controlador do cloud-controller-manager inicializar o nó, o kubelet remove esse taint. + +No caso de um nó estar prestes a ser expulso, o controlador de nó ou kubelet adicionam os taints relevantes com o efeito `NoExecute`. Se a condição de falha retorna ao normal, o kubelet ou controlador de nó podem remover esses taints. + +{{< note >}} +A camada de gerenciamento limita a taxa de adição de novos taints aos nós. Esse limite gerencia o número de expulsões que são disparadas quando muitos nós se tornam inalcançáveis ao mesmo tempo (por exemplo: se ocorre uma falha na rede). +{{< /note >}} + +Você pode especificar `tolerationSeconds` em um Pod para definir quanto tempo ele ficará alocado em um nó que está falhando ou está sem resposta. + +Por exemplo, você talvez queira manter uma aplicação com vários estados salvos localmente alocado em um nó por um longo período na ocorrência de uma divisão na rede, esperando que essa divisão se recuperará e assim a expulsão do pod pode ser evitada. +A tolerância que você define para esse Pod poderia ficar assim: + +```yaml +tolerations: +- key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 6000 +``` + +{{< note >}} +O Kubernetes automaticamente adiciona uma tolerância para `node.kubernetes.io/not-ready` e `node.kubernetes.io/unreachable` com `tolerationSeconds=300`, a menos que você, ou um controlador, defina essas tolerâncias explicitamente. + +Essas tolerâncias adicionadas automaticamente significam que Pods podem continuar alocados aos Nós por 5 minutos após um desses problemas ser detectado. +{{< /note >}} + +Pods do tipo [DaemonSet](/docs/concepts/workloads/controllers/daemonset/) são criados com tolerâncias `NoExecute` sem a propriedade `tolerationSeconds` para os seguintes taints: + + * `node.kubernetes.io/unreachable` + * `node.kubernetes.io/not-ready` + +Isso garante que esses pods do DaemonSet nunca sejam expulsos por conta desses problemas. + +## Taints por condições de nó + +A camada de gerenciamento, usando o {{}} do nó, cria taints automaticamente com o efeito `NoSchedule` para [condições de nó](/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions). + +O agendador verifica taints, não condições de nó, quando realiza suas decisões de agendamento. Isso garante que as condições de nó não afetem diretamente o agendamento. +Por exemplo, se a condição de nó `DiskPressure` está ativa, a camada de gerenciamento adiciona o taint `node.kubernetes.io/disk-pressure` e não aloca novos pods no nó afetado. Se a condição `MemoryPressure` está ativa, a camada de gerenciamento adiciona o taint `node.kubernetes.io/memory-pressure`. + +Você pode ignorar condições de nó para pods recém-criados adicionando tolerâncias correspondentes. A camada de controle também adiciona a tolerância `node.kubernetes.io/memory-pressure` em pods que possuem uma {{< glossary_tooltip text="classe de QoS" term_id="qos-class" >}} diferente de `BestEffort`. Isso ocorre porque o Kubernetes trata pods nas classes de QoS `Guaranteed` ou `Burstable` (até mesmo pods sem requisitos de memória definidos) como se fossem capazes de lidar com pressão de memória, enquanto novos pods com `BestEffort` não são alocados no nó afetado. + +O controlador DaemonSet adiciona automaticamente as seguintes tolerâncias de `NoSchedule` para todos os daemons, prevenindo que DaemonSets quebrem. + + * `node.kubernetes.io/memory-pressure` + * `node.kubernetes.io/disk-pressure` + * `node.kubernetes.io/pid-pressure` (1.14 ou superior) + * `node.kubernetes.io/unschedulable` (1.10 ou superior) + * `node.kubernetes.io/network-unavailable` (*somente rede do host*) + +Adicionando essas tolerâncias garante retro compatibilidade. Você também pode adicionar tolerâncias de forma arbitrária aos DaemonSets. + +## {{% heading "whatsnext" %}} + +* Leia sobre [Node-pressure Eviction](/docs/concepts/scheduling-eviction/node-pressure-eviction/) e como você pode configurá-la +* Leia sobre [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/) + + diff --git a/content/pt-br/examples/pods/pod-with-toleration.yaml b/content/pt-br/examples/pods/pod-with-toleration.yaml new file mode 100644 index 0000000000..79f2756a8c --- /dev/null +++ b/content/pt-br/examples/pods/pod-with-toleration.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Pod +metadata: + name: nginx + labels: + env: test +spec: + containers: + - name: nginx + image: nginx + imagePullPolicy: IfNotPresent + tolerations: + - key: "example-key" + operator: "Exists" + effect: "NoSchedule" From fd517017077bb29293440ce1c8359b9731def254 Mon Sep 17 00:00:00 2001 From: "yiting.jiang" Date: Wed, 13 Oct 2021 18:18:45 +0800 Subject: [PATCH 234/249] [zh] Update Dynamic Admission Control session:update requests for all status subresources --- .../extensible-admission-controllers.md | 51 ++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/content/zh/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/zh/docs/reference/access-authn-authz/extensible-admission-controllers.md index f41d34c8ee..3853dbfaee 100644 --- a/content/zh/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/zh/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -103,12 +103,15 @@ In the following, we describe how to quickly experiment with admission webhooks. -请参阅 Kubernetes e2e 测试中的 [admission webhook 服务器](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/main.go) 的实现。webhook 处理由 apiserver 发送的 `AdmissionReview` 请求，并且将其决定作为 `AdmissionReview` 对象以相同版本发送回去。 +请参阅 Kubernetes e2e 测试中的 +[admission webhook 服务器](https://github.com/kubernetes/kubernetes/blob/release-1.21/test/images/agnhost/webhook/main.go) +的实现。webhook 处理由 apiserver 发送的 `AdmissionReview` 请求，并且将其决定 +作为 `AdmissionReview` 对象以相同版本发送回去。 -示例准入 Webhook 服务器置 `ClientAuth` 字段为[空](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48)，默认为 `NoClientCert` 。这意味着 webhook 服务器不会验证客户端的身份，认为其是 apiservers。 -如果你需要双向 TLS 或其他方式来验证客户端，请参阅如何[对 apiservers 进行身份认证](#authenticate-apiservers)。 +示例准入 Webhook 服务器置 `ClientAuth` 字段为 +[空](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48)， +默认为 `NoClientCert` 。这意味着 webhook 服务器不会验证客户端的身份，认为其是 apiservers。 +如果你需要双向 TLS 或其他方式来验证客户端，请参阅 +如何[对 apiservers 进行身份认证](#authenticate-apiservers)。 -e2e 测试中的 webhook 服务器通过 [deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps) 部署在 Kubernetes 集群中。该测试还将创建一个 [service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core) 作为 webhook 服务器的前端。参见[相关代码](https://github.com/kubernetes/kubernetes/blob/v1.15.0/test/e2e/apimachinery/webhook.go#L301)。 +e2e 测试中的 webhook 服务器通过 +[deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps) +部署在 Kubernetes 集群中。该测试还将创建一个 +[service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core) +作为 webhook 服务器的前端。参见 +[相关代码](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/e2e/apimachinery/webhook.go#L748)。 -你可以通过 [ValidatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#validatingwebhookconfiguration-v1-admissionregistration-k8s-io) 或者 [MutatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#mutatingwebhookconfiguration-v1-admissionregistration-k8s-io) 动态配置哪些资源要被哪些准入 Webhook 处理。 +你可以通过 +[ValidatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#validatingwebhookconfiguration-v1-admissionregistration-k8s-io) +或者 +[MutatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#mutatingwebhookconfiguration-v1-admissionregistration-k8s-io) 动态配置哪些资源要被哪些准入 Webhook 处理。 scope 字段指定是仅集群范围的资源（Cluster）还是名字空间范围的资源资源（Namespaced）将与此规则匹配。`*` 表示没有范围限制。 @@ -338,12 +352,13 @@ plugins: -有关 `AdmissionConfiguration` 的更多信息，请参见 [AdmissionConfiguration schema](https://github.com/kubernetes/kubernetes/blob/v1.17.0/staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1/types.go#L27)。 +有关 `AdmissionConfiguration` 的更多信息，请参见 +[AdmissionConfiguration (v1) reference](/docs/reference/config-api/apiserver-webhookadmission.v1/)。 有关每个配置字段的详细信息，请参见 [webhook 配置](#webhook-配置)部分。 * 在 kubeConfig 文件中，提供证书凭据： @@ -401,7 +416,7 @@ Of course you need to set up the webhook server to handle these authentications. 请注意，将 `localhost` 或 `127.0.0.1` 用作 `host` 是有风险的， 除非你非常小心地在所有运行 apiserver 的、可能需要对此 webhook -进行调用的主机上运行。这样的安装可能不具有可移植性，即很难在新集群中启用。 +进行调用的主机上运行。这样的安装方式可能不具有可移植性，即很难在新集群中启用。 + +{{}} + + +你可以通过使用 Finalizers 提醒{{}} +在删除目标资源前执行特定的清理任务， +来控制资源的{{}}。 + + +Finalizers 通常不指定要执行的代码。 +相反，它们通常是特定资源上的键的列表，类似于注解。 +Kubernetes 自动指定了一些 Finalizers，但你也可以指定你自己的。 + + +## Finalizers 如何工作 {#how-finalizers-work} + +当你使用清单文件创建资源时，你可以在 `metadata.finalizers` 字段指定 Finalizers。 +当你试图删除该资源时，管理该资源的控制器会注意到 `finalizers` 字段中的值， +并进行以下操作： + + * 修改对象，将你开始执行删除的时间添加到 `metadata.deletionTimestamp` 字段。 + * 将该对象标记为只读，直到其 `metadata.finalizers` 字段为空。 + + +然后，控制器试图满足资源的 Finalizers 的条件。 +每当一个 Finalizer 的条件被满足时，控制器就会从资源的 `finalizers` 字段中删除该键。 +当该字段为空时，垃圾收集继续进行。 +你也可以使用 Finalizers 来阻止删除未被管理的资源。 + + +一个常见的 Finalizer 的例子是 `kubernetes.io/pv-protection`， +它用来防止意外删除 `PersistentVolume` 对象。 +当一个 `PersistentVolume` 对象被 Pod 使用时， +Kubernetes 会添加 `pv-protection` Finalizer。 +如果你试图删除 `PersistentVolume`，它将进入 `Terminating` 状态， +但是控制器因为该 Finalizer 存在而无法删除该资源。 +当 Pod 停止使用 `PersistentVolume` 时， +Kubernetes 清除 `pv-protection` Finalizer，控制器就会删除该卷。 + + +## 属主引用、标签和 Finalizers {#owners-labels-finalizers} + +与{{}}类似， +[属主引用](/zh/concepts/overview/working-with-objects/owners-dependents/) +描述了 Kubernetes 中对象之间的关系，但它们作用不同。 +当一个{{}} +管理类似于 Pod 的对象时，它使用标签来跟踪相关对象组的变化。 +例如，当 {{}} 创建一个或多个 Pod 时， +Job 控制器会给这些 Pod 应用上标签，并跟踪集群中的具有相同标签的 Pod 的变化。 + + +Job 控制器还为这些 Pod 添加了*属主引用*，指向创建 Pod 的 Job。 +如果你在这些 Pod 运行的时候删除了 Job， +Kubernetes 会使用属主引用（而不是标签）来确定集群中哪些 Pod 需要清理。 + +当 Kubernetes 识别到要删除的资源上的属主引用时，它也会处理 Finalizers。 + +在某些情况下，Finalizers 会阻止依赖对象的删除， +这可能导致目标属主对象，保持在只读状态的时间比预期的长，且没有被完全删除。 +在这些情况下，你应该检查目标属主和附属对象上的 Finalizers 和属主引用，来排查原因。 + +{{}} + +在对象卡在删除状态的情况下，尽量避免手动移除 Finalizers，以允许继续删除操作。 +Finalizers 通常因为特殊原因被添加到资源上，所以强行删除它们会导致集群出现问题。 +{{}} + + +## {{% heading "whatsnext" %}} + + +* 阅读 Kubernetes 博客的[使用 Finalizers 控制删除](/blog/2021/05/14/using-finalizers-to-control-deletion/)。 diff --git a/content/zh/docs/reference/glossary/finalizer.md b/content/zh/docs/reference/glossary/finalizer.md new file mode 100644 index 0000000000..9d3c48f4cf --- /dev/null +++ b/content/zh/docs/reference/glossary/finalizer.md @@ -0,0 +1,64 @@ +--- +title: Finalizer +id: finalizer +date: 2021-07-07 +full_link: /zh/docs/concepts/overview/working-with-objects/finalizers/ +short_description: > + 一个带有命名空间的键，告诉 Kubernetes 等到特定的条件被满足后， + 再完全删除被标记为删除的资源。 +aka: +tags: +- fundamental +- operation +--- + + + + + +Finalizer 是带有命名空间的键，告诉 Kubernetes 等到特定的条件被满足后， +再完全删除被标记为删除的资源。 +Finalizer 提醒{{}}清理被删除的对象拥有的资源。 + + + +当你告诉 Kubernetes 删除一个指定了 Finalizer 的对象时， +Kubernetes API 会将该对象标记为删除，使其进入只读状态。 +此时控制平面或其他组件会采取 Finalizer 所定义的行动， +而目标对象仍然处于终止中（Terminating）的状态。 +这些行动完成后，控制器会删除目标对象相关的 Finalizer。 +当 `metadata.finalizers` 字段为空时，Kubernetes 认为删除已完成。 + + +你可以使用 Finalizer 控制资源的{{}}。 +例如，你可以定义一个 Finalizer，在删除目标资源前清理相关资源或基础设施。 \ No newline at end of file From 046e3a16430e74f5c6dc62f497ff51614dec8b43 Mon Sep 17 00:00:00 2001 From: Min Wang <6271561+mallow111@users.noreply.github.com> Date: Sun, 24 Oct 2021 12:25:04 -0700 Subject: [PATCH 237/249] Update deployment.md --- content/en/docs/concepts/workloads/controllers/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index 5338c52b9a..6b2acc401e 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -164,7 +164,7 @@ Follow the steps given below to update your Deployment: 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image. ```shell - kubectl deployment.apps/nginx-deployment set image deployment/nginx-deployment nginx=nginx:1.16.1 + kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 ``` or use the following command: From 19e2e57c2fffdb005001b9076b8e83c08a595066 Mon Sep 17 00:00:00 2001 From: Jin Li Date: Fri, 22 Oct 2021 07:33:05 +0000 Subject: [PATCH 238/249] [zh]translate en/docs/reference/ports-and-protocols.md. amend --- .../zh/docs/reference/ports-and-protocols.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 content/zh/docs/reference/ports-and-protocols.md diff --git a/content/zh/docs/reference/ports-and-protocols.md b/content/zh/docs/reference/ports-and-protocols.md new file mode 100644 index 0000000000..4f496a7b3a --- /dev/null +++ b/content/zh/docs/reference/ports-and-protocols.md @@ -0,0 +1,77 @@ +--- +title: 端口和协议 +content_type: reference +weight: 50 +--- + + + +当你在一个有严格网络边界的环境里运行 Kubernetes，例如拥有物理网络防火墙或者拥有公有云中虚拟网络的自有数据中心，了解 Kubernetes 组件使用了哪些端口和协议是非常有用的。 + + +## 控制面 {#control-plane} + +| 协议 | 方向 | 端口范围 | 目的 | 使用者 | +|----------|-----------|------------|-------------------------|---------------------------| +| TCP | 入站 | 6443 | Kubernetes API server | 所有 | +| TCP | 入站 | 2379-2380 | etcd server client API | kube-apiserver, etcd | +| TCP | 入站 | 10250 | Kubelet API | 自身, 控制面 | +| TCP | 入站 | 10259 | kube-scheduler | 自身 | +| TCP | 入站 | 10257 | kube-controller-manager | 自身 | + +尽管 etcd 的端口也列举在控制面的部分，但你也可以在外部自己托管 etcd 集群或者自定义端口。 + + +## 工作节点 {#node} + +| 协议 | 方向 | 端口范围 | 目的 | 使用者 | +|----------|-----------|-------------|-----------------------|-------------------------| +| TCP | 入站 | 10250 | Kubelet API | 自身, 控制面 | +| TCP | 入站 | 30000-32767 | NodePort Services† | 所有 | + +† [NodePort Services](/zh/docs/concepts/services-networking/service/)的默认端口范围。 + +所有默认端口都可以重新配置。当使用自定义的端口时，你需要打开这些端口来代替这里提到的默认端口。 + +一个常见的例子是 API 服务器的端口有时会配置为443。或者你也可以使用默认端口，把 API 服务器放到一个监听443 端口的负载均衡器后面，并且路由所有请求到 API 服务器的默认端口。 From c771e3f2dfb9f57b28176c740a5f1f6e27dbfe1c Mon Sep 17 00:00:00 2001 From: gwnara Date: Mon, 25 Oct 2021 16:16:12 +0800 Subject: [PATCH 239/249] [zh]fix error word --- content/zh/docs/concepts/workloads/controllers/job.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/concepts/workloads/controllers/job.md b/content/zh/docs/concepts/workloads/controllers/job.md index 3487ca839a..2c482c140c 100644 --- a/content/zh/docs/concepts/workloads/controllers/job.md +++ b/content/zh/docs/concepts/workloads/controllers/job.md @@ -854,7 +854,7 @@ directly a result of toggling the `.spec.suspend` field. In the time between these two events, we see that no Pods were created, but Pod creation restarted as soon as the Job was resumed. --> -最后四个四件，特别是 "Suspended" 和 "Resumed" 事件，都是因为 `.spec.suspend` +最后四个事件，特别是 "Suspended" 和 "Resumed" 事件，都是因为 `.spec.suspend` 字段值被改来改去造成的。在这两个事件之间，我们看到没有 Pod 被创建，不过当 Job 被恢复执行时，Pod 创建操作立即被重启执行。 From 95d9f0c19faba795927494e30079012f7a9039ad Mon Sep 17 00:00:00 2001 From: RinkiyaKeDad Date: Mon, 25 Oct 2021 15:36:11 +0530 Subject: [PATCH 240/249] fixing whitespace for pod topology spread constraints page Signed-off-by: RinkiyaKeDad --- .../workloads/pods/pod-topology-spread-constraints.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md index 719fcee457..43f537c9a5 100644 --- a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md +++ b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md @@ -234,7 +234,7 @@ To overcome this situation, you can either increase the `maxSkew` or modify one The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined. - Suppose you have a 5-node cluster ranging from zoneA to zoneC: +Suppose you have a 5-node cluster ranging from zoneA to zoneC: {{}} graph BT @@ -268,7 +268,7 @@ The scheduler will skip the non-matching nodes from the skew calculations if the class zoneC cluster; {{< /mermaid >}} - and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. +and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} From a24fa75c7b2e02843f964b57ad079d8274c39f4e Mon Sep 17 00:00:00 2001 From: Deepak Sharma Date: Sun, 24 Oct 2021 16:27:41 +0530 Subject: [PATCH 241/249] fix broken blog link --- .../concepts/workloads/controllers/replicationcontroller.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md index 364df7dc72..d07ecee16c 100644 --- a/content/en/docs/concepts/workloads/controllers/replicationcontroller.md +++ b/content/en/docs/concepts/workloads/controllers/replicationcontroller.md @@ -248,7 +248,7 @@ The ReplicationController ensures that the desired number of pods matches its la The ReplicationController is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://issue.k8s.io/492)), which would change its `replicas` field. We will not add scheduling policies (for example, [spreading](https://issue.k8s.io/367#issuecomment-48428019)) to the ReplicationController. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsewhere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://issue.k8s.io/170)). -The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc. +The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://netflixtechblog.com/asgard-web-based-cloud-management-and-deployment-2c9fc4e4d3a1) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc. ## API Object From 8071fed3aa71d9c2fa2c1c14807a97982dd44544 Mon Sep 17 00:00:00 2001 From: Mikhail Sidorov <7780462+mikhailsidorov@users.noreply.github.com> Date: Mon, 25 Oct 2021 18:52:21 +0300 Subject: [PATCH 242/249] Fix missed word in russian translation --- .../docs/tutorials/kubernetes-basics/expose/expose-intro.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ru/docs/tutorials/kubernetes-basics/expose/expose-intro.html b/content/ru/docs/tutorials/kubernetes-basics/expose/expose-intro.html index 3ef66070ff..07cb3cd2df 100644 --- a/content/ru/docs/tutorials/kubernetes-basics/expose/expose-intro.html +++ b/content/ru/docs/tutorials/kubernetes-basics/expose/expose-intro.html @@ -31,7 +31,7 @@ weight: 10

    Сервис в Kubernetes — это абстрактный объект, который определяет логический набор подов и политику доступа к ним. Сервисы создают слабую связь между подами, которые от них зависят. Сервис создаётся в формате YAML (рекомендуемый формат) или JSON, как и все остальные объекты в Kubernetes. Как правило, набор подов для сервиса определяется LabelSelector (ниже описано, в каких случаях понадобиться сервис без указания selector в спецификации).

    -

    Хотя у каждого пода есть уникальный IP-адрес, эти IP-адреса не доступны за пределами кластера без использования сервиса. Сервисы позволяют приложениям принимать трафик. Сервисы могут по-разному открыты, в зависимости от указанного поля type в ServiceSpec:

    +

    Хотя у каждого пода есть уникальный IP-адрес, эти IP-адреса не доступны за пределами кластера без использования сервиса. Сервисы позволяют приложениям принимать трафик. Сервисы могут быть по-разному открыты, в зависимости от указанного поля type в ServiceSpec:

    • ClusterIP (по умолчанию) - открывает доступ к сервису по внутреннему IP-адресу в кластере. Этот тип делает сервис доступным только внутри кластера.
    • NodePort - открывает сервис на одном и том же порту каждого выбранного узла в кластере с помощью NAT. Делает сервис доступным вне кластера, используя <NodeIP>:<NodePort>. Является надмножеством ClusterIP.
      • From b992026a37e0234b508b078dca9d95ee4e40f793 Mon Sep 17 00:00:00 2001 From: Adithya Krishna Date: Mon, 25 Oct 2021 23:12:32 +0530 Subject: [PATCH 243/249] Made Requested Changes Signed-off-by: Adithya Krishna --- README-hi.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README-hi.md b/README-hi.md index 4dd156ec27..da5141af1d 100644 --- a/README-hi.md +++ b/README-hi.md @@ -25,9 +25,9 @@ आप हिंदी स्थानीयकरण के मैन्टेनरों तक पहुँच सकते हैं: -* Yashu Mittal ([Twitter](https://twitter.com/mittalyashu77), [GitHub](https://github.com/mittalyashu)) -* Divya Mohan ([Slack](https://kubernetes.slack.com/archives/D027R7BE804), [Twitter](https://twitter.com/Divya_Mohan02), [GitHub](https://github.com/divya-mohan0209)) * Anubhav Vardhan ([Slack](https://kubernetes.slack.com/archives/D0261C0A3R8), [Twitter](https://twitter.com/anubha_v_ardhan), [GitHub](https://github.com/anubha-v-ardhan)) +* Divya Mohan ([Slack](https://kubernetes.slack.com/archives/D027R7BE804), [Twitter](https://twitter.com/Divya_Mohan02), [GitHub](https://github.com/divya-mohan0209)) +* Yashu Mittal ([Twitter](https://twitter.com/mittalyashu77), [GitHub](https://github.com/mittalyashu)) * [Slack channel](https://kubernetes.slack.com/messages/kubernetes-docs-hi) From 7ebe37aa3a0bb02ac0cd9824a259dc76e6e253ca Mon Sep 17 00:00:00 2001 From: Min Wang <6271561+mallow111@users.noreply.github.com> Date: Mon, 25 Oct 2021 12:24:22 -0700 Subject: [PATCH 244/249] Update deployment.md --- content/en/docs/concepts/workloads/controllers/deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/controllers/deployment.md b/content/en/docs/concepts/workloads/controllers/deployment.md index 6b2acc401e..fe997dc2c9 100644 --- a/content/en/docs/concepts/workloads/controllers/deployment.md +++ b/content/en/docs/concepts/workloads/controllers/deployment.md @@ -164,7 +164,7 @@ Follow the steps given below to update your Deployment: 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image. ```shell - kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 + kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 ``` or use the following command: From 0162445d1e989829eb5e40a035a40a9fedfac502 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 26 Oct 2021 00:09:15 +0300 Subject: [PATCH 245/249] kubeadm: use git.k8s.io to link to kubeadm repo With the kubeadm repository changing branch from master -> main, use the "branchless" URL: git.k8s.io/kubeadm when linking to the HA guide. --- .../production-environment/tools/kubeadm/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md b/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md index 5206529f5c..ee4b7a7072 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md @@ -77,7 +77,7 @@ option. Your cluster requirements may need a different configuration. - Make sure the address of the load balancer always matches the address of kubeadm's `ControlPlaneEndpoint`. - - Read the [Options for Software Load Balancing](https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing) + - Read the [Options for Software Load Balancing](https://git.k8s.io/kubeadm/docs/ha-considerations.md#options-for-software-load-balancing) guide for more details. 1. Add the first control plane nodes to the load balancer and test the From 7150f42461a1bf57adcc22fa07c4d0728b9b1b30 Mon Sep 17 00:00:00 2001 From: Arhell Date: Tue, 26 Oct 2021 01:21:09 +0300 Subject: [PATCH 246/249] [zh] Add "KubeOps" operator SDK to third-party list --- content/zh/docs/concepts/extend-kubernetes/operator.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/zh/docs/concepts/extend-kubernetes/operator.md b/content/zh/docs/concepts/extend-kubernetes/operator.md index 9ccbc16c1d..4001e14f17 100644 --- a/content/zh/docs/concepts/extend-kubernetes/operator.md +++ b/content/zh/docs/concepts/extend-kubernetes/operator.md @@ -214,6 +214,7 @@ Operator. * [Charmed Operator Framework](https://juju.is/) * [kubebuilder](https://book.kubebuilder.io/) +* [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (dotnet operator SDK) * [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator) * [Metacontroller](https://metacontroller.github.io/metacontroller/intro.html) along with WebHooks that you implement yourself @@ -226,6 +227,7 @@ you implement yourself * [Charmed Operator Framework](https://juju.is/) * [kubebuilder](https://book.kubebuilder.io/) +* [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (dotnet operator SDK) * [KUDO](https://kudo.dev/) (Kubernetes 通用声明式 Operator) * [Metacontroller](https://metacontroller.github.io/metacontroller/intro.html)，可与 Webhooks 结合使用，以实现自己的功能。 * [Operator Framework](https://operatorframework.io) From e6ba790d73869ea2cda7b7bce1a3eae796b2c217 Mon Sep 17 00:00:00 2001 From: "Charles Emmanuel S. Ndiaye" Date: Tue, 26 Oct 2021 00:30:04 +0000 Subject: [PATCH 247/249] Suggest setup/release/notes/ translation in french --- content/fr/releases/notes.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 content/fr/releases/notes.md diff --git a/content/fr/releases/notes.md b/content/fr/releases/notes.md new file mode 100644 index 0000000000..803925fc5c --- /dev/null +++ b/content/fr/releases/notes.md @@ -0,0 +1,13 @@ +--- +linktitle: Notes de version +title: Notes +type: docs +description: > + Notes de version de Kubernetes. +sitemap: + priority: 0.5 +--- + +Les notes de version peuvent être trouvées en lisant le [Changelog](https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG) qui correspond à votre version de Kubernetes. Consultez le journal des modifications pour {{< skew lastVersion >}} sur [GitHub](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-{{< skew lastVersion >}}.md). + +Alternativement, les notes de version peuvent être recherchées et filtrées en ligne sur : [relnotes.k8s.io](https://relnotes.k8s.io). Consultez les notes de version filtrées pour {{< skew lastVersion >}} sur [relnotes.k8s.io](https://relnotes.k8s.io/?releaseVersions={{< skew lastVersion >}}.0). From 8fbccfcd8f201dbfc4028573b407295efe163f81 Mon Sep 17 00:00:00 2001 From: Shubham Kuchhal Date: Tue, 26 Oct 2021 13:51:38 +0530 Subject: [PATCH 248/249] Improvement: Correct the "empty" link in Dynamic Admission Control. --- .../access-authn-authz/extensible-admission-controllers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index 30841587b3..8d0f9694ce 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -67,7 +67,7 @@ See the [webhook request](#request) section for details on the data sent to webh See the [webhook response](#response) section for the data expected from webhooks. The example admission webhook server leaves the `ClientAuth` field -[empty](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48), +[empty](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/images/agnhost/webhook/config.go#L38-L39), which defaults to `NoClientCert`. This means that the webhook server does not authenticate the identity of the clients, supposedly apiservers. If you need mutual TLS or other ways to authenticate the clients, see From 10aa724db27f6d64b96100bbeb14a0dbad1d4896 Mon Sep 17 00:00:00 2001 From: lizzzcai Date: Tue, 26 Oct 2021 17:06:39 +0800 Subject: [PATCH 249/249] [zh] update docs for debug running pod --- .../debug-running-pod.md | 48 +++++++++++++++---- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/content/zh/docs/tasks/debug-application-cluster/debug-running-pod.md b/content/zh/docs/tasks/debug-application-cluster/debug-running-pod.md index 0eb101d4c3..976d84eb80 100644 --- a/content/zh/docs/tasks/debug-application-cluster/debug-running-pod.md +++ b/content/zh/docs/tasks/debug-application-cluster/debug-running-pod.md @@ -493,15 +493,43 @@ kubectl delete pod myapp myapp-debug -## 在节点上通过 shell 来调试 {#node-shell-session} +## 在节点上通过 shell 来进行调试 {#node-shell-session} -如果这些方法都不起作用，你可以找到运行 Pod 的主机并通过 SSH 进入该主机， -但是如果使用 Kubernetes API 中的工具，则通常不需要这样做。 -因此，如果你发现自己需要使用 ssh 进入主机，请在GitHub 上提交功能请求， -以描述你的用例以及这些工具不足的原因。 +如果这些方法都不起作用，你可以找到运行 Pod 的节点，然后在节点上部署一个运行在宿主名字空间的特权 Pod。 + +你可以通过`kubectl debug` 在节点上创建一个交互式 shell： + +```shell +kubectl debug node/mynode -it --image=ubuntu +``` + +``` +Creating debugging pod node-debugger-mynode-pdx84 with container debugger on node mynode. +If you don't see a command prompt, try pressing enter. +root@ek8s:/# +``` + + +当在节点上创建调试会话，注意以下要点： +* `kubectl debug` 基于节点的名字自动生成新的 Pod 的名字。 +* 新的调试容器运行在宿主命名空间里（IPC, 网络 还有PID命名空间）。 +* 节点的根文件系统会被挂载在 `/host`。 + +当你完成节点调试时，不要忘记清理调试 Pod： + +```shell +kubectl delete pod node-debugger-mynode-pdx84 +``` \ No newline at end of file