From ede7013dc777740069a4b6e9397b0dfe2c4c6365 Mon Sep 17 00:00:00 2001 From: Dohyun Jung Date: Mon, 14 Feb 2022 19:24:54 +0900 Subject: [PATCH 001/119] Update namespaces.md Delete content that doesn't exist in the English version. --- .../ko/docs/concepts/overview/working-with-objects/namespaces.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/ko/docs/concepts/overview/working-with-objects/namespaces.md b/content/ko/docs/concepts/overview/working-with-objects/namespaces.md index 03597eee50..3a74152a07 100644 --- a/content/ko/docs/concepts/overview/working-with-objects/namespaces.md +++ b/content/ko/docs/concepts/overview/working-with-objects/namespaces.md @@ -61,7 +61,6 @@ kube-system Active 1d * `default` 다른 네임스페이스가 없는 오브젝트를 위한 기본 네임스페이스 * `kube-system` 쿠버네티스 시스템에서 생성한 오브젝트를 위한 네임스페이스 * `kube-public` 이 네임스페이스는 자동으로 생성되며 모든 사용자(인증되지 않은 사용자 포함)가 읽기 권한으로 접근할 수 있다. 이 네임스페이스는 주로 전체 클러스터 중에 공개적으로 드러나서 읽을 수 있는 리소스를 위해 예약되어 있다. 이 네임스페이스의 공개적인 성격은 단지 관례이지 요구 사항은 아니다. - * `kube-node-lease` 클러스터가 스케일링될 때 노드 하트비트의 성능을 향상시키는 각 노드와 관련된 리스(lease) 오브젝트에 대한 네임스페이스 * `kube-node-lease` 이 네임스페이스는 각 노드와 연관된 [리스](/docs/reference/kubernetes-api/cluster-resources/lease-v1/) 오브젝트를 갖는다. 노드 리스는 kubelet이 [하트비트](/ko/docs/concepts/architecture/nodes/#하트비트)를 보내서 컨트롤 플레인이 노드의 장애를 탐지할 수 있게 한다. From ef297288fa7dbf9ac0b3a0d1efda28116bf2fdc1 Mon Sep 17 00:00:00 2001 From: Amit Sharma Date: Sat, 12 Mar 2022 14:13:06 +0530 Subject: [PATCH 002/119] Updated configure-liveness-readiness-startup-probes.md We don't need to pass -r flag with rm command to delete a file. --- .../configure-liveness-readiness-startup-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index f0fefe975a..6762acad34 100644 --- a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -59,7 +59,7 @@ and restarts it. When the container starts, it executes this command: ```shell -/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600" +/bin/sh -c "touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600" ``` For the first 30 seconds of the container's life, there is a `/tmp/healthy` file. From a60e9203e053fd69fd90054ec8a431b98066f056 Mon Sep 17 00:00:00 2001 From: Amit Sharma Date: Sat, 12 Mar 2022 14:42:48 +0530 Subject: [PATCH 003/119] Update exec-liveness.yaml --- content/en/examples/pods/probe/exec-liveness.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/examples/pods/probe/exec-liveness.yaml b/content/en/examples/pods/probe/exec-liveness.yaml index 07bf75f85c..6a9c9b3213 100644 --- a/content/en/examples/pods/probe/exec-liveness.yaml +++ b/content/en/examples/pods/probe/exec-liveness.yaml @@ -11,7 +11,7 @@ spec: args: - /bin/sh - -c - - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 + - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600 livenessProbe: exec: command: From 3f0f56f0f220be91e631c422e39767efdf0a9d1e Mon Sep 17 00:00:00 2001 From: Nir Rosenthal <32959392+nirroz93@users.noreply.github.com> Date: Fri, 18 Mar 2022 00:06:10 +0200 Subject: [PATCH 004/119] Changing note about limits without requests this is true for all limits (not only CPU and memory but also ephemeral storage) https://github.com/kubernetes/kubernetes/blob/4d08582d1fa21e1f5887e73380001ac827371553/pkg/apis/core/v1/defaults.go#L159 --- .../docs/concepts/configuration/manage-resources-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/configuration/manage-resources-containers.md b/content/en/docs/concepts/configuration/manage-resources-containers.md index 18f2fc5cb5..72d47f3d3e 100644 --- a/content/en/docs/concepts/configuration/manage-resources-containers.md +++ b/content/en/docs/concepts/configuration/manage-resources-containers.md @@ -49,7 +49,7 @@ runtimes can have different ways to implement the same restrictions. {{< note >}} If a container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own -CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches +CPU limit (or any other resource limit), but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches the limit. {{< /note >}} From 76e78444ef662f6369ad270e3dce44dfc8b844ec Mon Sep 17 00:00:00 2001 From: Nir Rosenthal <32959392+nirroz93@users.noreply.github.com> Date: Sat, 19 Mar 2022 17:39:59 +0200 Subject: [PATCH 005/119] per https://github.com/kubernetes/website/pull/32326#discussion_r830493057 --- .../concepts/configuration/manage-resources-containers.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/configuration/manage-resources-containers.md b/content/en/docs/concepts/configuration/manage-resources-containers.md index 72d47f3d3e..49d86e4c4c 100644 --- a/content/en/docs/concepts/configuration/manage-resources-containers.md +++ b/content/en/docs/concepts/configuration/manage-resources-containers.md @@ -47,10 +47,9 @@ or by enforcement (the system prevents the container from ever exceeding the lim runtimes can have different ways to implement the same restrictions. {{< note >}} -If a container specifies its own memory limit, but does not specify a memory request, Kubernetes -automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own -CPU limit (or any other resource limit), but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches -the limit. +If you specify a limit for a resource, but do not specify any request, and no admission-time +mechanism has applied a default request for that resource, then Kubernetes copies the limit +you specified and uses it as the requested value for the resource. {{< /note >}} ## Resource types From 00a6dc0c10eb198a2faf5454a1a6e2427138a61f Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Wed, 13 Apr 2022 17:09:02 -0700 Subject: [PATCH 006/119] [ru] Clean up various broken links --- content/ru/docs/concepts/cluster-administration/addons.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ru/docs/concepts/cluster-administration/addons.md b/content/ru/docs/concepts/cluster-administration/addons.md index 7e93ef82b2..5c6d6446b6 100644 --- a/content/ru/docs/concepts/cluster-administration/addons.md +++ b/content/ru/docs/concepts/cluster-administration/addons.md @@ -29,7 +29,7 @@ content_type: concept * [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) - это подключаемый модуль контроллера CNI на основе OVN для обеспечения облачной цепочки сервисных функций (SFC), несколько наложенных сетей OVN, динамического создания подсети, динамического создания виртуальных сетей, сети поставщика VLAN, сети прямого поставщика и подключаемого к другим Multi Сетевые плагины, идеально подходящие для облачных рабочих нагрузок на периферии в сети с несколькими кластерами. * [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) плагин для контейнера (NCP) обеспечивающий интеграцию между VMware NSX-T и контейнерами оркестраторов, таких как Kubernetes, а так же интеграцию между NSX-T и контейнеров на основе платформы CaaS/PaaS, таких как Pivotal Container Service (PKS) и OpenShift. * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) - эта платформа SDN, которая обеспечивает сетевое взаимодействие на основе политик между Kubernetes подами и не Kubernetes окружением, с отображением и мониторингом безопасности. -* [Romana](https://romana.io) - это сетевое решение уровня 3 для сетей подов, которое также поддерживает [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Подробности установки Kubeadm доступны [здесь](https://github.com/romana/romana/tree/master/containerize). +* [Romana](https://github.com/romana/romana) - это сетевое решение уровня 3 для сетей подов, которое также поддерживает [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Подробности установки Kubeadm доступны [здесь](https://github.com/romana/romana/tree/master/containerize). * [Weave Net](https://www.weave.works/docs/net/latest/kubernetes/kube-addon/) предоставляет сеть и обеспечивает сетевую политику, будет работать на обеих сторонах сетевого раздела и не требует внешней базы данных. ## Обнаружение служб From 62e61461d9ba46f22392080b3c0c24e5b26f0894 Mon Sep 17 00:00:00 2001 From: Humberto <85humberto@gmail.com> Date: Wed, 20 Apr 2022 11:22:52 -0300 Subject: [PATCH 007/119] Update components.md --- content/pt-br/docs/concepts/overview/components.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/pt-br/docs/concepts/overview/components.md b/content/pt-br/docs/concepts/overview/components.md index b03946c4ae..892be1a922 100644 --- a/content/pt-br/docs/concepts/overview/components.md +++ b/content/pt-br/docs/concepts/overview/components.md @@ -59,7 +59,7 @@ O cloud-controller-manager executa apenas controladores que são específicos pa Se você estiver executando o Kubernetes em suas próprias instalações ou em um ambiente de aprendizagem dentro de seu próprio PC, o cluster não possui um gerenciador de controlador de nuvem. -Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (exectuar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas. +Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (executar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas. Os seguintes controladores podem ter dependências de provedor de nuvem: From 88403cfc775ef54a24bf19f692dff7f0dc909d3b Mon Sep 17 00:00:00 2001 From: "Mr. Erlison" Date: Thu, 21 Apr 2022 13:27:28 -0300 Subject: [PATCH 008/119] Add content/pt-br/docs/reference/glossary/annotation.md --- .../docs/reference/glossary/annotation.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 content/pt-br/docs/reference/glossary/annotation.md diff --git a/content/pt-br/docs/reference/glossary/annotation.md b/content/pt-br/docs/reference/glossary/annotation.md new file mode 100644 index 0000000000..ae54f3c86a --- /dev/null +++ b/content/pt-br/docs/reference/glossary/annotation.md @@ -0,0 +1,21 @@ +--- +title: Anotação +id: annotation +date: 2018-04-12 +full_link: /docs/concepts/overview/working-with-objects/annotations +short_description: > + Um par de chave-valor (key-value) é usado para anexar metadados arbitrários não identificáveis a objetos. + +aka: +tags: +- fundamental +--- + Um par de chave-valor (key-value) é usado para anexar metadados arbitrários não identificáveis a objetos. + + + +Os metadados em uma anotação podem ser pequenos ou grandes, estruturados ou não estruturados, e podem incluir caracteres não permitidos pelos {{< glossary_tooltip text="rótulos" term_id="label" >}}. Clientes como ferramentas e bibliotecas podem recuperar esses metadados. + + + + [-] \ No newline at end of file From 5dc71848c349e59e68e8f7d695e268f5a2d20d30 Mon Sep 17 00:00:00 2001 From: "Mr. Erlison" Date: Thu, 21 Apr 2022 19:34:57 -0300 Subject: [PATCH 009/119] Removed final line with this [-] --- content/pt-br/docs/reference/glossary/annotation.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/pt-br/docs/reference/glossary/annotation.md b/content/pt-br/docs/reference/glossary/annotation.md index ae54f3c86a..5e3ca07b31 100644 --- a/content/pt-br/docs/reference/glossary/annotation.md +++ b/content/pt-br/docs/reference/glossary/annotation.md @@ -16,6 +16,3 @@ tags: Os metadados em uma anotação podem ser pequenos ou grandes, estruturados ou não estruturados, e podem incluir caracteres não permitidos pelos {{< glossary_tooltip text="rótulos" term_id="label" >}}. Clientes como ferramentas e bibliotecas podem recuperar esses metadados. - - - [-] \ No newline at end of file From 7f7f53dc5378159447182ae39ad2caf8c4faf1b1 Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Sun, 24 Apr 2022 21:14:20 +0800 Subject: [PATCH 010/119] [zh] Resync crictl --- .../tasks/debug-application-cluster/crictl.md | 329 +++++++++--------- 1 file changed, 160 insertions(+), 169 deletions(-) diff --git a/content/zh/docs/tasks/debug-application-cluster/crictl.md b/content/zh/docs/tasks/debug-application-cluster/crictl.md index f8e126f281..d928d1d0c5 100644 --- a/content/zh/docs/tasks/debug-application-cluster/crictl.md +++ b/content/zh/docs/tasks/debug-application-cluster/crictl.md @@ -45,8 +45,8 @@ Kubernetes node. `crictl` and its source are hosted in the -`crictl` 默认连接到 `unix:///var/run/dockershim.sock`。 -对于其他的运行时，你可以用多种不同的方法设置端点： +你可以用以下方法之一来为 `crictl` 设置端点： -- 通过设置参数 `--runtime-endpoint` 和 `--image-endpoint` -- 通过设置环境变量 `CONTAINER_RUNTIME_ENDPOINT` 和 `IMAGE_SERVICE_ENDPOINT` -- 通过在配置文件中设置端点 `--config=/etc/crictl.yaml` +- 设置参数 `--runtime-endpoint` 和 `--image-endpoint`。 +- 设置环境变量 `CONTAINER_RUNTIME_ENDPOINT` 和 `IMAGE_SERVICE_ENDPOINT`。 +- 在配置文件 `--config=/etc/crictl.yaml` 中设置端点。 + 要设置不同的文件，可以在运行 `crictl` 时使用 `--config=PATH_TO_FILE` 标志。 要查看或编辑当前配置，请查看或编辑 `/etc/crictl.yaml` 的内容。 +例如，使用 `containerd` 容器运行时的配置会类似于这样： -```shell -cat /etc/crictl.yaml ``` -``` -runtime-endpoint: unix:///var/run/dockershim.sock -image-endpoint: unix:///var/run/dockershim.sock +runtime-endpoint: unix:///var/run/containerd/containerd.sock +image-endpoint: unix:///var/run/containerd/containerd.sock timeout: 10 debug: true ``` + +要进一步了解 `crictl`，参阅 +[`crictl` 文档](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)。 + +输出类似于： + +``` POD ID CREATED STATE NAME NAMESPACE ATTEMPT 926f1b5a1d33a About a minute ago Ready sh-84d7dcf559-4r2gq default 0 4dccb216c4adb About a minute ago Ready nginx-65899c769f-wv2gp default 0 @@ -157,7 +169,12 @@ List pods by name: crictl pods --name nginx-65899c769f-wv2gp ``` -```none + +输出类似于这样： + +``` POD ID CREATED STATE NAME NAMESPACE ATTEMPT 4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0 ``` @@ -170,6 +187,12 @@ List pods by label: ```shell crictl pods --label run=nginx ``` + + +输出类似于这样： + ```none POD ID CREATED STATE NAME NAMESPACE ATTEMPT 4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0 @@ -187,6 +210,12 @@ List all images: ```shell crictl images ``` + + +输出类似于这样： + ```none IMAGE TAG IMAGE ID SIZE busybox latest 8c811b4aec35f 1.15MB @@ -203,6 +232,12 @@ List images by repository: ```shell crictl images nginx ``` + + +输出类似于这样： + ```none IMAGE TAG IMAGE ID SIZE nginx latest cd5239a0906a6 109MB @@ -216,6 +251,12 @@ Only list image IDs: ```shell crictl images -q ``` + + +输出类似于这样： + ```none sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a sha256:e179bbfe5d238de6069f3b03fccbecc3fb4f2019af741bfff1233c4d7b2970c5 @@ -235,6 +276,12 @@ List all containers: ```shell crictl ps -a ``` + + +输出类似于这样： + ```none CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT 1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 7 minutes ago Running sh 1 @@ -251,6 +298,12 @@ List running containers: ```shell crictl ps ``` + + +输出类似于这样： + ```none CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT 1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 6 minutes ago Running sh 1 @@ -266,6 +319,12 @@ CONTAINER ID IMAGE ```shell crictl exec -i -t 1f73f2d81bf98 ls ``` + + +输出类似于这样： + ```none bin dev etc home proc root sys tmp usr var ``` @@ -282,6 +341,12 @@ Get all container logs: ```shell crictl logs 87d3992f84f74 ``` + + +输出类似于这样： + ```none 10.240.0.96 - - [06/Jun/2018:02:45:49 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" 10.240.0.96 - - [06/Jun/2018:02:45:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" @@ -296,6 +361,12 @@ Get only the latest `N` lines of logs: ```shell crictl logs --tail=1 87d3992f84f74 ``` + + +输出类似于这样： + ```none 10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" ``` @@ -312,8 +383,10 @@ deleted by the Kubelet. 用 `crictl` 运行 Pod 沙盒对容器运行时排错很有帮助。 在运行的 Kubernetes 集群中，沙盒会随机地被 kubelet 停止和删除。 -1. - 编写下面的 JSON 文件： + +1. 编写下面的 JSON 文件： ```json { @@ -329,14 +402,18 @@ deleted by the Kubelet. } ``` -2. - 使用 `crictl runp` 命令应用 JSON 文件并运行沙盒。 + +2. 使用 `crictl runp` 命令应用 JSON 文件并运行沙盒。 ```shell crictl runp pod-config.json ``` - + 返回了沙盒的 ID。 - 拉取 busybox 镜像 + +1. 拉取 busybox 镜像 - ```bash + ```shell crictl pull busybox + ``` + ```none Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 ``` -2. - 创建 Pod 和容器的配置： + +2. 创建 Pod 和容器的配置： - + **Pod 配置**： - ```yaml + + ```json { "metadata": { "name": "nginx-sandbox", @@ -378,9 +464,12 @@ deleted by the Kubelet. } ``` - + **容器配置**： - ```yaml + + ```json { "metadata": { "name": "busybox" @@ -397,22 +486,31 @@ deleted by the Kubelet. } ``` -3. - 创建容器，传递先前创建的 Pod 的 ID、容器配置文件和 Pod 配置文件。返回容器的 ID。 + returned. +--> +3. 创建容器，传递先前创建的 Pod 的 ID、容器配置文件和 Pod 配置文件。返回容器的 ID。 ```bash crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json ``` -4. - 查询所有容器并确认新创建的容器状态为 `Created`。 + +4. 查询所有容器并确认新创建的容器状态为 `Created`。 ```bash crictl ps -a ``` + + 输出类似于这样： + ```none CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT 3e025dd50a72d busybox 32 seconds ago Created busybox 0 @@ -430,7 +528,13 @@ To start a container, pass its ID to `crictl start`: ```shell crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60 ``` -```none + + +输出类似于这样： + +``` 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60 ``` @@ -442,136 +546,23 @@ Check the container has its state set to `Running`. ```shell crictl ps ``` -```none -CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT -3e025dd50a72d busybox About a minute ago Running busybox 0 + + +输出类似于这样： + ``` - +CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT +3e025dd50a72d busybox About a minute ago Running busybox 0 +``` + +## {{% heading "whatsnext" %}} -更多信息请参考 [kubernetes-sigs/cri-tools](https://github.com/kubernetes-sigs/cri-tools)。 +* [进一步了解 `crictl`](https://github.com/kubernetes-sigs/cri-tools). +* [将 `docker` CLI 命令映射到 `crictl`](/zh/docs/reference/tools/map-crictl-dockercli/). - -## Docker CLI 和 crictl 的映射 - - -以下的映射表格只适用于 Docker CLI v1.40 和 crictl v1.19.0 版本。 -请注意该表格并不详尽。例如，其中不包含 Docker CLI 的实验性命令。 - - -{{< note >}} -尽管有些命令的输出缺少了一些数据列，CRICTL 的输出格式与 Docker CLI 是类似的。 -如果你的脚本程序需要解析命令的输出，请确认检查该特定命令的输出。 -{{< /note >}} - - -### 获取调试信息 - -{{< table caption="Docker CLI 和 crictl 的映射 - 获取调试信息" >}} - -docker cli | crictl | 描述 | 不支持的功能 --- | -- | -- | -- -`attach` | `attach` | 连接到一个运行中的容器 | `--detach-keys`, `--sig-proxy` -`exec` | `exec` | 在运行中的容器里运行一个命令 | `--privileged`, `--user`, `--detach-keys` -`images` | `images` | 列举镜像 |   -`info` | `info` | 显示系统级的信息 |   -`inspect` | `inspect`, `inspecti` | 返回容器、镜像或者任务的详细信息 |   -`logs` | `logs` | 获取容器的日志 | `--details` -`ps` | `ps` | 列举容器 |   -`stats` | `stats` | 实时显示容器的资源使用统计信息 | 列：NET/BLOCK I/O, PIDs -`version` | `version` | 显示运行时（Docker、ContainerD、或者其他) 的版本信息 |   -{{< /table >}} - - -### 进行改动 - -{{< table caption="Docker CLI 和 crictl 的映射 - 进行改动" >}} - -docker cli | crictl | 描述 | 不支持的功能 --- | -- | -- | -- -`create` | `create` | 创建一个新的容器 |   -`kill` | `stop` (timeout=0) | 杀死一个或多个正在运行的容器 | `--signal` -`pull` | `pull` | 从镜像仓库拉取镜像或者代码仓库 | `--all-tags`, `--disable-content-trust` -`rm` | `rm` | 移除一个或多个容器 |   -`rmi` | `rmi` | 移除一个或多个镜像 |   -`run` | `run` | 在新容器里运行一个命令 |   -`start` | `start` | 启动一个或多个停止的容器 | `--detach-keys` -`stop` | `stop` | 停止一个或多个正运行的容器 |   -`update` | `update` | 更新一个或多个容器的配置 | CRI 不支持 `--restart`、`--blkio-weight` 以及一些其他的资源限制选项。 -{{< /table >}} - - -### 仅 crictl 支持 - -{{< table caption="Docker CLI 和 crictl 的映射 - 仅 crictl 支持" >}} - -crictl | 描述 --- | -- -`imagefsinfo` | 返回镜像的文件系统信息 -`inspectp` | 显示一个或多个 Pod 的状态 -`port-forward` | 转发本地端口到 Pod -`pods` | 列举 Pod -`runp` | 运行一个新的 Pod -`rmp` | 移除一个或多个 Pod -`stopp` | 停止一个或多个正运行的 Pod -{{< /table >}} From 190bd0b9021c8b228ef48d325f8c230ec9ba101f Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Mon, 25 Apr 2022 01:45:02 +0800 Subject: [PATCH 011/119] [zh]Remove content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md --- .../kubeadm/generated/kubeadm_alpha.md | 69 ------------------- 1 file changed, 69 deletions(-) delete mode 100644 content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md diff --git a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md deleted file mode 100644 index f6978fcb6d..0000000000 --- a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md +++ /dev/null @@ -1,69 +0,0 @@ - - - -### 概要 - - - -kubeadm 实验子命令 - - - -### 选项 - - ---- - - - - - - - - - - -
-h, --help
- -alpha 操作的帮助命令 -
- - - -### 从父命令继承的选项 - - ---- - - - - - - - - - - -
--rootfs string
- -[实验] 指向 '真实' 宿主机的根文件系统的路径。 -
- From 5fbba61bbc48579301ea76f75208906a0a0cfaab Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Mon, 25 Apr 2022 21:39:18 +0800 Subject: [PATCH 012/119] [zh] Update docs/reference/glossary/pod-security-policy.md Signed-off-by: xin.li --- .../zh/docs/reference/glossary/pod-security-policy.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/reference/glossary/pod-security-policy.md b/content/zh/docs/reference/glossary/pod-security-policy.md index 5c8475292f..6f29aa8d5a 100644 --- a/content/zh/docs/reference/glossary/pod-security-policy.md +++ b/content/zh/docs/reference/glossary/pod-security-policy.md @@ -2,7 +2,7 @@ title: Pod 安全策略 id: pod-security-policy date: 2018-04-12 -full_link: /zh/docs/concepts/policy/pod-security-policy/ +full_link: /zh/docs/concepts/security/pod-security-policy/ short_description: > 为 Pod 的创建和更新操作启用细粒度的授权。 @@ -17,7 +17,7 @@ tags: title: Pod Security Policy id: pod-security-policy date: 2018-04-12 -full_link: /docs/concepts/policy/pod-security-policy/ +full_link: /docs/concepts/security/pod-security-policy/ short_description: > Enables fine-grained authorization of pod creation and updates. @@ -43,4 +43,9 @@ A cluster-level resource that controls security sensitive aspects of the Pod spe Pod 安全策略是集群级别的资源，它控制着 Pod 规约中的安全性敏感的内容。 `PodSecurityPolicy`对象定义了一组条件以及相关字段的默认值，Pod 运行时必须满足这些条件。Pod 安全策略控制实现上体现为一个可选的准入控制器。 + +PodSecurityPolicy 自 Kubernetes v1.21 起已弃用，并将在 v1.25 中删除。 +我们建议迁移到 [Pod 安全准入](/zh/docs/concepts/security/pod-security-admission/) 或第三方准入插件。 From 36f3fedebb72f320a56bda599a1497d6412d8647 Mon Sep 17 00:00:00 2001 From: zaunist Date: Mon, 25 Apr 2022 01:25:52 +0800 Subject: [PATCH 013/119] docs: Rsyc kubeadm-upgrade.md --- .../kubeadm/kubeadm-upgrade.md | 110 ++++++++++-------- 1 file changed, 63 insertions(+), 47 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 25f10b0fed..fdad20dfc4 100644 --- a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -72,16 +72,36 @@ The upgrade workflow at high level is the following: + ### 附加信息 -- 在对 kubelet 作次版本升版时需要[腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。 - 对于控制面节点，其上可能运行着 CoreDNS Pods 或者其它非常重要的负载。 +- 下述说明了在升级过程中何时腾空每个节点。如果你正在对任何 kubelet 进行小版本升级， + 你需要先腾空待升级的节点（或多个节点）。对于控制面节点，其上可能运行着 CoreDNS Pods + 或者其它非常重要的负载。更多信息见[腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。 - 升级后，因为容器规约的哈希值已更改，所有容器都会被重新启动。 + + +- 要验证 kubelet 服务在升级后是否成功重启，可以执行 `systemctl status kubelet` + 或 `journalctl -xeu kubelet` 查看服务日志。 +- 不建议使用 `kubeadm upgrade` 的 `--config 参数和 [kubeadm 配置 API 类型](/zh/docs/reference/config-api/kubeadm-config.v1beta3) + 来重新配置集群，这样会产生意想不到的结果。请按照[重新配置 kubeadm 集群](/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure) + 中的步骤来进行。 + 一旦该命令结束，你应该会看到： - ``` + ```console [upgrade/successful] SUCCESS! Your cluster was upgraded to "v{{< skew currentVersion >}}.x". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. @@ -259,7 +283,7 @@ Same as the first control plane node but use: --> 与第一个控制面节点相同，但是使用： -``` +```shell sudo kubeadm upgrade node ``` @@ -268,7 +292,7 @@ instead of: --> 而不是： -``` +```shell sudo kubeadm upgrade apply ``` @@ -299,34 +323,33 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no ### 升级 kubelet 和 kubectl -- 升级 kubelet 和 kubectl +- 升级 kubelet 和 kubectl： {{< tabs name="k8s_install_kubelet" >}} {{% tab name="Ubuntu、Debian 或 HypriotOS" %}} - 
-  # 用最新的补丁版本替换 {{< skew currentVersion >}}.x-00 中的 x
-  apt-mark unhold kubelet kubectl && \
-  apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \
-  apt-mark hold kubelet kubectl
-  - 
-  # 从 apt-get 的 1.1 版本开始，你也可以使用下面的方法：
-  apt-get update && \
-  apt-get install -y --allow-change-held-packages kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00
-
+ ```shell + # 用最新的补丁版本替换 {{< skew currentVersion >}}.x-00 中的 x + apt-mark unhold kubelet kubectl && \ + apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \ + apt-mark hold kubelet kubectl + ``` + {{% /tab %}} {{% tab name="CentOS、RHEL 或 Fedora" %}} - - 
 
-  # 用最新的补丁版本号替换 {{< skew currentVersion >}}.x-00 中的 x
-  yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
-
+ + ```shell + # 用最新的补丁版本号替换 {{< skew currentVersion >}}.x-00 中的 x + yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes + ``` + {{% /tab %}} {{< /tabs >}} +
-`STATUS` 应显示所有节点为 `Ready` 状态，并且版本号已经被更新。 +`STATUS` 应显示所有节点为 `Ready` 状态，并且版本号已经被更新。 `kubeadm upgrade node` 在工作节点上完成以下工作： -- 从集群取回 kubeadm `ClusterConfiguration`。 +- 从集群取回 kubeadm `ClusterConfiguration`。 - 为本节点升级 kubelet 配置。 - From 5e46c9bac27425830687be7c8b12f232033390ef Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Tue, 26 Apr 2022 13:15:50 +0800 Subject: [PATCH 014/119] [zh]Update content/zh/docs/concepts/architecture/nodes.md --- .../zh/docs/concepts/architecture/nodes.md | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/content/zh/docs/concepts/architecture/nodes.md b/content/zh/docs/concepts/architecture/nodes.md index 68ffb345b1..57774d7cd1 100644 --- a/content/zh/docs/concepts/architecture/nodes.md +++ b/content/zh/docs/concepts/architecture/nodes.md @@ -570,26 +570,28 @@ controller deletes the node from its list of nodes. The third is monitoring the nodes' health. The node controller is responsible for: -- In the case that a node becomes unreachable, updating the NodeReady condition - of within the Node's `.status`. In this case the node controller sets the - NodeReady condition to `ConditionUnknown`. +- In the case that a node becomes unreachable, updating the `Ready` condition + in the Node's `.status` field. In this case the node controller sets the + `Ready` condition to `Unknown`. - If a node remains unreachable: triggering [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/) for all of the Pods on the unreachable node. By default, the node controller - waits 5 minutes between marking the node as `ConditionUnknown` and submitting + waits 5 minutes between marking the node as `Unknown` and submitting the first eviction request. -The node controller checks the state of each node every `-node-monitor-period` seconds. +By default, the node controller checks the state of each node every 5 seconds. +This period can be configured using the `--node-monitor-period` flag on the +`kube-controller-manager` component. --> 第三个是监控节点的健康状况。节点控制器负责： -- 在节点不可达的情况下，在 Node 的 `.status` 中更新 NodeReady 状况。 +- 在节点不可达的情况下，在 Node 的 `.status` 中更新 `Ready` 状况。 在这种情况下，节点控制器将 NodeReady 状况更新为 `Unknown` 。 - 如果节点仍然无法访问：对于不可达节点上的所有 Pod 触发 [API-发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)。 默认情况下，节点控制器在将节点标记为 `Unknown` 后等待 5 分钟提交第一个驱逐请求。 -节点控制器每隔 `--node-monitor-period` 秒检查每个节点的状态。 +默认情况下，节点控制器每5秒检查一次节点状态，可以使用 `kube-controller-manager` 组件上的 `--node-monitor-period` 参数来配置周期。 当一个可用区域（Availability Zone）中的节点变为不健康时，节点的驱逐行为将发生改变。 -节点控制器会同时检查可用区域中不健康（NodeReady 状况为 `Unknown` 或 `False`） +节点控制器会同时检查可用区域中不健康（`Ready` 状况为 `Unknown` 或 `False`） 的节点的百分比： ## 节点拓扑 {#node-topology} -{{< feature-state state="alpha" for_k8s_version="v1.16" >}} +{{< feature-state state="beta" for_k8s_version="v1.18" >}} 注意，默认情况下，下面描述的两个配置选项，`ShutdownGracePeriod` 和 From 32b87c471aa34fee1f3da80aa9893f9bec248d1a Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Tue, 26 Apr 2022 20:49:42 +0800 Subject: [PATCH 015/119] [zh] Add examples/tls Signed-off-by: xin.li --- .../zh/examples/tls/server-signing-config.json | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 content/zh/examples/tls/server-signing-config.json diff --git a/content/zh/examples/tls/server-signing-config.json b/content/zh/examples/tls/server-signing-config.json new file mode 100644 index 0000000000..86860d7369 --- /dev/null +++ b/content/zh/examples/tls/server-signing-config.json @@ -0,0 +1,15 @@ +{ + "signing": { + "default": { + "usages": [ + "digital signature", + "key encipherment", + "server auth" + ], + "expiry": "876000h", + "ca_constraint": { + "is_ca": false + } + } + } +} \ No newline at end of file From 38aada60dbbf8545b3e908b3dafe1ba9af900426 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Tue, 26 Apr 2022 20:54:03 +0800 Subject: [PATCH 016/119] [zh] Create examples/mongodb Signed-off-by: xin.li --- .../application/mongodb/mongo-deployment.yaml | 31 +++++++++++++++++++ .../application/mongodb/mongo-service.yaml | 14 +++++++++ 2 files changed, 45 insertions(+) create mode 100644 content/zh/examples/application/mongodb/mongo-deployment.yaml create mode 100644 content/zh/examples/application/mongodb/mongo-service.yaml diff --git a/content/zh/examples/application/mongodb/mongo-deployment.yaml b/content/zh/examples/application/mongodb/mongo-deployment.yaml new file mode 100644 index 0000000000..04908ce25b --- /dev/null +++ b/content/zh/examples/application/mongodb/mongo-deployment.yaml @@ -0,0 +1,31 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: mongo + labels: + app.kubernetes.io/name: mongo + app.kubernetes.io/component: backend +spec: + selector: + matchLabels: + app.kubernetes.io/name: mongo + app.kubernetes.io/component: backend + replicas: 1 + template: + metadata: + labels: + app.kubernetes.io/name: mongo + app.kubernetes.io/component: backend + spec: + containers: + - name: mongo + image: mongo:4.2 + args: + - --bind_ip + - 0.0.0.0 + resources: + requests: + cpu: 100m + memory: 100Mi + ports: + - containerPort: 27017 diff --git a/content/zh/examples/application/mongodb/mongo-service.yaml b/content/zh/examples/application/mongodb/mongo-service.yaml new file mode 100644 index 0000000000..b9cef607bc --- /dev/null +++ b/content/zh/examples/application/mongodb/mongo-service.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Service +metadata: + name: mongo + labels: + app.kubernetes.io/name: mongo + app.kubernetes.io/component: backend +spec: + ports: + - port: 27017 + targetPort: 27017 + selector: + app.kubernetes.io/name: mongo + app.kubernetes.io/component: backend From e0e1b3095ce233272a13c8f04b2080acf772d44e Mon Sep 17 00:00:00 2001 From: kinzhi Date: Tue, 26 Apr 2022 21:03:04 +0800 Subject: [PATCH 017/119] Update content/zh/docs/concepts/architecture/nodes.md Co-authored-by: Qiming Teng --- content/zh/docs/concepts/architecture/nodes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/zh/docs/concepts/architecture/nodes.md b/content/zh/docs/concepts/architecture/nodes.md index 57774d7cd1..1676e635d8 100644 --- a/content/zh/docs/concepts/architecture/nodes.md +++ b/content/zh/docs/concepts/architecture/nodes.md @@ -591,7 +591,8 @@ This period can be configured using the `--node-monitor-period` flag on the [API-发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)。 默认情况下，节点控制器在将节点标记为 `Unknown` 后等待 5 分钟提交第一个驱逐请求。 -默认情况下，节点控制器每5秒检查一次节点状态，可以使用 `kube-controller-manager` 组件上的 `--node-monitor-period` 参数来配置周期。 +默认情况下，节点控制器每 5 秒检查一次节点状态，可以使用 `kube-controller-manager` +组件上的 `--node-monitor-period` 参数来配置周期。 ## 使用 Mermaid 来可视化 你可以使用 [Mermaid JS](https://mermaidjs.github.io) 来进行可视化展示。 -Mermaid JS 版本在 [/layouts/partials/head.html](https://github.com/kubernetes/website/blob/master/layouts/partials/head.html) +Mermaid JS 版本在 [/layouts/partials/head.html](https://github.com/kubernetes/website/blob/main/layouts/partials/head.html) 中设置。 对于控制平面节点，执行额外的步骤： @@ -78,8 +76,6 @@ For control-plane nodes additional steps are performed: 1. 添加新的本地 etcd 成员。 -1. 将此节点添加到 kubeadm 集群的 ClusterStatus。 - @@ -123,6 +119,13 @@ For example: sudo kubeadm join --skip-phases=preflight --config=config.yaml ``` +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + + +或者，你可以使用 `JoinConfiguration` 中的 `skipPhases` 字段。 + @@ -523,11 +526,11 @@ the [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/ 命令转换。 -有关配置的字段和用法的更多信息，你可以导航到我们的 API 参考页 -并从[列表]中选择一个版本(https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#pkg-subdirectories)。 +有关配置的字段和用法的更多信息，你可以导航到我们的 +[API 参考页](/zh/docs/reference/config-api/kubeadm-config.v1beta3/)。 ## {{% heading "whatsnext" %}} From a9c422d8169ce04eaeaad618f477444674e5020a Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Wed, 27 Apr 2022 09:12:38 +0800 Subject: [PATCH 025/119] Fix dockershim alias --- content/zh/blog/_posts/2020-12-02-dockershim-faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/blog/_posts/2020-12-02-dockershim-faq.md b/content/zh/blog/_posts/2020-12-02-dockershim-faq.md index 5b8d896a96..e0c683255f 100644 --- a/content/zh/blog/_posts/2020-12-02-dockershim-faq.md +++ b/content/zh/blog/_posts/2020-12-02-dockershim-faq.md @@ -3,7 +3,7 @@ layout: blog title: "弃用 Dockershim 的常见问题" date: 2020-12-02 slug: dockershim-faq -aliases: [ '/dockershim' ] +aliases: [ '/zh/dockershim' ] --- + + + From 10e566a222a139d326440ccea639bcbaedd48104 Mon Sep 17 00:00:00 2001 From: chengleqi Date: Wed, 27 Apr 2022 12:15:30 +0800 Subject: [PATCH 028/119] Update configure-projected-volume-storage.md --- .../configure-projected-volume-storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/tasks/configure-pod-container/configure-projected-volume-storage.md b/content/zh/docs/tasks/configure-pod-container/configure-projected-volume-storage.md index d0ff476b38..66230434d9 100644 --- a/content/zh/docs/tasks/configure-pod-container/configure-projected-volume-storage.md +++ b/content/zh/docs/tasks/configure-pod-container/configure-projected-volume-storage.md @@ -59,7 +59,7 @@ Here is the configuration file for the Pod: ```shell # 创建包含用户名和密码的文件: echo -n "admin" > ./username.txt - echo -n "1f2d1e2e67df" > ./password.txt--> + echo -n "1f2d1e2e67df" > ./password.txt # 将上述文件引用到 Secret： kubectl create secret generic user --from-file=./username.txt From 446de63a6cfd2d29e5c3b2f267c1caf7cdaaab78 Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Sat, 23 Apr 2022 11:07:09 +0800 Subject: [PATCH 029/119] [zh] Update controlplane to node communication page --- .../control-plane-node-communication.md | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/content/zh/docs/concepts/architecture/control-plane-node-communication.md b/content/zh/docs/concepts/architecture/control-plane-node-communication.md index 3e2beaafd9..8cf742e272 100644 --- a/content/zh/docs/concepts/architecture/control-plane-node-communication.md +++ b/content/zh/docs/concepts/architecture/control-plane-node-communication.md @@ -30,9 +30,9 @@ One or more forms of [authorization](/docs/reference/access-authn-authz/authoriz ## 节点到控制面 Kubernetes 采用的是中心辐射型（Hub-and-Spoke）API 模式。 -所有从集群（或所运行的 Pods）发出的 API 调用都终止于 apiserver。 +所有从集群（或所运行的 Pods）发出的 API 调用都终止于 API 服务器。 其它控制面组件都没有被设计为可暴露远程服务。 -apiserver 被配置为在一个安全的 HTTPS 端口（通常为 443）上监听远程连接请求， +API 服务器被配置为在一个安全的 HTTPS 端口（通常为 443）上监听远程连接请求， 并启用一种或多种形式的客户端[身份认证](/zh/docs/reference/access-authn-authz/authentication/)机制。 一种或多种客户端[鉴权机制](/zh/docs/reference/access-authn-authz/authorization/)应该被启用， 特别是在允许使用[匿名请求](/zh/docs/reference/access-authn-authz/authentication/#anonymous-requests) @@ -41,7 +41,7 @@ apiserver 被配置为在一个安全的 HTTPS 端口（通常为 443）上监 -应该使用集群的公共根证书开通节点，这样它们就能够基于有效的客户端凭据安全地连接 apiserver。 +应该使用集群的公共根证书开通节点，这样它们就能够基于有效的客户端凭据安全地连接 API 服务器。 一种好的方法是以客户端证书的形式将客户端凭据提供给 kubelet。 请查看 [kubelet TLS 启动引导](/zh/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) 以了解如何自动提供 kubelet 客户端证书。 @@ -52,12 +52,12 @@ The `kubernetes` service (in `default` namespace) is configured with a virtual I The control plane components also communicate with the cluster apiserver over the secure port. --> -想要连接到 apiserver 的 Pod 可以使用服务账号安全地进行连接。 +想要连接到 API 服务器的 Pod 可以使用服务账号安全地进行连接。 当 Pod 被实例化时，Kubernetes 自动把公共根证书和一个有效的持有者令牌注入到 Pod 里。 `kubernetes` 服务（位于 `default` 名字空间中）配置了一个虚拟 IP 地址，用于（通过 kube-proxy）转发 -请求到 apiserver 的 HTTPS 末端。 +请求到 API 服务器的 HTTPS 末端。 -控制面组件也通过安全端口与集群的 apiserver 通信。 +控制面组件也通过安全端口与集群的 API 服务器通信。 ## 控制面到节点 -从控制面（apiserver）到节点有两种主要的通信路径。 -第一种是从 apiserver 到集群中每个节点上运行的 kubelet 进程。 -第二种是从 apiserver 通过它的代理功能连接到任何节点、Pod 或者服务。 +从控制面（API 服务器）到节点有两种主要的通信路径。 +第一种是从 API 服务器到集群中每个节点上运行的 kubelet 进程。 +第二种是从 API 服务器通过它的代理功能连接到任何节点、Pod 或者服务。 ### API 服务器到 kubelet -从 apiserver 到 kubelet 的连接用于： +从 API 服务器到 kubelet 的连接用于： * 获取 Pod 日志 * 挂接（通过 kubectl）到运行中的 Pod * 提供 kubelet 的端口转发功能。 这些连接终止于 kubelet 的 HTTPS 末端。 -默认情况下，apiserver 不检查 kubelet 的服务证书。这使得此类连接容易受到中间人攻击， +默认情况下，API 服务器不检查 kubelet 的服务证书。这使得此类连接容易受到中间人攻击， 在非受信网络或公开网络上运行也是 **不安全的**。 -为了对这个连接进行认证，使用 `--kubelet-certificate-authority` 标志给 apiserver -提供一个根证书包，用于 kubelet 的服务证书。 +为了对这个连接进行认证，使用 `--kubelet-certificate-authority` 标志给 API +服务器提供一个根证书包，用于 kubelet 的服务证书。 -如果无法实现这点，又要求避免在非受信网络或公共网络上进行连接，可在 apiserver 和 +如果无法实现这点，又要求避免在非受信网络或公共网络上进行连接，可在 API 服务器和 kubelet 之间使用 [SSH 隧道](#ssh-tunnels)。 最后，应该启用 @@ -122,9 +122,9 @@ kubelet 之间使用 [SSH 隧道](#ssh-tunnels)。 The connections from the apiserver to a node, pod, or service default to plain HTTP connections and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will not validate the certificate provided by the HTTPS endpoint nor provide client credentials so while the connection will be encrypted, it will not provide any guarantees of integrity. These connections **are not currently safe** to run over untrusted and/or public networks. --> -### apiserver 到节点、Pod 和服务 +### API 服务器到节点、Pod 和服务 -从 apiserver 到节点、Pod 或服务的连接默认为纯 HTTP 方式，因此既没有认证，也没有加密。 +从 API 服务器到节点、Pod 或服务的连接默认为纯 HTTP 方式，因此既没有认证，也没有加密。 这些连接可通过给 API URL 中的节点、Pod 或服务名称添加前缀 `https:` 来运行在安全的 HTTPS 连接上。 不过这些连接既不会验证 HTTPS 末端提供的证书，也不会提供客户端证书。 因此，虽然连接是加密的，仍无法提供任何完整性保证。 @@ -140,8 +140,8 @@ SSH tunnels are currently deprecated so you shouldn't opt to use them unless you --> ### SSH 隧道 {#ssh-tunnels} -Kubernetes 支持使用 SSH 隧道来保护从控制面到节点的通信路径。在这种配置下，apiserver -建立一个到集群中各节点的 SSH 隧道（连接到在 22 端口监听的 SSH 服务） +Kubernetes 支持使用 SSH 隧道来保护从控制面到节点的通信路径。在这种配置下，API +服务器建立一个到集群中各节点的 SSH 隧道（连接到在 22 端口监听的 SSH 服务） 并通过这个隧道传输所有到 kubelet、节点、Pod 或服务的请求。 这一隧道保证通信不会被暴露到集群节点所运行的网络之外。 From 3c54dacb3e7e755db99aa654694396c21076c919 Mon Sep 17 00:00:00 2001 From: nasa9084 Date: Sat, 23 Apr 2022 03:17:17 +0900 Subject: [PATCH 030/119] Update README-ja.md Update README-ja.md Co-authored-by: Toshiaki Inukai <82919057+t-inu@users.noreply.github.com> fix anchor --- README-ja.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/README-ja.md b/README-ja.md index 91e624c610..c4acd30d84 100644 --- a/README-ja.md +++ b/README-ja.md @@ -4,6 +4,9 @@ このリポジトリには、[KubernetesのWebサイトとドキュメント](https://kubernetes.io/)をビルドするために必要な全アセットが格納されています。貢献に興味を持っていただきありがとうございます！ +- [ドキュメントに貢献する](#contributing-to-the-docs) +- [翻訳された`README.md`一覧](#localization-readmemds) + # リポジトリの使い方 Hugo(Extended version)を使用してWebサイトをローカルで実行することも、コンテナランタイムで実行することもできます。コンテナランタイムを使用することを強くお勧めします。これにより、本番Webサイトとのデプロイメントの一貫性が得られます。 @@ -56,6 +59,43 @@ make serve これで、Hugoのサーバーが1313番ポートを使って開始します。お使いのブラウザにて http://localhost:1313 にアクセスしてください。リポジトリ内のソースファイルに変更を加えると、HugoがWebサイトの内容を更新してブラウザに反映します。 +## API reference pagesをビルドする + +`content/en/docs/reference/kubernetes-api`に配置されているAPIリファレンスページはを使ってSwagger仕様書からビルドされています。 + +新しいKubernetesリリースのためにリファレンスページをアップデートするには、次の手順を実行します: + +1. `api-ref-generator`サブモジュールをプルする: + + ```bash + git submodule update --init --recursive --depth 1 + ``` + +2. Swagger仕様書を更新する: + + ```bash + curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json + ``` + +3. 新しいリリースの変更を反映するため、`api-ref-assets/config/`で`toc.yaml`と`fields.yaml`を適用する。 + +4. 次に、ページをビルドする: + + ```bash + make api-reference + ``` + + コンテナイメージからサイトを作成・サーブする事でローカルで結果をテストすることができます: + + ```bash + make container-image + make container-serve + ``` + + APIリファレンスを見るために、ブラウザでを開いてください。 + +5. 新しいコントラクトのすべての変更が設定ファイル`toc.yaml`と`fields.yaml`に反映されたら、新しく生成されたAPIリファレンスページとともにPull Requestを作成します。 + ## トラブルシューティング ### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version @@ -107,7 +147,7 @@ sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist - [Slack](https://kubernetes.slack.com/messages/kubernetes-docs-ja) - [メーリングリスト](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) -## ドキュメントに貢献する +## ドキュメントに貢献する {#contributing-to-the-docs} GitHubの画面右上にある**Fork**ボタンをクリックすると、お使いのGitHubアカウントに紐付いた本リポジトリのコピーが作成され、このコピーのことを*フォーク*と呼びます。フォークリポジトリの中ではお好きなように変更を加えていただいて構いません。加えた変更をこのリポジトリに追加したい任意のタイミングにて、フォークリポジトリからPull Reqeustを作成してください。 @@ -124,7 +164,15 @@ Kubernetesのドキュメントへの貢献に関する詳細については以 * [ドキュメントのスタイルガイド](https://kubernetes.io/docs/contribute/style/style-guide/) * [Kubernetesドキュメントの翻訳方法](https://kubernetes.io/docs/contribute/localization/) -## 翻訳された`README.md`一覧 +### New Contributor Ambassadors + +コントリビュートする時に何か助けが必要なら、[New Contributor Ambassadors](https://kubernetes.io/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador)に聞いてみると良いでしょう。彼らはSIG Docsのapproverで、最初の数回のPull Requestを通して新しいコントリビューターを指導し助けることを責務としています。New Contributors Ambassadorsにコンタクトするには、[Kubernetes Slack](https://slack.k8s.io)が最適な場所です。現在のSIG DocsのNew Contributor Ambassadorは次の通りです: + +| 名前 | Slack | GitHub | +| -------------------------- | -------------------------- | -------------------------- | +| Arsh Sharma | @arsh | @RinkiyaKeDad | + +## 翻訳された`README.md`一覧 {#localization-readmemds} | Language | Language | |---|---| From a9a1b0d2b6d0884b8cd2e9903656f6e76bb1e566 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 27 Apr 2022 15:50:21 +0800 Subject: [PATCH 031/119] [zh] Sync booz-allen Signed-off-by: xin.li --- .../booz-allen/booz-allen-featured-logo.svg | 1 + .../booz-allen/booz-allen_featured_logo.png | Bin 0 -> 1989 bytes content/zh/case-studies/booz-allen/index.html | 80 ++++++++++++++++++ 3 files changed, 81 insertions(+) create mode 100644 content/zh/case-studies/booz-allen/booz-allen-featured-logo.svg create mode 100644 content/zh/case-studies/booz-allen/booz-allen_featured_logo.png create mode 100644 content/zh/case-studies/booz-allen/index.html diff --git a/content/zh/case-studies/booz-allen/booz-allen-featured-logo.svg b/content/zh/case-studies/booz-allen/booz-allen-featured-logo.svg new file mode 100644 index 0000000000..3ce58c68f7 --- /dev/null +++ b/content/zh/case-studies/booz-allen/booz-allen-featured-logo.svg @@ -0,0 +1 @@ +booz-allen-featured \ No newline at end of file diff --git a/content/zh/case-studies/booz-allen/booz-allen_featured_logo.png b/content/zh/case-studies/booz-allen/booz-allen_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..f9bc64ba3bd2bb30fec26ad2c1d3f5d216a32bb9 GIT binary patch literal 1989 zcmciD=RX^Y9tLpLT2VE^anL%gEe8>Mwzx)P#;QG%N>WG7C}M;*O;N=at-X5GC@NG@ zdqim@r9~-@O&rvUAVs;(%lil1`{MU|KA-31^X8ijv$fy@hy&Qz*!ZBgO&!j2;{4*d z+0XwTtQ&qFV&P`4;f}$m@JM(VlFisJ7=gSF4TPhS4oJ8kHk5)iU}NJ#LrsmHqNcv* zDTfO~h0Wpo)jcKnZoCm!#yx>#&W9EWa_k!1f8JdbEPqghaGnOgtw4ZOVs(Kq-)&+=BSOovdUTYKhLsh+st9RSoPnzOhpb5Gn z^JqVS>C!nus@(Zn?P6Dav!f0adv+SvQGwdg+1319@!Kt2`-6?~6LGH!fP|!BG?TGn zMdZ_MuLPE+r=|V9otm1;#V`5S`_R%u5Qjh?osLp8GMwv*4+a3m*WE(UgFgK{y7_Rj z(a{?j``DhIDIr|1ujl+ln;kKtf4zJ@;aUz)MV`)1yO_y4S6rD7c7#Q(D^Qi=JcQJET*b8D z(j5IeDBLy+(T{p}gXcO0PkKIT?+-lF(Ju!sPF~iv2B{@gTxRL%jQIebg=QJ5O>$5i zW5oZ(s42;~Wj+tJh7a0mS-LO&{u{Ri0G8vTHkGvGZ3cI?`pkLQMX->e_0-(ltwhh;)fwDdNClWa zSqg~=DpW;;Q;eVm57d&TDkD?_Kljnb)B+>IIQ>gI`JB^mik;e|x z2xHi;(poWz^$2zH*tk}%KF|cX$~0SRtM}B&c{%F&Q>I7pWm-jJ2GIZ1M=@Sf~&P2Ds8}_Yn#}QY8 zxYF&9;^2jr{SkebBJe>Ped+gd?!g~js z{36a9!^&$3>X}(?WhvnvWYD11^aX|c^C!3JN?mm@}mCw8wt@cMu3U= z!-5w(jVk}9Ws9ndI-+)4qO2hh=~bFK|1zoIbkEf=|9erG;0WBXakoWR%56RRy8Nt- z#c1dX?EYGdfU|6d-yc?Xqog%;A^xlzO1Kpslv$~)Lcg+?dx5F)j;wK1(vGAA9% zR2#p>4No-aX8p=nw1r^HWP~?<5Pia08c5$&kgmGwSG(J!oGnF@V429M6Rot|&FQ7- zVQjh8N>eKst@2Ax>-;i|LVoemDBgUt`O%f2JWc95ItFUnfZT zP8nIIOPNh6GF2a?8mBJU3TcR$&bvNgb4Z8@80GyyET5Tu7Dn|syuJ2TP6t8?#i+LP zsXIko$i1^b(2Q~?oMBFU0{#IfgkzcRzwZLhPNFue2xI;q`uvx7rXo*4!uEu70#E7S b3!H4hpZ3Gs$thgtF0(<+Y)xxTd>{P>_Mg9c literal 0 HcmV?d00001 diff --git a/content/zh/case-studies/booz-allen/index.html b/content/zh/case-studies/booz-allen/index.html new file mode 100644 index 0000000000..7b53dc01ae --- /dev/null +++ b/content/zh/case-studies/booz-allen/index.html @@ -0,0 +1,80 @@ +--- +title: Booz Allen Case Study +linkTitle: Booz Allen Hamilton +case_study_styles: true +cid: caseStudies +logo: booz-allen-featured-logo.svg +featured: true +weight: 2 +quote: > + Kubernetes is a great solution for us. It allows us to rapidly iterate on our clients' demands. + +new_case_study_styles: true +heading_background: /images/case-studies/booz-allen/banner4.jpg +heading_title_text: Booz Allen Hamilton +use_gradient_overlay: true +subheading: > + How Booz Allen Hamilton Is Helping Modernize the Federal Government with Kubernetes +case_study_details: + - Company: Booz Allen Hamilton + - Location: United States + - Industry: Government +--- + +

Challenge

+ +

In 2017, Booz Allen Hamilton's Strategic Innovation Group worked with the federal government to relaunch the decade-old recreation.gov website, which provides information and real-time booking for more than 100,000 campsites and facilities on federal lands across the country. The infrastructure needed to be agile, reliable, and scalable—as well as repeatable for the other federal agencies that are among Booz Allen Hamilton's customers.

+ +

Solution

+ +

"The only way that we thought we could be successful with this problem across all the different agencies is to create a microservice architecture and containers, so that we could be very dynamic and very agile to any given agency for whatever requirements that they may have," says Booz Allen Hamilton Senior Lead Technologist Martin Folkoff. To meet those requirements, Folkoff's team looked to Kubernetes for orchestration.

+ +

Impact

+ +

With the recreation.gov Kubernetes platform, changes can be implemented in about 30 minutes, compared to the multiple hours or even days legacy government applications require to review the code, get approval, and deploy the fix. Recreation.gov deploys to production on average 10 times a day. With monitoring, security, and logging built in, developers can create and publish new services to production within a week. Additionally, Folkoff says, "supporting the large, existing monoliths in the government is extremely expensive," and migrating into a more modern platform has resulted in perhaps 50% cost savings.

+ +{{< case-studies/quote + image="/images/case-studies/booz-allen/banner2.jpg" + author="JOSH BOYD, CHIEF TECHNOLOGIST AT BOOZ ALLEN HAMILTON" +>}} +"When there's a regulatory change in an agency, or a legislative change in Congress, or an executive order that changes the way you do business, how do I deploy that and get that out to the people who need it rapidly? At the end of the day, that's the problem we're trying to help the government solve with tools like Kubernetes." +{{< /case-studies/quote >}} +​ +​{{< case-studies/lead >}} +The White House launched an IT modernization effort in 2017, and in addition to improving cybersecurity and shifting to the public cloud and a consolidated IT model, "the federal government is looking to provide a better experience to citizens in every way that we interact with the government through every channel," says Booz Allen Hamilton Senior Lead Technologist Martin Folkoff. +{{< /case-studies/lead >}} + +

To that end, Folkoff's Strategic Innovation Group worked with the federal government last year to relaunch the decade-old recreation.gov website, which provides information and real-time booking for more than 100,000 campsites and facilities on federal lands across the country.

+ +

The infrastructure needed to be agile, reliable, and scalable—as well as repeatable for the other federal agencies that are among Booz Allen Hamilton's customers. "The only way that we thought we could be successful with this problem across all the different agencies is to create a microservice architecture, so that we could be very dynamic and very agile to any given agency for whatever requirements that they may have," says Folkoff.

+ +{{< case-studies/quote author="MARTIN FOLKOFF, SENIOR LEAD TECHNOLOGIST AT BOOZ ALLEN HAMILTON" >}} +"With CNCF, there's a lot of focus on scale, and so there's a lot of comfort knowing that as the project grows, we're going to be comfortable using that tool set." +{{< /case-studies/quote >}} + +

Booz Allen Hamilton, which has provided consulting services to the federal government for more than a century, introduced microservices, Docker containers, and AWS to its federal agency clients about five years ago. The next logical step was Kubernetes for orchestration. "Knowing that we had to be really agile and really reliable and scalable, we felt that the only technology that we know that can enable those kinds of things are the ones the CNCF provides," Folkoff says. "One of the things that is always important for the government is to make sure that the things that we build really endure. Using technology that is supported across multiple different companies and has strong governance gives people a lot of confidence."

+ +

Kubernetes was also aligned with the government's open source and IT modernization initiatives, so there has been an uptick in its usage at federal agencies over the past two years. "Now that Kubernetes is becoming offered as a service by the cloud providers like AWS and Microsoft, we're starting to see even more interest," says Chief Technologist Josh Boyd. Adds Folkoff: "With CNCF, there's a lot of focus on scale, and so there's a lot of comfort knowing that as the project grows, we're going to be comfortable using that tool set."

+ +

The greenfield recreation.gov project allowed the team to build a new Kubernetes-enabled site running on AWS, and the migration lasted only a week, when the old site didn't take bookings. "For the actual transition, we just swapped a DNS server, and it only took about 35 seconds between the old site being down and our new site being up and available," Folkoff adds.

+​ +{{< case-studies/quote + image="/images/case-studies/booz-allen/banner1.png" + author="MARTIN FOLKOFF, SENIOR LEAD TECHNOLOGIST AT BOOZ ALLEN HAMILTON" +>}} +"Kubernetes alone enables a dramatic reduction in cost as resources are prioritized to the day's event" +{{< /case-studies/quote >}} + +

In addition to its work with the Department of Interior for recreation.gov, Booz Allen Hamilton has brought Kubernetes to various Defense, Intelligence, and civilian agencies. Says Boyd: "When there's a regulatory change in an agency, or a legislative change in Congress, or an executive order that changes the way you do business, how do I deploy that and get that out to the people who need it rapidly? At the end of the day, that's the problem we're trying to help the government solve with tools like Kubernetes."

+ +

For recreation.gov, the impact was clear and immediate. With the Kubernetes platform, Folkoff says, "if a new requirement for a permit comes out, we have the ability to design and develop and implement that completely independently of reserving a campsite. It provides a much better experience to users." Today, changes can be implemented in about 30 minutes, compared to the multiple hours or even days legacy government applications require to review the code, get approval, and deploy the fix. Recreation.gov deploys to production on average 10 times a day.

+ +

Developer velocity has been improved. "When I want to do monitoring or security or logging, I don't have to do anything to my services or my application to enable that anymore," says Boyd. "I get all of this magic just by being on the Kubernetes platform." With all of those things built in, developers can create and publish new services to production within one week.

+ +

Additionally, Folkoff says, "supporting the large, existing monoliths in the government is extremely expensive," and migrating into a more modern platform has resulted in perhaps 50% cost savings. "Kubernetes alone enables a dramatic reduction in cost as resources are prioritized to the day's event," he says. "For example, during a popular campsite release, camping-related services are scaled out while permit services are scaled down."

+ +

So far, "Kubernetes is a great solution for us," says Folkoff. "It allows us to rapidly iterate on our clients' demands." Looking ahead, the team sees further adoption of the Kubernetes platform across federal agencies. Says Boyd: "You get the ability for the rapid delivery of business value for your customers. You now have observability into everything that you're doing. You don't have these onesies and twosies unicorn servers anymore. Now everything that you deploy is deployed in the same way, it's all instrumented the same way, and it's all built and deployed the same way through our CI/CD processes."

+ +

They also see a push toward re-platforming. "There's still a lot of legacy workloads out there," says Boyd. "We've got the new challenges of greenfield development and integration with legacy systems, but also that brown field of 'Hey, how do I take this legacy monolith and get it onto a platform where now it's instrumented with all the magic of the Kubernetes platform without having to do a whole lot to my application?' I think re-platforming is a pretty big use case for the government right now."

+ +

And given the success that they've had with Kubernetes so far, Boyd says, "I think at this point that technology is becoming pretty easy to sell." Adds Folkoff: "People are really excited about being able to deploy, scale, be reliable, and do cheaper maintenance of all of this."

From 69b402cccfc70d407c549f40a6aba7ce86f50837 Mon Sep 17 00:00:00 2001 From: kinzhi Date: Wed, 27 Apr 2022 16:12:10 +0800 Subject: [PATCH 032/119] [zh]Update content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml (#33234) * [zh]Update content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml * [zh]Update content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml * Update content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml Co-authored-by: Qiming Teng Co-authored-by: Qiming Teng --- .../admin/dns/dns-horizontal-autoscaler.yaml | 88 +++++++++++++++---- 1 file changed, 71 insertions(+), 17 deletions(-) diff --git a/content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml b/content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml index dff87cf851..f7e7660e7a 100644 --- a/content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml +++ b/content/zh/examples/admin/dns/dns-horizontal-autoscaler.yaml @@ -1,33 +1,87 @@ +kind: ServiceAccount +apiVersion: v1 +metadata: + name: kube-dns-autoscaler + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:kube-dns-autoscaler +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["replicationcontrollers/scale"] + verbs: ["get", "update"] + - apiGroups: ["apps"] + resources: ["deployments/scale", "replicasets/scale"] + verbs: ["get", "update"] +# 待以下 issue 修复后，请删除 Configmaps +# kubernetes-incubator/cluster-proportional-autoscaler#16 + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: system:kube-dns-autoscaler +subjects: + - kind: ServiceAccount + name: kube-dns-autoscaler + namespace: kube-system +roleRef: + kind: ClusterRole + name: system:kube-dns-autoscaler + apiGroup: rbac.authorization.k8s.io + +--- apiVersion: apps/v1 kind: Deployment metadata: - name: dns-autoscaler + name: kube-dns-autoscaler namespace: kube-system labels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler + kubernetes.io/cluster-service: "true" spec: selector: matchLabels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler template: metadata: labels: - k8s-app: dns-autoscaler + k8s-app: kube-dns-autoscaler spec: + priorityClassName: system-cluster-critical + securityContext: + seccompProfile: + type: RuntimeDefault + supplementalGroups: [ 65534 ] + fsGroup: 65534 + nodeSelector: + kubernetes.io/os: linux containers: - name: autoscaler - image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.6.0 + image: k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.4 resources: - requests: - cpu: 20m - memory: 10Mi + requests: + cpu: "20m" + memory: "10Mi" command: - - /cluster-proportional-autoscaler - - --namespace=kube-system - - --configmap=dns-autoscaler - - --target= - # When cluster is using large nodes(with more cores), "coresPerReplica" should dominate. - # If using small nodes, "nodesPerReplica" should dominate. - - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"min":1}} - - --logtostderr=true - - --v=2 + - /cluster-proportional-autoscaler + - --namespace=kube-system + - --configmap=kube-dns-autoscaler + # 应该保持目标与 cluster/addons/dns/kube-dns.yaml.base 同步 + - --target= + #当集群使用大节点（有更多核）时，“coresPerReplica”应该占主导地位。 + #如果使用小节点，“nodesPerReplica“ 应该占主导地位。 + - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true,"includeUnschedulableNodes":true}} + - --logtostderr=true + - --v=2 + tolerations: + - key: "CriticalAddonsOnly" + operator: "Exists" + serviceAccountName: kube-dns-autoscaler From 77277242665f7f773bfd0b313bb618fbe56342e4 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 27 Apr 2022 16:14:03 +0800 Subject: [PATCH 033/119] --amend Signed-off-by: xin.li --- content/zh/docs/reference/glossary/pod-security-policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/reference/glossary/pod-security-policy.md b/content/zh/docs/reference/glossary/pod-security-policy.md index 6f29aa8d5a..f80416fb44 100644 --- a/content/zh/docs/reference/glossary/pod-security-policy.md +++ b/content/zh/docs/reference/glossary/pod-security-policy.md @@ -47,5 +47,5 @@ Pod 安全策略是集群级别的资源，它控制着 Pod 规约中的安全 PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. We recommend migrating to [Pod Security Admission](/docs/concepts/security/pod-security-admission/), or a 3rd party admission plugin. --> PodSecurityPolicy 自 Kubernetes v1.21 起已弃用，并将在 v1.25 中删除。 -我们建议迁移到 [Pod 安全准入](/zh/docs/concepts/security/pod-security-admission/) 或第三方准入插件。 +我们建议迁移到 [Pod 安全准入](/zh/docs/concepts/security/pod-security-admission/)或第三方准入插件。 From 2983e62cb0c9014e5394fcbef5e268abc6917491 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 27 Apr 2022 16:21:30 +0800 Subject: [PATCH 034/119] [zh] Sync denso Signed-off-by: xin.li --- .../denso/denso_featured_logo.svg | 1 + content/zh/case-studies/denso/index.html | 82 +++++++++++++++++++ 2 files changed, 83 insertions(+) create mode 100644 content/zh/case-studies/denso/denso_featured_logo.svg create mode 100644 content/zh/case-studies/denso/index.html diff --git a/content/zh/case-studies/denso/denso_featured_logo.svg b/content/zh/case-studies/denso/denso_featured_logo.svg new file mode 100644 index 0000000000..e2b26b2c8c --- /dev/null +++ b/content/zh/case-studies/denso/denso_featured_logo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/zh/case-studies/denso/index.html b/content/zh/case-studies/denso/index.html new file mode 100644 index 0000000000..31fb1279b9 --- /dev/null +++ b/content/zh/case-studies/denso/index.html @@ -0,0 +1,82 @@ +--- +title: Denso Case Study +linkTitle: Denso +case_study_styles: true +cid: caseStudies +logo: denso_featured_logo.svg +featured: true +weight: 4 +quote: > + We got Kubernetes experts involved on our team, and it dramatically accelerated development speed. + +new_case_study_styles: true +heading_background: /images/case-studies/denso/banner2.jpg +heading_title_text: Denso +use_gradient_overlay: true +subheading: > + How DENSO Is Fueling Development on the Vehicle Edge with Kubernetes +case_study_details: + - Company: Denso + - Location: Japan + - Industry: Automotive, Edge +--- + +

Challenge

+ +

DENSO Corporation is one of the biggest automotive components suppliers in the world. With the advent of connected cars, the company launched a Digital Innovation Department to expand into software, working on vehicle edge and vehicle cloud products. But there were several technical challenges to creating an integrated vehicle edge/cloud platform: "the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles," says R&D Product Manager Seiichi Koizumi.

+ +

Solution

+ +

Koizumi's team realized that because mobility services evolve every day, they needed the flexibility of the cloud native ecosystem for their platform. After considering other orchestrators, DENSO went with Kubernetes for orchestration and added Prometheus, Fluentd, Envoy, Istio, and Helm to the platform. Today, DENSO is using a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes (GKE, EKS, AKS).

+ +

Impact

+ +

Critical layer features can take 2-3 years to implement in the traditional, waterfall model of development at DENSO. With the Kubernetes platform and agile methods, there's a 2-month development cycle for non-critical software. Now, ten new applications are released a year, and a new prototype is introduced every week. "By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation," says Koizumi.

+ +{{< case-studies/quote + image="/images/case-studies/denso/banner1.png" + author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO" +>}} +"Another disruptive innovation is coming, so to survive in this situation, we need to change our culture." +{{< /case-studies/quote >}} + +{{< case-studies/lead >}} +Spun off from Toyota in 1949, DENSO Corporation is one of the top automotive suppliers in the world today, with consolidated net revenue of $48.3 billion. +{{< /case-studies/lead >}} + +

The company's mission is "contributing to a better world by creating value together with a vision for the future"—and part of that vision in recent years has been development on the vehicle edge and vehicle cloud.

+ +

With the advent of connected cars, DENSO established a Digital Innovation Department to expand its business beyond the critical layer of the engine, braking systems, and other automotive parts into the non-critical analytics and entertainment layer. Comparing connected cars to smartphones, R&D Product Manager Seiichi Koizumi says DENSO wants the ability to quickly and easily develop and install apps for the "blank slate" of the car, and iterate them based on the driver's preferences. Thus "we need a flexible application platform," he says.

+ +

But working on vehicle edge and vehicle cloud products meant there were several technical challenges: "the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles," says Koizumi. "We are tackling these challenges to create an integrated vehicle edge/cloud platform."

+ +{{< case-studies/quote author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO" >}} +"We got Kubernetes experts involved on our team, and it dramatically accelerated development speed." +{{< /case-studies/quote >}} + +

Koizumi's team realized that because mobility services evolve every day, they needed the flexibility of the cloud native ecosystem for their platform. As they evaluated technologies, they were led by these criteria: Because their service-enabler business needed to support multiple cloud and on-premise environments, the solution needed to be cloud agnostic, with no vendor lock-in and open governance. It also had to support an edge-cloud integrated environment.

+ +

After considering other orchestrators, DENSO went with Kubernetes for orchestration and added Prometheus, Fluentd, Envoy, Istio, and Helm to the platform. During implementation, the team used "design thinking to clarify use cases and their value proposition," says Koizumi. Next, an agile development team worked on a POC, then an MVP, in DevOps style. "Even in the development phase, we are keeping a channel to end users," he adds.

+ +

One lesson learned during this process was the value of bringing in experts. "We tried to learn Kubernetes and cloud native technologies from scratch, but it took more time than expected," says Koizumi. "We got Kubernetes experts involved on our team, and it dramatically accelerated development speed."

+ +{{< case-studies/quote + image="/images/case-studies/denso/banner4.jpg" + author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO" +>}} +"By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation." +{{< /case-studies/quote >}} + +

Today, DENSO is using a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes on GKE, EKS, and AKS. "We are developing a vehicle edge/cloud integrated platform based on a microservice and service mesh architecture," says Koizumi. "We extend cloud into multiple vehicle edges and manage it as a unified platform."

+ +

Cloud native has enabled DENSO to deliver applications via its new dash cam, which has a secure connection that collects data to the cloud. "It's like a smartphone," he says. "We are installing new applications and getting the data through the cloud, and we can keep updating new applications all through the dash cam."

+ +

The unified cloud native platform, combined with agile development, has had a positive impact on productivity. Critical layer features—those involving engines or braking systems, for example—can take 2-3 years to implement at DENSO, because of the time needed to test safety, but also because of the traditional, waterfall model of development. With the Kubernetes platform and agile methods, there's a 2-month development cycle for non-critical software. Now, ten new applications are released a year, and with the department's scrum-style development, a new prototype is introduced every week.

+ +

Application portability has also led to greater developer efficiency. "There's no need to care about differences in the multi-cloud platform anymore," says Koizumi. Now, "we are also trying to have the same portability between vehicle edge and cloud platform."

+ +

Another improvement: Automotive Tier-1 suppliers like DENSO always have multiple Tier-2 suppliers. "To provide automotive-grade high-availability services, we tried to do the same thing on a multi-cloud platform," says Koizumi. Before Kubernetes, maintaining two different systems simultaneously was difficult. "By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation," he says.

+ +

Cloud native has also profoundly changed the culture at DENSO. The Digital Innovation Department is known as "Noah's Ark," and it has grown from 2 members to 70—with plans to more than double in the next year. The way they operate is completely different from the traditional Japanese automotive culture. But just as the company embraced change brought by hybrid cars in the past decade, Koizumi says, they're doing it again now, as technology companies have moved into the connected car space. "Another disruptive innovation is coming," he says, "so to survive in this situation, we need to change our culture."

+ +

Looking ahead, Koizumi and his team are expecting serverless and zero-trust security architecture to be important enhancements of Kubernetes. They are glad DENSO has come along for the ride. "Mobility service businesses require agility and flexibility," he says. "DENSO is trying to bring cloud native flexibility into the vehicle infrastructure."

From 7f57114949be450b4cb826f8277aee89c1c40f3f Mon Sep 17 00:00:00 2001 From: Arhell Date: Wed, 27 Apr 2022 12:39:02 +0300 Subject: [PATCH 035/119] [id] Update external provisioner specification reference --- content/id/docs/concepts/storage/storage-classes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/id/docs/concepts/storage/storage-classes.md b/content/id/docs/concepts/storage/storage-classes.md index a3684755af..e5f3265768 100644 --- a/content/id/docs/concepts/storage/storage-classes.md +++ b/content/id/docs/concepts/storage/storage-classes.md @@ -89,7 +89,7 @@ Kamu tidak dibatasi untuk hanya menggunakan _provisioner_ internal yang disediak pada list yang tersedia (yang memiliki nama dengan prefix "kubernetes.io" dan didistribusikan bersamaan dengan Kubernetes). Kamu juga dapat menjalankan dan mendefinisikan _provisioner_ eksternal yang merupakan program independen selama -program tersebut menerapkan [spesifikasi](https://git.k8s.io/community/contributors/design-proposals/storage/volume-provisioning.md) +program tersebut menerapkan [spesifikasi](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-provisioning.md) yang didefinisikan oleh Kubernetes. Penulis dari _provisioner_ eksternal Kubernetes memiliki kuasa penuh akan tempat dimana kode sumber yang mereka tulis, bagaimana mekanisme penyediaan (_provisioning_) dilakukan, serta bagaimana hal tersebut dapat dijalankan, From 511c74564d2466c4677e8daf5e0f6680dab88850 Mon Sep 17 00:00:00 2001 From: Jens-Uwe Walther Date: Wed, 27 Apr 2022 12:14:03 +0200 Subject: [PATCH 036/119] Remove gp3 from supported EBS volume types gp3 is not supported for the in-tree plugin "kubernetes.io/aws-ebs", only for external EBS CSI driver. So we have to remove it here See following issue which shows that it does not work: https://github.com/kubernetes/website/issues/33036#issuecomment-1110766774 --- content/en/docs/concepts/storage/storage-classes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/concepts/storage/storage-classes.md b/content/en/docs/concepts/storage/storage-classes.md index 788f592abe..53ee88a2e7 100644 --- a/content/en/docs/concepts/storage/storage-classes.md +++ b/content/en/docs/concepts/storage/storage-classes.md @@ -49,7 +49,7 @@ metadata: name: standard provisioner: kubernetes.io/aws-ebs parameters: - type: gp3 + type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: @@ -271,9 +271,9 @@ parameters: fsType: ext4 ``` -* `type`: `io1`, `gp2`, `gp3`, `sc1`, `st1`. See +* `type`: `io1`, `gp2`, `sc1`, `st1`. See [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html) - for details. Default: `gp3`. + for details. Default: `gp2`. * `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are generally round-robin-ed across all active zones where Kubernetes cluster has a node. `zone` and `zones` parameters must not be used at the same time. From 107ff07d789731fb85635400e0e211f64762394d Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 27 Apr 2022 18:14:39 +0800 Subject: [PATCH 037/119] [zh] Update case-studies/ing Signed-off-by: xin.li --- content/zh/case-studies/ing/index.html | 78 +++++++++++++++++++ .../zh/case-studies/ing/ing_featured_logo.svg | 1 + 2 files changed, 79 insertions(+) create mode 100644 content/zh/case-studies/ing/index.html create mode 100644 content/zh/case-studies/ing/ing_featured_logo.svg diff --git a/content/zh/case-studies/ing/index.html b/content/zh/case-studies/ing/index.html new file mode 100644 index 0000000000..037ba9775d --- /dev/null +++ b/content/zh/case-studies/ing/index.html @@ -0,0 +1,78 @@ +--- +title: ING Case Study +linkTitle: ING +case_study_styles: true +cid: caseStudies +weight: 50 +featured: true +quote: > + The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that's quite feasible to us. + +new_case_study_styles: true +heading_background: /images/case-studies/ing/banner1.jpg +heading_title_logo: /images/ing_logo.png +subheading: > + Driving Banking Innovation with Cloud Native +case_study_details: + - Company: ING + - Location: Amsterdam, Netherlands + - Industry: Finance +--- + +

Challenge

+ +

After undergoing an agile transformation, ING realized it needed a standardized platform to support the work their developers were doing. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it's not really useful for a company to have one hundred wheels, instead of one good wheel.

+ +

Solution

+ +

Using Kubernetes for container orchestration and Docker for containerization, the ING team began building an internal public cloud for its CI/CD pipeline and green-field applications. The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. The bank-account management app Yolt in the U.K. (and soon France and Italy) market already is live hosted on a Kubernetes framework. At least two greenfield projects currently on the Kubernetes framework will be going into production later this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform.

+ +

Impact

+ +

"Cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Infrastructure Architect Onno Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready."

+ +{{< case-studies/quote author="Thijs Ebbers, Infrastructure Architect, ING">}} +"The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that's quite feasible to us." +{{< /case-studies/quote >}} + +{{< case-studies/lead >}} +ING has long embraced innovation in banking, launching the internet-based ING Direct in 1997. +{{< /case-studies/lead >}} + +

In that same spirit, the company underwent an agile transformation a few years ago. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it's not really useful for a company to have one hundred wheels, instead of one good wheel."

+ +

Looking to standardize the deployment process within the company's strict security guidelines, the team looked at several solutions and found that in the past year, "Kubernetes won the container management framework wars," says Ebbers. "We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That's one of the reasons we got Kubernetes."

+ +

They also embraced Docker to address a major pain point in ING's CI/CD pipeline. Before containerization, "Every development team had to order a VM, and it was quite a heavy delivery model for them," says Infrastructure Architect Onno Van der Voort. "Another use case for containerization is when the application travels through the pipeline, they fire up Docker containers to do test work against the applications and after they've done the work, the containers get killed again."

+ +{{< case-studies/quote + image="/images/case-studies/ing/banner3.jpg" + author="Thijs Ebbers, Infrastructure Architect, ING" +>}} +"We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That's one of the reasons we got Kubernetes." +{{< /case-studies/quote >}} + +

Because of industry regulations, applications are only allowed to go through the pipeline, where compliance is enforced, rather than be deployed directly into a container. "We have to run the complete platform of services we need, many routing from different places," says Van der Voort. "We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It's complex." For that reason, ING has chosen to start on the OpenShift Origin Kubernetes distribution.

+ +

Already, "cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready."

+ +

The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. Some legacy applications are also being rewritten as cloud native in order to run on the framework. At least two smaller greenfield projects built on Kubernetes will go into production this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform.

+ +{{< case-studies/quote + image="/images/case-studies/ing/banner4.jpg" + author="Onno Van der Voort, Infrastructure Architect, ING" +>}} +"We have to run the complete platform of services we need, many routing from different places. We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It's complex." +{{< /case-studies/quote >}} + +

The team, however, doesn't see the bank's back-end systems going onto the Kubernetes platform. "Our philosophy is it only makes sense to move things to cloud if they are cloud native," says Van der Voort. "If you have traditional architecture, build traditional patterns, it doesn't hold any value to go to the cloud." Adds Cloud Platform Architect Alfonso Fernandez-Barandiaran: "ING has a strategy about where we will go, in order to improve our agility. So it's not about how cool this technology is, it's about finding the right technology and the right approach."

+ +

The Kubernetes framework will be hosting some greenfield projects that are high priority for ING: applications the company is developing in response to PSD2, the European Commission directive requiring more innovative online and mobile payments that went into effect at the beginning of 2018. For example, a bank-account management app called Yolt, serving the U.K. market (and soon France and Italy), was built on a Kubernetes platform and has gone into production. ING is also developing blockchain-enabled applications that will live on the Kubernetes platform. "We've been contacted by a lot of development teams that have ideas with what they want to do with containers," says Ebbers.

+ +{{< case-studies/quote author="Alfonso Fernandez-Barandiaran, Cloud Platform Architect, ING" >}} +Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology." +{{< /case-studies/quote >}} + +

Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology," says Fernandez-Barandiaran.

+ +

The results, after all, are worth the effort. "The big cloud native promise to our business is the ability to go from idea to production within 48 hours," says Ebbers. "That would require all these projects to be mature. We are some years away from this, but that's quite feasible to us."

diff --git a/content/zh/case-studies/ing/ing_featured_logo.svg b/content/zh/case-studies/ing/ing_featured_logo.svg new file mode 100644 index 0000000000..5a2df497c7 --- /dev/null +++ b/content/zh/case-studies/ing/ing_featured_logo.svg @@ -0,0 +1 @@ +kubernetes.io-logos \ No newline at end of file From 4993a0d743cd4bdd934ac92ee143f3cf20030d76 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Wed, 27 Apr 2022 19:03:39 +0800 Subject: [PATCH 038/119] [zh] Update workloads/pods/_index.md Signed-off-by: xin.li --- .../zh/docs/concepts/workloads/pods/_index.md | 50 +++++++++++++++---- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/content/zh/docs/concepts/workloads/pods/_index.md b/content/zh/docs/concepts/workloads/pods/_index.md index 4689f0aca3..5eec324a1a 100644 --- a/content/zh/docs/concepts/workloads/pods/_index.md +++ b/content/zh/docs/concepts/workloads/pods/_index.md @@ -92,6 +92,37 @@ Pod 的共享上下文包括一组 Linux 名字空间、控制组（cgroup）和 +## 使用 Pod {#using-pods} + +下面是一个 Pod 示例，它由一个运行镜像 `nginx:1.14.2` 的容器组成。 + +{{< codenew file="pods/simple-pod.yaml" >}} + +要创建上面显示的 Pod，请运行以下命令： + +```shell +kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml +``` + + +Pod 通常不是直接创建的，而是使用工作负载资源创建的。 +有关如何将 Pod 用于工作负载资源的更多信息，请参阅 [使用 Pod](#working-with-pods)。 + +### 用于管理 pod 的工作负载资源 + + -## 使用 Pod {#using-pods} - 通常你不需要直接创建 Pod，甚至单实例 Pod。 相反，你会使用诸如 {{< glossary_tooltip text="Deployment" term_id="deployment" >}} 或 @@ -187,9 +216,9 @@ that updates those files from a remote source, as in the following diagram: --> 例如，你可能有一个容器，为共享卷中的文件提供 Web 服务器支持，以及一个单独的 -“sidecar（挂斗）”容器负责从远端更新这些文件，如下图所示： +"边车 (sidercar)" 容器负责从远端更新这些文件，如下图所示： -{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}} +{{< figure src="/images/docs/pod.svg" alt="Pod creation diagram" class="diagram-medium" >}} -在节点上，{{< glossary_tooltip term_id="kubelet" text="kubelet" >}}并不直接监测 +在节点上，{{< glossary_tooltip term_id="kubelet" text="kubelet" >}} 并不直接监测 或管理与 Pod 模版相关的细节或模版的更新，这些细节都被抽象出来。 这种抽象和关注点分离简化了整个系统的语义，并且使得用户可以在不改变现有代码的 前提下就能扩展集群的行为。 @@ -412,7 +441,7 @@ Kubernetes 并不禁止你直接管理 Pod。对运行中的 Pod 的某些字段 Pods enable data sharing and communication among their constituent containters. --> -### 资源共享和通信 {#resource-sharing-and-communication} +### 资源共享和通信 {#resource-sharing-and-communication} Pod 使它的成员容器间能够进行数据共享和通信。 @@ -459,14 +488,13 @@ can find each other via `localhost`. The containers in a Pod can also communicat with each other using standard inter-process communications like SystemV semaphores or POSIX shared memory. Containers in different Pods have distinct IP addresses and can not communicate by IPC without -[special configuration](/docs/concepts/policy/pod-security-policy/). +and can not communicate by OS-level IPC without special configuration. Containers that want to interact with a container running in a different Pod can use IP networking to communicate. --> 在同一个 Pod 内，所有容器共享一个 IP 地址和端口空间，并且可以通过 `localhost` 发现对方。 他们也能通过如 SystemV 信号量或 POSIX 共享内存这类标准的进程间通信方式互相通信。 -不同 Pod 中的容器的 IP 地址互不相同，没有 -[特殊配置](/zh/docs/concepts/policy/pod-security-policy/) 就不能使用 IPC 进行通信。 +不同 Pod 中的容器的 IP 地址互不相同，没有特殊配置，无法通过 OS 级 IPC 进行通信就不能使用 IPC 进行通信。 如果某容器希望与运行于其他 Pod 中的容器通信，可以通过 IP 联网的方式实现。 你不限于指定此处列出的 "内置" 制备器（其名称前缀为 "kubernetes.io" 并打包在 Kubernetes 中）。 你还可以运行和指定外部制备器，这些独立的程序遵循由 Kubernetes 定义的 -[规范](https://git.k8s.io/community/contributors/design-proposals/storage/volume-provisioning.md)。 +[规范](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-provisioning.md)。 外部供应商的作者完全可以自由决定他们的代码保存于何处、打包方式、运行方式、使用的插件（包括 Flex）等。 代码仓库 [kubernetes-sigs/sig-storage-lib-external-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner) 包含一个用于为外部制备器编写功能实现的类库。你可以访问代码仓库 From 3660d10034d7cb583506dbeca022bc9b8784c8ae Mon Sep 17 00:00:00 2001 From: Qiming Teng Date: Sat, 23 Apr 2022 11:00:54 +0800 Subject: [PATCH 043/119] [zh] Update the nodes concept page --- .../zh/docs/concepts/architecture/nodes.md | 62 +++++++++---------- 1 file changed, 28 insertions(+), 34 deletions(-) diff --git a/content/zh/docs/concepts/architecture/nodes.md b/content/zh/docs/concepts/architecture/nodes.md index 1676e635d8..625c114df4 100644 --- a/content/zh/docs/concepts/architecture/nodes.md +++ b/content/zh/docs/concepts/architecture/nodes.md @@ -63,7 +63,7 @@ valid. For example, if you try to create a Node from the following JSON manifest 1. 节点上的 `kubelet` 向控制面执行自注册； 2. 你，或者别的什么人，手动添加一个 Node 对象。 -在你创建了 Node {{< glossary_tooltip text="object" term_id="object" >}}或者节点上的 +在你创建了 Node {{< glossary_tooltip text="对象" term_id="object" >}}或者节点上的 `kubelet` 执行了自注册操作之后，控制面会检查新的 Node 对象是否合法。 例如，如果你尝试使用下面的 JSON 对象来创建 Node 对象： @@ -100,10 +100,9 @@ it becomes healthy. You, or a {{< glossary_tooltip term_id="controller" text="controller">}}, must explicitly delete the Node object to stop that health checking. --> -Kubernetes 会一直保存着非法节点对应的对象，并持续检查该节点是否已经 -变得健康。 -你，或者某个{{< glossary_tooltip term_id="controller" text="控制器">}}必需显式地 -删除该 Node 对象以停止健康检查操作。 +Kubernetes 会一直保存着非法节点对应的对象，并持续检查该节点是否已经变得健康。 +你，或者某个{{< glossary_tooltip term_id="controller" text="控制器">}}必须显式地删除该 +Node 对象以停止健康检查操作。 {{< /note >}} ### 节点名称唯一性 {#node-name-uniqueness} -节点的[名称](/docs/concepts/overview/working-with-objects/names#names)用来标识 Node 对象。 +节点的[名称](/zh/docs/concepts/overview/working-with-objects/names#names)用来标识 Node 对象。 没有两个 Node 可以同时使用相同的名称。 Kubernetes 还假定名字相同的资源是同一个对象。 就 Node 而言，隐式假定使用相同名称的实例会具有相同的状态（例如网络配置、根磁盘内容） 和类似节点标签这类属性。这可能在节点被更改但其名称未变时导致系统状态不一致。 @@ -177,8 +176,8 @@ When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and kubelets are only authorized to create/modify their own Node resource. --> 启用[Node 鉴权模式](/zh/docs/reference/access-authn-authz/node/)和 -[NodeRestriction 准入插件](/zh/docs/reference/access-authn-authz/admission-controllers/#noderestriction) -时，仅授权 `kubelet` 创建或修改其自己的节点资源。 +[NodeRestriction 准入插件](/zh/docs/reference/access-authn-authz/admission-controllers/#noderestriction)时， +仅授权 `kubelet` 创建或修改其自己的节点资源。 {{< note >}} -更多细节参考[安全腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。 +更多细节参考[安全地腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。 {{< note >}} ### 状况 {#condition} -`conditions` 字段描述了所有 `Running` 节点的状态。状况的示例包括： +`conditions` 字段描述了所有 `Running` 节点的状况。状况的示例包括： 节点控制器在确认 Pod 在集群中已经停止运行前，不会强制删除它们。 -你可以看到这些可能在无法访问的节点上运行的 Pod 处于 `Terminating` 或者 `Unknown` 状态。 +你可以看到可能在这些无法访问的节点上运行的 Pod 处于 `Terminating` 或者 `Unknown` 状态。 如果 kubernetes 不能基于下层基础设施推断出某节点是否已经永久离开了集群， 集群管理员可能需要手动删除该节点对象。 从 Kubernetes 删除节点对象将导致 API 服务器删除节点上所有运行的 Pod 对象并释放它们的名字。 @@ -480,7 +479,7 @@ the Kubernetes API. ### 信息（Info） {#info} Info 指的是节点的一般信息，如内核版本、Kubernetes 版本（`kubelet` 和 `kube-proxy` 版本）、 -容器运行时详细信息，以及 节点使用的操作系统。 +容器运行时详细信息，以及节点使用的操作系统。 `kubelet` 从节点收集这些信息并将其发布到 Kubernetes API。 ## 心跳 {#heartbeats} + Kubernetes 节点发送的心跳帮助你的集群确定每个节点的可用性，并在检测到故障时采取行动。 对于节点，有两种形式的心跳: @@ -504,7 +504,7 @@ Kubernetes 节点发送的心跳帮助你的集群确定每个节点的可用性 Each Node has an associated Lease object. --> * 更新节点的 `.status` -* `kube-node-lease` {{}}中的 +* `kube-node-lease` {{}}中的 [Lease（租约）](/docs/reference/kubernetes-api/cluster-resources/lease-v1/)对象。 每个节点都有一个关联的 Lease 对象。 @@ -588,7 +588,7 @@ This period can be configured using the `--node-monitor-period` flag on the - 在节点不可达的情况下，在 Node 的 `.status` 中更新 `Ready` 状况。 在这种情况下，节点控制器将 NodeReady 状况更新为 `Unknown` 。 - 如果节点仍然无法访问：对于不可达节点上的所有 Pod 触发 - [API-发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)。 + [API 发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)操作。 默认情况下，节点控制器在将节点标记为 `Unknown` 后等待 5 分钟提交第一个驱逐请求。 默认情况下，节点控制器每 5 秒检查一次节点状态，可以使用 `kube-controller-manager` @@ -772,8 +772,8 @@ Note that by default, both configuration options described below, thus not activating the graceful node shutdown functionality. To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values. --> -注意，默认情况下，下面描述的两个配置选项，`ShutdownGracePeriod` 和 -`ShutdownGracePeriodCriticalPods` 都是被设置为 0 的，因此不会激活体面节点关闭功能。 +注意，默认情况下，下面描述的两个配置选项，`shutdownGracePeriod` 和 +`shutdownGracePeriodCriticalPods` 都是被设置为 0 的，因此不会激活体面节点关闭功能。 要激活此功能特性，这两个 kubelet 配置选项要适当配置，并设置为非零值。 -例如，如果设置了 `ShutdownGracePeriod=30s` 和 `ShutdownGracePeriodCriticalPods=10s`， +例如，如果设置了 `shutdownGracePeriod=30s` 和 `shutdownGracePeriodCriticalPods=10s`， 则 kubelet 将延迟 30 秒关闭节点。 在关闭期间，将保留前 20（30 - 10）秒用于体面终止常规 Pod， 而保留最后 10 秒用于终止[关键 Pod](/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical)。 @@ -820,11 +820,6 @@ reserved for terminating [critical pods](/docs/tasks/administer-cluster/guarante When pods were evicted during the graceful node shutdown, they are marked as failed. Running `kubectl get pods` shows the status of the the evicted pods as `Shutdown`. And `kubectl describe pod` indicates that the pod was evicted because of node shutdown: - -``` -Reason: Terminated -Message: Pod was terminated in response to imminent node shutdown. -``` --> {{< note >}} 当 Pod 在正常节点关闭期间被驱逐时，它们会被标记为已经失败（Failed）。 @@ -973,8 +968,8 @@ containing the pod priority class values and their respective shutdown periods. --> 如果此功能特性被启用，但没有提供配置数据，则不会出现排序操作。 -使用此功能特性需要启用 `GracefulNodeShutdownBasedOnPodPriority` 功能特性， -并将 kubelet 配置中的 `ShutdownGracePeriodByPodPriority` 设置为期望的配置， +使用此功能特性需要启用 `GracefulNodeShutdownBasedOnPodPriority` 特性门控， +并将 kubelet 配置中的 `shutdownGracePeriodByPodPriority` 设置为期望的配置， 其中包含 Pod 的优先级类数值以及对应的关闭期限。 要在节点上启用交换内存，必须启用kubelet 的 `NodeSwap` 特性门控， 同时使用 `--fail-swap-on` 命令行参数或者将 `failSwapOn` -[配置](/zh/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) -设置为 false。 +[配置](/zh/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)设置为 false。 + + +生成证书密钥 certificate-key 操作的帮助命令 +

@@ -64,10 +81,12 @@ certificate-key 操作的帮助命令 +

[实验] 到 '真实' 主机根文件系统的路径。 +

From bf5afce482aa3c7fdd33f1ffea750d8fc123324a Mon Sep 17 00:00:00 2001 From: howieyuen Date: Thu, 28 Apr 2022 11:47:18 +0800 Subject: [PATCH 045/119] [zh]resync tls task: Manage TLS Certificates in a Cluster --- .../tasks/tls/managing-tls-in-a-cluster.md | 382 +++++++++++++----- 1 file changed, 285 insertions(+), 97 deletions(-) diff --git a/content/zh/docs/tasks/tls/managing-tls-in-a-cluster.md b/content/zh/docs/tasks/tls/managing-tls-in-a-cluster.md index 3c0605c60c..f6204c527f 100644 --- a/content/zh/docs/tasks/tls/managing-tls-in-a-cluster.md +++ b/content/zh/docs/tasks/tls/managing-tls-in-a-cluster.md @@ -29,49 +29,78 @@ Kubernetes 提供 `certificates.k8s.io` API，可让你配置由你控制的证 {{< note >}} -使用 `certificates.k8s.io` API 创建的证书由指定 CA 颁发。将集群配置为使用集群根目录 -CA 可以达到这个目的，但是你永远不要依赖这一假定。不要以为 -这些证书将针对群根目录 CA 进行验证。 +使用 `certificates.k8s.io` API 创建的证书由指定 [CA](#a-note-to-cluster-administrators) 颁发。 +将集群配置为使用集群根目录 CA 可以达到这个目的，但是你永远不要依赖这一假定。 +不要以为这些证书将针对群根目录 CA 进行验证。 {{< /note >}} ## {{% heading "prerequisites" %}} -{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} +{{< include "task-tutorial-prereqs.md" >}} + + +你需要 `cfssl` 工具。 +你可以从 [https://github.com/cloudflare/cfssl/releases](https://github.com/cloudflare/cfssl/releases) +下载 `cfssl`。 + +本文中某些步骤使用 `jq` 工具。如果你没有 `jq`，你可以通过操作系统的软件源安装， +或者从 [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/) 获取。 ## 集群中的 TLS 信任 -信任 Pod 中运行的应用程序所提供的 CA 通常需要一些额外的应用程序配置。 +信任 Pod 中运行的应用程序所提供的[自定义 CA](#a-note-to-cluster-administrators) 通常需要一些额外的应用程序配置。 你需要将 CA 证书包添加到 TLS 客户端或服务器信任的 CA 证书列表中。 例如，你可以使用 Golang TLS 配置通过解析证书链并将解析的证书添加到 -[`tls.Config`](https://godoc.org/crypto/tls#Config) 结构中的 `RootCAs` +[`tls.Config`](https://pkg.go.dev/crypto/tls#Config) 结构中的 `RootCAs` 字段中。 -你可以用你的应用能够访问到的 -[ConfigMap](/zh/docs/tasks/configure-pod-container/configure-pod-configmap) -的形式来发布 CA 证书。 +{{< note >}} + +即使自定义 CA 证书可能包含在文件系统中（在 ConfigMap `kube-root-ca.crt` 中）， +除了验证内部 Kubernetes 端点之外，你不应将该证书颁发机构用于任何目的。 +内部 Kubernetes 端点的一个示例是默认命名空间中名为 `kubernetes` 的服务。 + +如果你想为你的工作负载使用自定义证书颁发机构，你应该单独生成该 CA， +并使用你的 Pod 有读权限的 [ConfigMap](/zh/docs/tasks/configure-pod-container/configure-pod-configmap) +分发该 CA 证书。 +{{< /note >}} -## 下载并安装 CFSSL - -本例中使用的 cfssl 工具可以在 [github.com/cloudflare/cfssl/releases](https://github.com/cloudflare/cfssl/releases) 下载。 - - 其中 `192.0.2.24` 是服务的集群 IP，`my-svc.my-namespace.svc.cluster.local` 是服务的 DNS 名称，`10.0.34.2` 是 Pod 的 IP，而 `my-pod.my-namespace.pod.cluster.local` 是 Pod 的 DNS 名称。 -你能看到以下的输出： +你能看到的输出类似于： ``` -2017/03/21 06:48:17 [INFO] generate received request -2017/03/21 06:48:17 [INFO] received CSR -2017/03/21 06:48:17 [INFO] generating key: ecdsa-256 -2017/03/21 06:48:17 [INFO] encoded CSR +2022/02/01 11:45:32 [INFO] generate received request +2022/02/01 11:45:32 [INFO] received CSR +2022/02/01 11:45:32 [INFO] generating key: ecdsa-256 +2022/02/01 11:45:32 [INFO] encoded CSR ``` 此命令生成两个文件；它生成包含 PEM 编码 -[pkcs#10](https://tools.ietf.org/html/rfc2986) 证书请求的 `server.csr`， +[PKCS#10](https://tools.ietf.org/html/rfc2986) 证书请求的 `server.csr`， 以及 PEM 编码密钥的 `server-key.pem`，用于待生成的证书。 -## 创建证书签名请求对象发送到 Kubernetes API +## 创建证书签名请求（CSR）对象发送到 Kubernetes API 使用以下命令创建 CSR YAML 文件，并发送到 API 服务器： @@ -178,7 +192,7 @@ metadata: name: my-svc.my-namespace spec: request: $(cat server.csr | base64 | tr -d '\n') - signerName: kubernetes.io/kubelet-serving + signerName: example.com/serving usages: - digital signature - key encipherment @@ -188,9 +202,9 @@ EOF 请注意，在步骤 1 中创建的 `server.csr` 文件是 base64 编码并存储在 -`.spec.request` 字段中的。我们还要求提供 “digital signature（数字签名）”， +`.spec.request` 字段中的。你还要求提供 “digital signature（数字签名）”， “密钥加密（key encipherment）” 和 “服务器身份验证（server auth）” 密钥用途， -由 `kubernetes.io/kubelet-serving` 签名程序签名的证书。 +由 `example.com/serving` 示例签名程序签名的证书。 你也可以要求使用特定的 `signerName`。更多信息可参阅 [支持的签署者名称](/zh/docs/reference/access-authn-authz/certificate-signing-requests/#signers)。 @@ -215,28 +229,30 @@ kubectl describe csr my-svc.my-namespace Name: my-svc.my-namespace Labels: Annotations: -CreationTimestamp: Tue, 21 Mar 2017 07:03:51 -0700 +CreationTimestamp: Tue, 01 Feb 2022 11:49:15 -0500 Requesting User: yourname@example.com +Signer: example.com/serving Status: Pending Subject: - Common Name: my-svc.my-namespace.svc.cluster.local + Common Name: my-pod.my-namespace.pod.cluster.local Serial Number: Subject Alternative Names: - DNS Names: my-svc.my-namespace.svc.cluster.local + DNS Names: my-pod.my-namespace.pod.cluster.local + my-svc.my-namespace.svc.cluster.local IP Addresses: 192.0.2.24 10.0.34.2 Events: ``` -## 批准证书签名请求 +## 批准证书签名请求（CSR） {#get-the-certificate-signing-request-approved} [证书签名请求](/zh/docs/reference/access-authn-authz/certificate-signing-requests/) 的批准或者是通过自动批准过程完成的，或由集群管理员一次性完成。 @@ -250,29 +266,167 @@ kubectl certificate approve my-svc.my-namespace certificatesigningrequest.certificates.k8s.io/my-svc.my-namespace approved ``` - -## 下载证书并使用它 - -CSR 被签署并获得批准后，你应该看到以下内容： + +你现在应该能看到如下输出： ```shell kubectl get csr ``` ```none -NAME AGE REQUESTOR CONDITION -my-svc.my-namespace 10m yourname@example.com Approved,Issued +NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION +my-svc.my-namespace 10m example.com/serving yourname@example.com Approved +``` + + +这意味着证书请求已被批准，并正在等待请求的签名者对其签名。 + + +## 签名证书签名请求（CSR） {#sign-the-certificate-signing-request} + +接下来，你将扮演证书签署者的角色，颁发证书并将其上传到 API 服务器。 + +签名者通常会使用其 `signerName` 查看对象的 CertificateSigningRequest API， +检查它们是否已被批准，为这些请求签署证书，并使用已颁发的证书更新 API 对象状态。 + + +### 创建证书颁发机构 + +你需要授权在新证书上提供数字签名。 + +首先，通过运行以下命令创建签名证书： + +```shell +cat < +你应该看到类似于以下的输出： + +```none +2022/02/01 11:50:39 [INFO] generating a new CA key and certificate from CSR +2022/02/01 11:50:39 [INFO] generate received request +2022/02/01 11:50:39 [INFO] received CSR +2022/02/01 11:50:39 [INFO] generating key: rsa-2048 +2022/02/01 11:50:39 [INFO] encoded CSR +2022/02/01 11:50:39 [INFO] signed certificate with serial number 263983151013686720899716354349605500797834580472 +``` + + +这会产生一个证书颁发机构密钥文件（`ca-key.pem`）和证书（`ca.pem`）。 + + + +### 颁发证书 + +{{< codenew file="tls/server-signing-config.json" >}} + + +使用 `server-signing-config.json` 签名配置、证书颁发机构密钥文件和证书来签署证书请求： + +```shell +kubectl get csr my-svc.my-namespace -o jsonpath='{.spec.request}' | \ + base64 --decode | \ + cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - | \ + cfssljson -bare ca-signed-server +``` + + +你应该看到类似于以下的输出： + +``` +2022/02/01 11:52:26 [INFO] signed certificate with serial number 576048928624926584381415936700914530534472870337 +``` + + +这会生成一个签名的服务证书文件，`ca-signed-server.pem`。 + + + +### 上传签名证书 + +最后，在 API 对象的状态中填充签名证书： + +```shell +kubectl get csr my-svc.my-namespace -o json | \ + jq '.status.certificate = "'$(base64 ca-signed-server.pem | tr -d '\n')'"' | \ + kubectl replace --raw /apis/certificates.k8s.io/v1/certificatesigningrequests/my-svc.my-namespace/status -f - +``` + +{{< note >}} + +这使用命令行工具 [`jq`](https://stedolan.github.io/jq/) +在 `.status.certificate` 字段中填充 base64 编码的内容。 +如果你没有 `jq` 工具，你还可以将 JSON 输出保存到文件中，手动填充此字段，然后上传结果文件。 +{{< /note >}} + + +批准 CSR 并上传签名证书后，运行： + +```shell +kubectl get csr +``` + + +输入类似于： + +```none +NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION +my-svc.my-namespace 20m example.com/serving yourname@example.com Approved,Issued ``` -你可以通过运行以下命令下载颁发的证书并将其保存到 `server.crt` 文件中： +## 下载证书并使用它 + +现在，作为请求用户，你可以通过运行以下命令下载颁发的证书并将其保存到 `server.crt` 文件中： + +CSR 被签署并获得批准后，你应该看到以下内容： ```shell kubectl get csr my-svc.my-namespace -o jsonpath='{.status.certificate}' \ @@ -280,31 +434,74 @@ kubectl get csr my-svc.my-namespace -o jsonpath='{.status.certificate}' \ ``` -现在你可以将 `server.crt` 和 `server-key.pem` 作为键值对来启动 HTTPS 服务器。 +现在你可以将 `server.crt` 和 `server-key.pem` 填充到 +{{}} 中， +稍后你可以将其挂载到 Pod 中（例如，用于提供 HTTPS 的网络服务器）。 + +```shell +kubectl create secret tls server --cert server.crt --key server-key.pem +``` + +```none +secret/server created +``` + + +最后，你可以将 `ca.pem` 填充到 +{{}} +并将其用作信任根来验证服务证书： + +```shell +kubectl create configmap example-serving-ca --from-file ca.crt=ca.pem +``` + +```none +configmap/example-serving-ca created +``` -## 批准证书签名请求 +## 批准证书签名请求（CSR） {#approving-certificate-signing-requests} Kubernetes 管理员（具有适当权限）可以使用 `kubectl certificate approve` 和 -`kubectl certificate deny` 命令手动批准（或拒绝）证书签名请求。 +`kubectl certificate deny` 命令手动批准（或拒绝）证书签名请求（CSR）。 但是，如果你打算大量使用此 API，则可以考虑编写自动化的证书控制器。 +{{< caution >}} + +批准证书 CSR 的能力决定了在你的环境中谁信任谁。 +不应广泛或轻率地授予批准 CSR 的能力。 + +在授予 `approve` 权限之前，你应该确保自己充分了解批准人的验证要求**和**颁发特定证书的后果。 +{{< /caution >}} + -无论上述机器或人使用 kubectl，批准者的作用是验证 CSR 满足如下两个要求： +无论上述机器或人使用 kubectl，“批准者”的作用是验证 CSR 满足如下两个要求： 当且仅当满足这两个要求时，审批者应该批准 CSR，否则拒绝 CSR。 - -## 关于批准权限的警告 - -批准 CSR 的能力决定了群集中的信任关系。这也包括 Kubernetes API 所信任的人。 -批准 CSR 的能力不能过于广泛和轻率。 -在给予本许可之前，应充分了解上一节中提到的挑战和发布特定证书的后果。 +有关证书批准和访问控制的更多信息， +请阅读[证书签名请求](/zh/docs/reference/access-authn-authz/certificate-signing-requests/)参考页。 ## 给集群管理员的一个建议 -本教程假设已经为 certificates API 配置了签名者。Kubernetes 控制器管理器 -提供了一个签名者的默认实现。要启用它，请为控制器管理器设置 +本页面假设已经为 certificates API 配置了签名者。 +Kubernetes 控制器管理器提供了一个签名者的默认实现。要启用它，请为控制器管理器设置 `--cluster-signing-cert-file` 和 `--cluster-signing-key-file` 参数， 使之取值为你的证书机构的密钥对的路径。 - From b00a8192a1006730532cd747b19f9402f789eb88 Mon Sep 17 00:00:00 2001 From: Craig Box Date: Thu, 28 Apr 2022 23:19:49 +1200 Subject: [PATCH 046/119] Add 1.23 blog post --- ...04-29-kubernetes-1.23-release-interview.md | 319 ++++++++++++++++++ 1 file changed, 319 insertions(+) create mode 100644 content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md diff --git a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md new file mode 100644 index 0000000000..3b62c25477 --- /dev/null +++ b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md @@ -0,0 +1,319 @@ +--- +layout: blog +title: "Frontiers, fsGroups and frogs: the Kubernetes 1.23 release interview" +date: 2022-04-29 +--- + +**Author**: Craig Box (Google) + +One of the highlights of hosting the weekly [Kubernetes Podcast from Google](https://kubernetespodcast.com/) is talking to the release managers for each new Kubernetes version. The release team is constantly refreshing. Many working their way from small documentation fixes, step up to shadow roles, and then eventually lead a release. + +As we prepare for the 1.24 release next week, [in accordance with long-standing tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the story of 1.23. The release was led by [Rey Lejano](https://twitter.com/reylejano), a Field Engineer at SUSE. [I spoke to Rey](https://kubernetespodcast.com/episode/167-kubernetes-1.23/) in December, as he was awaiting the birth of his first child. + +Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/), so you hear all our stories from the Cloud Native community, including the story of 1.24 next week. + +*This transcript has been lightly edited and condensed for clarity.* + +--- + +**CRAIG BOX: I'd like to start with what is, of course, on top of everyone's mind at the moment. Let's talk African clawed frogs!** + +REY LEJANO: [CHUCKLES] Oh, you mean [Xenopus lavis](https://en.wikipedia.org/wiki/African_clawed_frog), the scientific name for the African clawed frog? + +**CRAIG BOX: Of course.** + +REY LEJANO: Not many people know, but my background and my degree is actually in microbiology, from the University of California Davis. I did some research for about four years in biochemistry, in a biochemistry lab, and I [do have a research paper published](https://www.sciencedirect.com/science/article/pii/). It's actually on glycoproteins, particularly something called "cortical granule lectin". We used frogs, because they generate lots and lots of eggs, from which we can extract the protein. That protein prevents polyspermy. When the sperm goes into the egg, the egg releases a glycoprotein, cortical granule lectin, to the membrane, and prevents any other sperm from going inside the egg. + +**CRAIG BOX: Were you able to take anything from the testing that we did on frogs and generalize that to higher-order mammals, perhaps?** + +REY LEJANO: Yes. Since mammals also have cortical granule lectin, we were able to analyze both the convergence and the evolutionary pattern, not just from multiple species of frogs, but also into mammals as well. + +**CRAIG BOX: Now, there's a couple of different threads to unravel here. When you were young, what led you into the fields of biology, and perhaps more the technical side of it?** + +REY LEJANO: I think it was mostly from family, since I do have a family history in the medical field that goes back generations. So I kind of felt like that was the natural path going into college. + +**CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?** + +REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://en.wikipedia.org/wiki/ITIL). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually. + +**CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.** + +REY LEJANO: Yes, absolutely. It's not very cloud native. [CHUCKLES] + +**CRAIG BOX: Not at all.** + +REY LEJANO: You don't really hear about it in the cloud native landscape. Definitely, you can tell someone's been in the field for a little bit, if they specialize or have worked with ITIL before. + +**CRAIG BOX: You mentioned that you wanted to get out of the basement. That is quite often where people put the programmers. Did they just give you a bit of light in the new basement?** + +REY LEJANO: [LAUGHS] They did give us much better lighting. Able to get some vitamin D sometimes, as well. + +**CRAIG BOX: To wrap up the discussion about your previous career — over the course of the last year, with all of the things that have happened in the world, I could imagine that microbiology skills may be more in demand than perhaps they were when you studied them?** + +REY LEJANO: Oh, absolutely. I could definitely see a big increase of numbers of people going into the field. Also, reading what's going on with the world currently kind of brings back all the education I've learned in the past, as well. + +**CRAIG BOX: Do you keep in touch with people you went through school with?** + +REY LEJANO: Just some close friends, but not in the microbiology field. + +**CRAIG BOX: One thing that I think will probably happen as a result of the pandemic is a renewed interest in some of these STEM fields. It will be interesting to see what impact that has on society at large.** + +REY LEJANO: Yeah. I think that'll be great. + +**CRAIG BOX: You mentioned working at a consultancy doing IT management, application performance monitoring, and so on. When did Kubernetes come into your professional life?** + +REY LEJANO: One of my good friends at the company I worked at, left in mid-2015. He went on to a company that was pretty heavily into Docker. He taught me a little bit. I did my first "docker run" around 2015, maybe 2016. Then, one of the applications we were using for the ITIL framework was containerized around 2018 or so, also in Kubernetes. At that time, it was pretty buggy. That was my initial introduction to Kubernetes and containerised applications. + +Then I left that company, and I actually joined my friend over at [RX-M](https://rx-m.com/), which is a cloud native consultancy and training firm. They specialize in Docker and Kubernetes. I was able to get my feet wet. I got my CKD, got my CKA as well. And they were really, really great at encouraging us to learn more about Kubernetes and also to be involved in the community. + +**CRAIG BOX: You will have seen, then, the life cycle of people adopting Kubernetes and containerization at large, through your own initial journey and then through helping customers. How would you characterize how that journey has changed from the early days to perhaps today?** + +REY LEJANO: I think the early days, there was a lot of questions of, why do I have to containerize? Why can't I just stay with virtual machines? + +**CRAIG BOX: It's a line item on your CV.** + +REY LEJANO: [CHUCKLES] It is. And nowadays, I think people know the value of using containers, of orchestrating containers with Kubernetes. I don't want to say "jumping on the bandwagon", but it's become the de-facto standard to orchestrate containers. + +**CRAIG BOX: It's not something that a consultancy needs to go out and pitch to customers that they should be doing. They're just taking it as, that will happen, and starting a bit further down the path, perhaps.** + +REY LEJANO: Absolutely. + +**CRAIG BOX: Working at a consultancy like that, how much time do you get to work on improving process, perhaps for multiple customers, and then looking at how you can upstream that work, versus paid work that you do for just an individual customer at a time?** + +REY LEJANO: Back then, it would vary. They helped me introduce myself, and I learned a lot about the cloud native landscape and Kubernetes itself. They helped educate me as to how the cloud native landscape, and the tools around it, can be used together. My boss at that company, Randy, he actually encouraged us to start contributing upstream, and encouraged me to join the release team. He just said, this is a great opportunity. Definitely helped me with starting with the contributions early on. + +**CRAIG BOX: Was the release team the way that you got involved with upstream Kubernetes contribution?** + +REY LEJANO: Actually, no. My first contribution was with SIG Docs. I met Taylor Dolezal — he was the release team lead for 1.19, but he is involved with SIG Docs as well. I met him at KubeCon 2019, I sat at his table during a luncheon. I remember Paris Pittman was hosting this luncheon at the Marriott. Taylor says he was involved with SIG Docs. He encouraged me to join. I started joining into meetings, started doing a few drive-by PRs. That's what we call them — drive-by — little typo fixes. Then did a little bit more, started to send better or higher quality pull requests, and also reviewing PRs. + +**CRAIG BOX: When did you first formally take your release team role?** + +REY LEJANO: That was in [1.18](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md), in December. My boss at the time encouraged me to apply. I did, was lucky enough to get accepted for the release notes shadow. Then from there, stayed in with release notes for a few cycles, then went into Docs, naturally then led Docs, then went to Enhancements, and now I'm the release lead for 1.23. + +**CRAIG BOX: I don't know that a lot of people think about what goes into a good release note. What would you say does?** + +REY LEJANO: [CHUCKLES] You have to tell the end user what has changed or what effect that they might see in the release notes. It doesn't have to be highly technical. It could just be a few lines, and just saying what has changed, what they have to do if they have to do anything as well. + +**CRAIG BOX: As you moved through the process of shadowing, how did you learn from the people who were leading those roles?** + +REY LEJANO: I said this a few times when I was the release lead for this cycle. You get out of the release team as much as you put in, or it directly aligns to how much you put in. I learned a lot. I went into the release team having that mindset of learning from the role leads, learning from the other shadows, as well. That's actually a saying that my first role lead told me. I still carry it to heart, and that was back in 1.18. That was Eddie, in the very first meeting we had, and I still carry it to heart. + +**CRAIG BOX: You, of course, were [the release lead for 1.23](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23). First of all, congratulations on the release.** + +REY LEJANO: Thank you very much. + +**CRAIG BOX: The theme for this release is [The Next Frontier](https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/). Tell me the story of how we came to the theme and then the logo.** + +REY LEJANO: The Next Frontier represents a few things. It not only represents the next enhancements in this release, but Kubernetes itself also has a history of Star Trek references. The original codename for Kubernetes was Project Seven, a reference to Seven of Nine, originally from Star Trek Voyager. Also the seven spokes in the helm in the logo of Kubernetes as well. And, of course, Borg, the predecessor to Kubernetes. + +The Next Frontier continues that Star Trek reference. It's a fusion of two titles in the Star Trek universe. One is [Star Trek V, the Final Frontier](https://en.wikipedia.org/wiki/Star_Trek_V:_The_Final_Frontier), and the Star Trek: The Next Generation. + +**CRAIG BOX: Do you have any opinion on the fact that Star Trek V was an odd-numbered movie, and they are [canonically referred to as being lesser than the even-numbered ones](https://screenrant.com/star-trek-movies-odd-number-curse-explained/)?** + +REY LEJANO: I can't say, because I am such a sci-fi nerd that I love all of them even though they're bad. Even the post-Next Generation movies, after the series, I still liked all of them, even though I know some weren't that great. + +**CRAIG BOX: Am I right in remembering that Star Trek V was the one directed by William Shatner?** + +REY LEJANO: Yes, that is correct. + +**CRAIG BOX: I think that says it all.** + +REY LEJANO: [CHUCKLES] Yes. + +**CRAIG BOX: Now, I understand that the theme comes from a part of the [SIG Release charter](https://github.com/kubernetes/community/blob/master/sig-release/charter.md)?** + +REY LEJANO: Yes. There's a line in the SIG Release charter, "ensure there is a consistent group of community members in place to support the release process across time." With the release team, we have new shadows that join every single release cycle. With this, we're growing with this community. We're growing the release team members. We're growing SIG Release. We're growing the Kubernetes community itself. For a lot of people, this is their first time contributing to open source, so that's why I say it's their new open source frontier. + +**CRAIG BOX: And the logo is obviously very Star Trek-inspired. It sort of surprised me that it took that long for someone to go this route.** + +REY LEJANO: I was very surprised as well. I had to relearn Adobe Illustrator to create the logo. + +**CRAIG BOX: This your own work, is it?** + +REY LEJANO: This is my own work. + +**CRAIG BOX: It's very nice.** + +REY LEJANO: Thank you very much. Funny, the galaxy actually took me the longest time versus the ship. Took me a few days to get that correct. I'm always fine-tuning it, so there might be a final change when this is actually released. + +**CRAIG BOX: No frontier is ever truly final.** + +REY LEJANO: True, very true. + +**CRAIG BOX: Moving now from the theme of the release to the substance, perhaps, what is new in 1.23?** + +REY LEJANO: We have 47 enhancements. I'm going to run through most of the stable ones, if not all of them, some of the key Beta ones, and a few of the Alpha enhancements for 1.23. + +One of the key enhancements is [dual-stack IPv4/IPv6](https://github.com/kubernetes/enhancements/issues/563), which went GA in 1.23. + +Some background info: dual-stack was introduced as Alpha in 1.15. You probably saw a keynote at KubeCon 2019. Back then, the way dual-stack worked was that you needed two services — you needed a service per IP family. You would need a service for IPv4 and a service for IPv6. It was refactored in 1.20. In 1.21, it was in Beta; clusters were enabled to be dual-stack by default. + +And then in 1.23 we did remove the IPv6 dual-stack feature flag. It's not mandatory to use dual-stack. It's actually not "default" still. The pods, the services still default to single-stack. There are some requirements to be able to use dual-stack. The nodes have to be routable on IPv4 and IPv6 network interfaces. You need a CNI plugin that supports dual-stack. The pods themselves have to be configured to be dual-stack. And the services need the ipFamilyPolicy field to specify prefer dual-stack, or require dual-stack. + +**CRAIG BOX: This sounds like there's an implication in this that v4 is still required. Do you see a world where we can actually move to v6-only clusters?** + +REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to come. I remember a long time ago, they kept saying "it's going to be all IPv6", and that was decades ago. + +**CRAIG BOX: I think I may have mentioned on the show before, but there was [a meeting in London that Vint Cerf attended](https://www.youtube.com/watch?v=AEaJtZVimqs), and he gave a public presentation at the time to say, now is the time of v6. And that was 10 years ago at least. It's still not the time of v6, and my desktop still doesn't have Linux on it. One day.** + +REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23. + +One of the other highlights of 1.23 is [pod security admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) going to Beta. I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running. + +There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed. + +**CRAIG BOX: You mentioned there that PodSecurityPolicy is due to be deprecated in two releases' time. Are we lining up these features so that pod security admission will be GA at that time?** + +REY LEJANO: Yes. Absolutely. I'll talk about that for another feature in a little bit as well. There's also another feature that went to GA. It was an API that went to GA, and therefore the Beta API is now deprecated. I'll talk about that a little bit. + +**CRAIG BOX: All right. Let's talk about what's next on the list.** + +REY LEJANO: Let's move on to more stable enhancements. One is the [TTL controller](https://github.com/kubernetes/enhancements/issues/592). This cleans up jobs and pods after the jobs are finished. There is a TTL timer that starts when the job or pod is finished. This TTL controller watches all the jobs, and ttlSecondsAfterFinished needs to be set. The controller will see if the ttlSecondsAfterFinished, combined with the last transition time, if it's greater than now. If it is, then it will delete the job and the pods of that job. + +**CRAIG BOX: Loosely, it could be called a garbage collector?** + +REY LEJANO: Yes. Garbage collector for pods and jobs, or jobs and pods. + +**CRAIG BOX: If Kubernetes is truly becoming a programming language, it of course has to have a garbage collector implemented.** + +REY LEJANO: Yeah. There's another one, too, coming in Alpha. [CHUCKLES] + +**CRAIG BOX: Tell me about that.** + +REY LEJANO: That one is coming in in Alpha. It's actually one of my favorite features, because there's only a few that I'm going to highlight today. [PVCs for StafeulSet will be cleaned up](https://github.com/kubernetes/enhancements/issues/1847). It will auto-delete PVCs created by StatefulSets, when you delete that StatefulSet. + +**CRAIG BOX: What's next on our tour of stable features?** + +REY LEJANO: Next one is, [skip volume ownership change goes to stable](https://github.com/kubernetes/enhancements/issues/695). This is from SIG Storage. There are times when you're running a stateful application, like many databases, they're sensitive to permission bits changing underneath. Currently, when a volume is bind mounted inside the container, the permissions of that volume will change recursively. It might take a really long time. + +Now, there's a field, the fsGroupChangePolicy, which allows you, as a user, to tell Kubernetes how you want the permission and ownership change for that volume to happen. You can set it to always, to always change permissions, or just on mismatch, to only do it when the permission ownership changes at the top level is different from what is expected. + +**CRAIG BOX: It does feel like a lot of these enhancements came from a very particular use case where someone said, "hey, this didn't work for me and I've plumbed in a feature that works with exactly the thing I need to have".** + +REY LEJANO: Absolutely. People create issues for these, then create Kubernetes enhancement proposals, and then get targeted for releases. + +**CRAIG BOX: Another GA feature in this release — ephemeral volumes.** + +REY LEJANO: We've always been able to use empty dir for ephemeral volumes, but now we could actually have [ephemeral inline volumes](https://github.com/kubernetes/enhancements/issues/1698), meaning that you could take your standard CSI driver and be able to use ephemeral volumes with it. + +**CRAIG BOX: And, a long time coming, [CronJobs](https://github.com/kubernetes/enhancements/issues/19).** + +REY LEJANO: CronJobs is a funny one, because it was stable before 1.23. For 1.23, it was still tracked,but it was just cleaning up some of the old controller. With CronJobs, there's a v2 controller. What was cleaned up in 1.23 is just the old v1 controller. + +**CRAIG BOX: Were there any other duplications or major cleanups of note in this release?** + +REY LEJANO: Yeah. There were a few you might see in the major themes. One's a little tricky, around FlexVolumes. This is one of the efforts from SIG Storage. They have an effort to migrate in-tree plugins to CSI drivers. This is a little tricky, because FlexVolumes were actually deprecated in November 2020. We're [formally announcing it in 1.23](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors). + +**CRAIG BOX: FlexVolumes, in my mind, predate CSI as a concept. So it's about time to get rid of them.** + +REY LEJANO: Yes, it is. There's another deprecation, just some [klog specific flags](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog), but other than that, there are no other big deprecations in 1.23. + +**CRAIG BOX: The buzzword of the last KubeCon, and in some ways the theme of the last 12 months, has been secure software supply chain. What work is Kubernetes doing to improve in this area?** + +REY LEJANO: For 1.23, Kubernetes is now SLSA compliant at Level 1, which means that provenance attestation files that describe the staging and release phases of the release process are satisfactory for the SLSA framework. + +**CRAIG BOX: What needs to happen to step up to further levels?** + +REY LEJANO: Level 1 means a few things — that the build is scripted; that the provenance is available, meaning that the artifacts are verified and they're handed over from one phase to the next; and describes how the artifact is produced. Level 2 means that the source is version-controlled, which it is, provenance is authenticated, provenance is service-generated, and there is a build service. There are four levels of SLSA compliance. + +**CRAIG BOX: It does seem like the levels were largely influenced by what it takes to build a big, secure project like this. It doesn't seem like it will take a lot of extra work to move up to verifiable provenance, for example. There's probably just a few lines of script required to meet many of those requirements.** + +REY LEJANO: Absolutely. I feel like we're almost there; we'll see what will come out of 1.24. And I do want to give a big shout-out to SIG Release and Release Engineering, primarily to Adolfo García Veytia, who is aka Puerco on GitHub and on Slack. He's been driving this forward. + +**CRAIG BOX: You've mentioned some APIs that are being graduated in time to replace their deprecated version. Tell me about the new HPA API.** + +REY LEJANO: The [horizontal pod autoscaler v2 API](https://github.com/kubernetes/enhancements/issues/2702), is now stable, which means that the v2beta2 API is deprecated. Just for everyone's knowledge, the v1 API is not being deprecated. The difference is that v2 adds support for multiple and custom metrics to be used for HPA. + +**CRAIG BOX: There's also now a facility to validate my CRDs with an expression language.** + +REY LEJANO: Yeah. You can use the [Common Expression Language, or CEL](https://github.com/google/cel-spec), to validate your CRDs, so you no longer need to use webhooks. This also makes the CRDs more self-contained and declarative, because the rules are now kept within the CRD object definition. + +**CRAIG BOX: What new features, perhaps coming in Alpha or Beta, have taken your interest?** + +REY LEJANO: Aside from pod security policies, I really love [ephemeral containers](https://github.com/kubernetes/enhancements/issues/277) supporting kubectl debug. It launches an ephemeral container and a running pod, shares those pod namespaces, and you can do all your troubleshooting with just running kubectl debug. + +**CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.** + +REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do --watch, and it will also sort with the --watch option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well. + +**CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?** + +REY LEJANO: More of what I look out for to help the customers. + +**CRAIG BOX: Right.** + +REY LEJANO: I really love kubectl events. Really love the PVCs being cleaned up with StatefulSets. Most of it's for selfish reasons that it will improve troubleshooting efforts. [CHUCKLES] + +**CRAIG BOX: I have always hoped that a release team lead would say to me, "yes, I have selfish reasons. And I finally got something I wanted in."** + +REY LEJANO: [LAUGHS] + +**CRAIG BOX: Perhaps I should run to be release team lead, just so I can finally get init containers fixed once and for all.** + +REY LEJANO: Oh, init containers, I've been looking for that for a while. I've actually created animated GIFs on how init containers will be run with that Kubernetes enhancement proposal, but it's halted currently. + +**CRAIG BOX: One day.** + +REY LEJANO: One day. Maybe I shouldn't stay halted. + +**CRAIG BOX: You mentioned there are obviously the things you look out for. Are there any things that are coming down the line, perhaps Alpha features or maybe even just proposals you've seen lately, that you're personally really looking forward to seeing which way they go?** + +REY LEJANO: Yeah. Oone is a very interesting one, it affects the whole community, so it's not just for personal reasons. As you may have known, Dockershim is deprecated. And we did release a blog that it will be removed in 1.24. + +**CRAIG BOX: Scared a bunch of people.** + +REY LEJANO: Scared a bunch of people. From a survey, we saw that a lot of people are still using Docker and Dockershim. One of the enhancements for 1.23 is, [kubelet CRI goes to Beta](https://github.com/kubernetes/enhancements/issues/2040). This promotes the CRI API, which is required. This had to be in Beta for Dockershim to be removed in 1.24. + +**CRAIG BOX: Now, in the last release team lead interview, [we spoke with Savitha Raghunathan](https://kubernetespodcast.com/episode/157-kubernetes-1.22/), and she talked about what she would advise you as her successor. It was to look out for the mental health of the team members. How were you able to take that advice on board?** + +REY LEJANO: That was great advice from Savitha. A few things I've made note of with each release team meeting. After each release team meeting, I stop the recording, because we do record all the meetings and post them on YouTube. And I open up the floor to anyone who wants to say anything that's not recorded, that's not going to be on the agenda. Also, I tell people not to work on weekends. I broke this rule once, but other than that, I told people it could wait. Just be mindful of your mental health. + +**CRAIG BOX: It's just been announced that [James Laverack from Jetstack](https://twitter.com/JamesLaverack/status/1466834312993644551) will be the release team lead for 1.24. James and I shared an interesting Mexican dinner at the last KubeCon in San Diego.** + +REY LEJANO: Oh, nice. I didn't know you knew James. + +**CRAIG BOX: The British tech scene. We're a very small world. What will your advice to James be?** + +REY LEJANO: What I would tell James for 1.24 is use teachable moments in the release team meetings. When you're a shadow for the first time, it's very daunting. It's very difficult, because you don't know the repos. You don't know the release process. Everyone around you seems like they know the release process, and very familiar with what the release process is. But as a first-time shadow, you don't know all the vernacular for the community. I just advise to use teachable moments. Take a few minutes in the release team meetings to make it a little easier for new shadows to ramp up and to be familiar with the release process. + +**CRAIG BOX: Has there been major evolution in the process in the time that you've been involved? Or do you think that it's effectively doing what it needs to do?** + +REY LEJANO: It's always evolving. I remember my first time in release notes, 1.18, we said that our goal was to automate and program our way out so that we don't have a release notes team anymore. That's changed [CHUCKLES] quite a bit. Although there's been significant advancements in the release notes process by Adolfo and also James, they've created a subcommand in krel to generate release notes. + +But nowadays, all their release notes are richer. Still not there at the automation process yet. Every release cycle, there is something a little bit different. For this release cycle, we had a production readiness review deadline. It was a soft deadline. A production readiness review is a review by several people in the community. It's actually been required since 1.21, and it ensures that the enhancements are observable, scalable, supportable, and it's safe to operate in production, and could also be disabled or rolled back. In 1.23, we had a deadline to have the production readiness review completed by a specific date. + +**CRAIG BOX: How have you found the change of schedule to three releases per year rather than four?** + +REY LEJANO: Moving to three releases a year from four, in my opinion, has been an improvement, because we support the last three releases, and now we can actually support the last releases in a calendar year instead of having 9 months out of 12 months of the year. + +**CRAIG BOX: The next event on the calendar is a [Kubernetes contributor celebration](https://www.kubernetes.dev/events/kcc2021/) starting next Monday. What can we expect from that event?** + +REY LEJANO: This is our second time running this virtual event. It's a virtual celebration to recognize the whole community and all of our accomplishments of the year, and also contributors. There's a number of events during this week of celebration. It starts the week of December 13. + +There's events like the Kubernetes Contributor Awards, where SIGs honor and recognize the hard work of the community and contributors. There's also a DevOps party game as well. There is a cloud native bake-off. I do highly suggest people to go to [kubernetes.dev/celebration](https://www.kubernetes.dev/events/past-events/2021/kcc2021/) to learn more. + +**CRAIG BOX: How exactly does one judge a virtual bake-off?** + +REY LEJANO: That I don't know. [CHUCKLES] + +**CRAIG BOX: I tasted my scones. I think they're the best. I rate them 10 out of 10.** + +REY LEJANO: Yeah. That is very difficult to do virtually. I would have to say, probably what the dish is, how closely it is tied with Kubernetes or open source or to CNCF. There's a few judges. I know Josh Berkus and Rin Oliver are a few of the judges running the bake-off. + +**CRAIG BOX: Yes. We spoke with Josh about his love of the kitchen, and so he seems like a perfect fit for that role.** + +REY LEJANO: He is. + +**CRAIG BOX: Finally, your wife and yourself are expecting your first child in January. Have you had a production readiness review for that?** + +REY LEJANO: I think we failed that review. [CHUCKLES] + +**CRAIG BOX: There's still time.** + +REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and "--apply" again. + +--- + +_[Rey Lejano](https://twitter.com/reylejano) is a field engineer at SUSE, by way of Rancher Labs, and was the release team lead for Kubernetes 1.23. He is now also a co-chair for SIG Docs. His son Liam is now 3 and a half months old._ + +_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._ From f5e420e226299879af510c78722e0522c6044efa Mon Sep 17 00:00:00 2001 From: Craig Box Date: Thu, 28 Apr 2022 23:26:53 +1200 Subject: [PATCH 047/119] You know, I actually wanted some --s --- .../_posts/2022-04-29-kubernetes-1.23-release-interview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md index 3b62c25477..cc8cad881b 100644 --- a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md +++ b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md @@ -234,7 +234,7 @@ REY LEJANO: Aside from pod security policies, I really love [ephemeral container **CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.** -REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do --watch, and it will also sort with the --watch option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well. +REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do `--watch`, and it will also sort with the `--watch` option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well. **CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?** @@ -310,7 +310,7 @@ REY LEJANO: I think we failed that review. [CHUCKLES] **CRAIG BOX: There's still time.** -REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and "--apply" again. +REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and `--apply` again. --- From 279fbc658d89898531ff38713acbcbd66d79368f Mon Sep 17 00:00:00 2001 From: SataQiu Date: Thu, 28 Apr 2022 19:53:21 +0800 Subject: [PATCH 048/119] Update content/en/docs/reference/labels-annotations-taints/audit-annotations.md Signed-off-by: SataQiu --- .../reference/labels-annotations-taints/audit-annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/labels-annotations-taints/audit-annotations.md b/content/en/docs/reference/labels-annotations-taints/audit-annotations.md index a0ef3a1531..18714f43df 100644 --- a/content/en/docs/reference/labels-annotations-taints/audit-annotations.md +++ b/content/en/docs/reference/labels-annotations-taints/audit-annotations.md @@ -68,7 +68,7 @@ See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more informatio ## authorization.k8s.io/reason -Example: `authorization.k8s.io/decision: "Human-readable reason for the decision"` +Example: `authorization.k8s.io/reason: "Human-readable reason for the decision"` This annotation gives reason for the [decision](#authorization-k8s-io-decision) in Kubernetes audit logs. From 0e56bf9cae76bf36c91d6948927797940771f28e Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Wed, 30 Mar 2022 19:14:36 +0200 Subject: [PATCH 049/119] blog: storage capacity GA --- .../2022-05-06-storage-capacity-GA/index.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md diff --git a/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md new file mode 100644 index 0000000000..35d6838f51 --- /dev/null +++ b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md @@ -0,0 +1,79 @@ +--- +layout: blog +title: "Storage Capacity Tracking reaches GA in Kubernetes 1.24" +date: 2022-05-06 +slug: storage-capacity-ga +--- + + **Authors:** Patrick Ohly (Intel) + +The v1.24 release of Kubernetes brings [storage capacity](/docs/concepts/storage/storage-capacity/) +tracking as a generally available feature. + +## Problems we have solved + +As explained in more detail in the [previous blog post about this +feature](/blog/2021/04/14/local-storage-features-go-beta/), storage capacity +tracking allows a CSI driver to publish information about remaining +capacity. The kube-scheduler then uses that information to pick suitable nodes +for a Pod when that Pod has volumes that still need to be provisioned. + +Without this information, a Pod may get stuck without ever being scheduled onto +a suitable node because kube-scheduler has to choose blindly and always ends up +picking a node for which the volume cannot be provisioned because the +underlying storage system managed by the CSI driver does not have sufficient +capacity left. + +Because CSI drivers publish storage capacity information that gets used at a +later time when it might not be up-to-date anymore, it can still happen that a +node is picked that doesn't work out after all. Volume provisioning recovers +from that by informing the scheduler that it needs to try again with a +different node. + +[Load +tests](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/storage-capacity-tracking.md) +that were done again for promotion to GA confirmed that all storage in a +cluster can be consumed by Pods with storage capacity tracking whereas Pods got +stuck without it. + +## Problems we have *not* solved + +Recovery from a failed volume provisioning attempt has one known limitation: if a Pod +uses two volumes and only one of them could be provisioned, then all future +scheduling decisions are limited by the already provisioned volume. If that +volume is local to a node and the other volume cannot be provisioned there, the +Pod is stuck. This problem pre-dates storage capacity tracking and while the +additional information makes it less likely to occur, it cannot be avoided in +all cases, except of course by only using one volume per Pod. + +An idea for solving this was proposed in a [KEP +draft](https://github.com/kubernetes/enhancements/pull/1703): volumes that were +provisioned and haven't been used yet cannot have any valuable data and +therefore could be freed and provisioned again elsewhere. SIG Storage is +looking for interested developers who want to continue working on this. + +Also not solved is support in Cluster Autoscaler for Pods with volumes. For CSI +drivers with storage capacity tracking, a prototype was developed and discussed +in [a PR](https://github.com/kubernetes/autoscaler/pull/3887). It was meant to +work with arbitrary CSI drivers, but that flexibility made it hard to configure +and slowed down scale up operations: because autoscaler was unable to simulate +volume provisioning, it only scaled the cluster by one node at a time, which +was seen as insufficient. + +Therefore that PR was not merged and a different approach with tighter coupling +between autoscaler and CSI driver will be needed. For this a better +understanding is needed about which local storage CSI drivers are used in +combination with cluster autoscaling. Should this lead to a new KEP, then users +will have to try out an implementation in practice before it can move to beta +or GA. So please reach out to SIG Storage if you have an interest in this +topic. + +## Acknowledgements + +Thanks a lot to the members of the community who have contributed to this +feature or given feedback including members of [SIG +Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling), +[SIG +Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling), +and of course [SIG +Storage](https://github.com/kubernetes/community/tree/master/sig-storage)! From 78a765f16c4155a44db1f3ae76c3d6a01a2cb41f Mon Sep 17 00:00:00 2001 From: Ricardo Katz Date: Thu, 28 Apr 2022 11:37:50 -0300 Subject: [PATCH 050/119] Add Ingress NGINX v1.2.0 blogpost (#32965) * Add Ingress NGINX v1.2.0 blogpost * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Apply suggestions from code review Co-authored-by: Tim Bannister * Apply suggestions from code review Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Update content/en/blog/_posts/2022-04-15-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Tim Bannister * Fix some reviews * Change publishing date Co-authored-by: Tim Bannister --- .../index.md | 155 ++++++++++++++++++ .../ingress-post-chroot.png | Bin 0 -> 60860 bytes .../ingress-pre-chroot.png | Bin 0 -> 51860 bytes 3 files changed, 155 insertions(+) create mode 100644 content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md create mode 100644 content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png create mode 100644 content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-pre-chroot.png diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md new file mode 100644 index 0000000000..7fc83cad03 --- /dev/null +++ b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md @@ -0,0 +1,155 @@ +--- +layout: blog +title: 'Increasing the security bar in Ingress-NGINX v1.2.0' +date: 2022-04-28 +slug: ingress-nginx-1-2-0 +--- + +**Authors:** Ricardo Katz (VMware), James Strong (Chainguard) + +The [Ingress](/docs/concepts/services-networking/ingress/) may be one of the most targeted components +of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing +multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to +TLS certificates and their private keys). + +While it is a risky component in your architecture, it is still the most popular way to properly expose your services. + +Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't +do all proper sanitization before turning the configuration into an `nginx.conf` file, which may lead to information +disclosure risks. + +While we understand this risk and the real need to fix this, it's not an easy process to do, so we took another approach to reduce (but not remove!) this risk in the current (v1.2.0) release. + +## Meet Ingress NGINX v1.2.0 and the chrooted NGINX process + +One of the main challenges is that Ingress-NGINX runs the web proxy server (NGINX) alongside the Ingress +controller (the component that has access to Kubernetes API that and that creates the `nginx.conf` file). + +So, NGINX does have the same access to the filesystem of the controller (and Kubernetes service account token, and other configurations from the container). While splitting those components is our end goal, the project needed a fast response; that lead us to the idea of using `chroot()`. + +Let's take a look into what an Ingress-NGINX container looked like before this change: + +![Ingress NGINX pre chroot](ingress-pre-chroot.png) + +As we can see, the same container (not the Pod, the container!) that provides HTTP Proxy is the one that watches Ingress objects and writes the Container Volume + +Now, meet the new architecture: + +![Ingress NGINX post chroot](ingress-post-chroot.png) + +What does all of this mean? A basic summary is: that we are isolating the NGINX service as a container inside the +controller container. + +While this is not strictly true, to understand what was done here, it's good to understand how +Linux containers (and underlying mechanisms such as kernel namespaces) work. +You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article +[What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/). +(As you read that, bear in mind that Linux kernel namespaces are a different thing from +[Kubernetes namespaces](h/docs/concepts/overview/working-with-objects/namespaces/)). + +## Skip the talk, what do I need to use this new approach? + +While this increases the security, we made this feature an opt-in in this release so you can have +time to make the right adjustments in your environment(s). This new feature is only available from +release v1.2.0 of the Ingress-NGINX controller. + +There are two required changes in your deployments to use this feature: +* Append the suffix "-chroot" to the container image name. For example: `gcr.io/k8s-staging-ingress-nginx/controller-chroot:v1.2.0` +* In your Pod template for the Ingress controller, find where you add the capability `NET_BIND_SERVICE` and add the capability `SYS_CHROOT`. After you edit the manifest, you'll see a snippet like: + +```yaml +capabilities: + drop: + - ALL + add: + - NET_BIND_SERVICE + - SYS_CHROOT +``` + +If you deploy the controller using the official Helm chart then change the following setting in +`values.yaml`: + +```yaml +controller: + image: + chroot: true +``` + +Ingress controllers are normally set up cluster-wide (the IngressClass API is cluster scoped). If you manage the +Ingress-NGINX controller but you're not the overall cluster operator, then check with your cluster admin about +whether you can use the `SYS_CHROOT` capability, **before** you enable it in your deployment. + +## OK, but how does this increase the security of my Ingress controller? + +Take the following configuration snippet and imagine, for some reason it was added to your `nginx.conf`: +``` +location /randomthing/ { + alias /; + autoindex on; +} +``` + +If you deploy this configuration, someone can call `http://website.example/randomthing` and get some listing (and access) to the whole filesystem of the Ingress controller. + +Now, can you spot the difference between chrooted and non chrooted Nginx on the listings below? + +| Without extra `chroot()` | With extra `chroot()` | +|----------------------------|--------| +| `bin` | `bin` | +| `dev` | `dev` | +| `etc` | `etc` | +| `home` | | +| `lib` | `lib` | +| `media` | | +| `mnt` | | +| `opt` | `opt` | +| `proc` | `proc` | +| `root` | | +| `run` | `run` | +| `sbin` | | +| `srv` | | +| `sys` | | +| `tmp` | `tmp` | +| `usr` | `usr` | +| `var` | `var` | +| `dbg` | | +| `nginx-ingress-controller` | | +| `wait-shutdown` | | + +The one in left side is not chrooted. So NGINX has full access to the filesystem. The one in right side is chrooted, so a new filesystem with only the required files to make NGINX work is created. + +## What about other security improvements in this release? + +We know that the new `chroot()` mechanism helps address some portion of the risk, but still, someone +can try to inject commands to read, for example, the `nginx.conf` file and extract sensitive information. + +So, another change in this release (this is opt-out!) is the _deep inspector_. +We know that some directives or regular expressions may be dangerous to NGINX, so the deep inspector +checks all fields from an Ingress object (during its reconciliation, and also with a +[validating admission webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)) +to verify if any fields contains these dangerous directives. + +The ingress controller already does this for annotations, and our goal is to move this existing validation to happen inside +deep inspection as part of a future release. + +You can take a look into the existing rules in [https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go](https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go). + +Due to the nature of inspecting and matching all strings within relevant Ingress objects, this new feature may consume a bit more CPU. You can disable it by running the ingress controller with the command line argument `--deep-inspect=false`. + +## What's next? + +This is not our final goal. Our final goal is to split the control plane and the data plane processes. +In fact, doing so will help us also achieve a [Gateway](https://gateway-api.sigs.k8s.io/) API implementation, +as we may have a different controller as soon as it "knows" what to provide to the data plane +(we need some help here!!) + +Some other projects in Kubernetes already take this approach +(like [KPNG](​​https://github.com/kubernetes-sigs/kpng), the proposed replacement for `kube-proxy`), +and we plan to align with them and get the same experience for Ingress-NGINX. + +## Further reading + +If you want to take a look into how chrooting was done in Ingress NGINX, take a look +into [https://github.com/kubernetes/ingress-nginx/pull/8337](https://github.com/kubernetes/ingress-nginx/pull/8337) +The release v1.2.0 containing all the changes can be found at +[https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0) diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/ingress-post-chroot.png new file mode 100644 index 0000000000000000000000000000000000000000..d5d588a3bb61ce4db5f593f8ef69fe35fd35a07f GIT binary patch literal 60860 zcmcF~1zS|#7w&+83W9)uw6ugEAe~A|mvnb`HzHk9(jX$;&CoS;igXO!-Ob(e```N! z?(wl^4tuY?@?GybCqzz0?BxrB7Z3>KiV>H%7!fW+TSKGNLmkSb?1Bg#nyAmrW zdS0^8pV;SQDH=GROP`T{!7vnlL^;lRNsQk%+#rIqH@Fw1;d zFh_7M$NRJse!IAM%7DvM*MGlIu>|b3oUbArM)d@S!wHg3dF|p<-#M4fwG|oWbt-l; zi;Ta6XE>=XHNw=2|Ecbz-v%r-)>I5QB@2n`V@kdv6BK=Eq@+tGQz#l5@^O!b_trmB zhcZE`gyjtvu3W&2-Xs5qwgcXXI0@Mc<+Q#dl8g|$twS1LaQ zf1A$f5fKqGg-D_gupV;-EG;yEkMJEuR2_wEtgMWz9U(&YMtY7$ zhHqWW98KSfeUX+^@k7UhK;A;W2!B#?o!OmtaaA-T?K-mg>v-^G+T)-J)9{@lrjnqx zu$g5#GM~&W|3#5 zNf{5w;~5ae#>Sj^$O;RuBI%cNPBq@~##*w+4o(l!_V!NG4lOVKSX^InA6Qqy@tv+xu09o%drKeinqy(2#!F`FvTL$Bn@)>wt*;pRpKC2*5^x6O~wkH^)@F&@9} zlit5=TkRtaFKnQf8H&7odI?!1T=b8v%o;rs0o%TyrOg*kg+c@^9T}MA?K-==9IiX@ zWdZ^nt}pT1Kg-SzeNSV3^5jP6^!#GsXKLzk_vl@w4@6IN11}mSHk6qB_`?T*(H!s6 zuni_YJnv)6X@?~|UdgWvqSNNt?{u0zx}mOh4R(xghLe+%+hb(sT2YzlWJH-J?^!NmCRiL8EgH?s-&bSLj|$&Lo1tTX$nSP6YKQ zD&WYIf`X-$OP=9|`Hvn63k!GZ(x3;Gd-QuN4`^7toOXLRE)euk5N1Nr)d`~-LWzkO zq9N~+FBcLfd3TmkqVtxP+InF0QKqydd&h#FqMG_=-(E7@ zS$wtMY{SmqUm)>G74W*(S}Yr#7GAwqL@e|yB6*zNgL(UJ=N%9F*TTuF!2Hp%DESI~ z<}8Dd;NTvi&AF|Ufd?mAAL)#Y&E@4r#^pKKeJgdALrBkEmVQw1n*z}lx@rRogKlC} zF8TG4ny8d?w|p-C`vuW%EKJO3F+u&$=ukGt=PCzg6Fihr^(1K{4GL`MR2t^@ze@uTk)i; z3BIy%M*-oViJb?rRKTb5SaXN0#DgU*F@)cweG%SAKLdh$6c|Fu*#?YHBFVE+aqeu} zd2&9(P6&y=08e-ffkQ}>`Qam6>k5`bSvOHmlG5eY$dUMIlEV9`z+3Hy;|f%FW6MSk zzi3@IEnc_pSlRvifrBu7YvQ0W-6ZXSDv-POWj}lE7wPi)S<4z65N?*__BS8HMTGaC zT-=Hr7gcbo)1eW~cgNBDl#RN9eTA**Sja<{Y&=QuD>i!4S36gY1>;7Ogjdh=OoD>v zto*o+AMd}{f*OuqG<(mk?rj)3l7Ky<8SLyFa_n}l88*H5DKuD#i9K=GHP5|ZgbN%0Z#Auc63vU+};b3Irg3%9@@i?^fR2B zrDy(eaEBqz?j>JiPP^fdJFM{;Bi+=pT0Iw;9XNbXH6oyD`*!y>s&{*9qVU{GX2+3(v&&iIH6t33WtvOEyv*Pjk8i{UTMq98F5(^5GFfKHwXRI^+sGJM)n#ms_12RoV z_F69A*<3%Vhc+}LfqkQbHB*2&VvKOZ)}E(?XbUxO?^rsIlwtr4^`?8?Hof_{F6|PQ zWrVwE1AL-aszX(nvhbLbi>JiZN7LuRGvMS+8c5W_opM|!Yahf!L~}_NT#Ay}#<@E- zdC6m<`IB*?i#|X-pu=YHtO$>!KC1s|=U8xarZmeXQq%9NUqS>>7(U{V9BYeLJo zlu#RAjNGVKs&rJuONf8D0sPjI&ue-O7jKs~V1lf||S^-V3TL#mC7B?*G35|(Z7R5Yn z%DCudNM-7am)-j1i_7blj+B^g5Y7h9mF;!&*=jSL?0<3pNW+IH$Pu|@(!}1`jsWhk zNxxrW%_wSu2|ZZw^nO;{=k$>~aOr8&$8ck+S>}Tc*d$+$P}+Xu_tM+b1Q{@Wcl^|KD zb)X-Um+))>Q#SnGt(_=2-m-i1(@Ke(r^06}zVz+k%JF)8>fJf%QX#`ms%B#DB$Gmy z%pL3|x05R}$GowKN%iKzda9KDeh0hre{Rl2a0iVGuY7r?Yp2`A=GE7|CF+i*%fKPk zo|+m1)g!7`Nol;#4;{HZ(QzqJ6imz&X#y?^NfEz(1w}>t`!2k265jN>DNb$IruTQE zr>lFz+8aqjgIw-!<34ExmT$wI)T~bqGH}Z*#ugZeZ|3M_<#Qyy;NF#g8%Jd~oOu`& zi{-+#Pv!4uh^25@d^!7q&-ELL?3u{|Q&|4OFP5etCzHiZe#h-Q-q8g8%e;+^TRwv} zzNULj6Q;jSR~ozX&Wknsq_1|PYjXxKCA%8<(`*JdIXq?Ey***CYRg9k-gjUi9p?;OrIjy8@|zJIAcv2>*Dv^7&o z0l(!!2D!cY-WKbq{77wx#r@C~YZ<#T4hj#Rw7YOL*}cII;dCJLg8vp!E;Bv1S8qOi zTurFbpvUtiq}TJ`iNmt~@|*iR-Rp?WUTiA@xr<(cDvxS50}~$!-606IH+Q^ZcgMlB z!;iQh!usulq!(x5srN|!U5#$T$05O5^E4KTXrvi`u#@Jz7AuiW<@8j&J5A6z32fOk zb3c~N*4}yfaw+yTlgb+tJQH#XM!Q?vH`#Q!25T)l!JHn^1!r_}zf}1lF|jdfErzV! zQQwy~SH-i{jc66><#H@R80xJY7k3JGFE4G$RA0yUBr%Bv8Y~;EsMQ*>=E|nF>6gn@ ze3tL~Iy}Sc&RD|u&rBgg#HUDip$PWT27VdX-7O4bxe4;=iDydW2xRQ_`9fdJ~FIbA9vp&|Wa|~;>LreJ> zyb{!L>AYzOYD7=}oejT~*-}KZ`p}zg&WVuNT&)oAi2nLn6nj#%9Z${b??qpB`orO>V$Px zx-UI_fjcZ4@id;9XR`&Xmwy1=7~rHCr zdmPqdl)xy=ya>yz@-D}he9)G*RAP`}lmTn9dhM2a@}E#3f4hSOM?|i5Oo~4Vg5hRVlioLQMVCb8CHz)>wIdGVB>0bYGw{xPpA(7YoY5dMl zSsx1T_d7o=shP>5yN3U@=*>oG2#9tlJB`7s;8I}0=hIG!KF05V%fxQ0(M|lbP>&s? zudQ?)dU2n6&V=ft6X_2d@EvWkbF9Cj%-_MC-B`(kH=a91<-;4n#-w_JI0>qipjRlk*D?Y z2@*Ue?#Fi@bb;UG28q1eG}`;{Xl3FIDFJK2jVRN(dj7)oQW}1C5)>4aoZ>e@Q7tMz z({-Z7V6nse)=Av!8&3uGKo8+2TtOc{BeH59<#FS_S~8wjplpE!4dhaM$d*p{d0Xy9 z*J7P0wb8uCg-6mxs<@s*dNQ9oHdVO?vGFa}~Zh6tNK< zy!;W_lukuU7J{Ko_({V41;}fCaa2027JI@7TYiHl9PRvRfrtmot~Lzk$TxW#8JFBe zlHx`g%^cg%YL`3{0REHLV+zUfzGBU%&6m=?q!M5t=@0+9;+AQm0SZ({1=OPAv~Hen z-pyc>zO$R^!#Z_`8v1_4gOyIbLUckF8 zb7iQUkY1fm&?pB9-PrR^BXEVIWGb*W1Fv#I#&DuO#FO9Q9Xc**tmq)*m-Se(l=>kR z?mr68C=5_Fx1pek?hdz6pi z&GkI~jOJ;#JwdH^ICBe{NzYU>ouhco>h7{sVWh0O=A7mNGk2J{a}L0;hCne5yyN-M zSa+vjO^vNSD%06&Ds*)8!~yCVXS0yXFxGC2Af$6LqW*jJ_^q*fb}e-|sLYi`&2*!e z!8Tj*mwX8rt!4$J5VvSUYKp7A@$Pe=9)_%oxhQlyJLzOaV9JvHHxmpLu=!| zc`7`c>k?ggo8~J$d0TJcKe0#?5cPpf_S4OeS0Eg;-3uSV`f%5W^B-bT<5ImF{rw+# z33$I`(vgKoD8Cb3<6h&$A&`-f*mfF@P5dEzavlMSzJWUiLF{xK|G?&4{t3t6jECvP zDan^7VH;V_vF5^B;p_U}@3?L~x;$fI6ULZZDtkaR2?cz%oQ zwZ(rZ5*q(r>g;K}%~pK~uS9aS#j2sSv=s_(>W*a?FEXOFD_J#SPf0hw9WWpP=*mqF zhb~&6!>ZCBUzXG*25ZkoTgaK1nu;p^c=C$=kQ9a&}I6vSN2fODhH4qsU#0`vKXFA7jOOdeo554qI zDE6kJo1l{fI>?``T7NHaZ~ch7+-gt8`|_5i;9s$z?ZQBZ8Y2eA7YV7ps)KFqmyjs7 zLJ`mgRiFS-1uJ|fxJfo6r{p2$^+3Cm^P`aN9>N5K9HnQL8ak z8Qy|KE>7Y9Wrbu&_+b0*Z;)<6IXRZsmy)-xPR=VYI_osUr-35yIPefPYM~fk<=fHG zh5kdN5Pl8Ld#w*%!SUkGxQVR3yiXj|qINN?9^ZoIA$nIF(GUK0fYHB(ZmzB5>j(8X zaq1XM1%x~-%xRHu+n;^UhU}+XPg;)6Q>>hCN-{$@cA>AlXaiGMbvIoeU2dhoRj&@* zjUPTFObZOa#^>X6dDyZIN==xPq^h?5B7@p~p9Brg_9@oi(kc9qB&}4e63r3lEcwvM zPd~>u@4C`q-8gxtD2hQcoGZ|rvX!WLwI?JTNs2C+VR=>ol7{pHKQ-M??P0XKwT;#| z_SChriGvTuJ+94n4xLU>(qCkj#%h}>nfarKO}sm6Nu`bNOC|F(Yj34h^pSd^d4BL| zTwpsfb2n|ksu3CKGt~=S=MUJ}IAIOBH*7TPr>Et$n0Ukq&4bfd1f}`?Py7kd@G7P) zZ@G_Tjb*Qb^1{w1)1q4NsOVo0|8;V#pKu#_%=$i-dL#E@YrY zZ8-$ag$D|T-sS>6E2lTh_~Ro@UyU7dYH}AT2f!;Bi7-U|jQL7hxM}J5DmqB5yyD!* zh~qu{rx`qFO{4Ok=xEiNQEgEl@2b&uf&i=g2xkmd3}@BGGtvwEG=9rW@vYCi7k!G2 z4^}>q$XyI7Hnw;s_+4buit(dYuXe>&CZd=f+abqzy&X$lVm3Q=&BsFz0G}B2;jyy5 zJb4ESwNc?u{ky;K0Y5qJG|TC;-cGEw13W%UDSP?$rXMSkgfrdj;hHMt%3{V70qH8W zG?31<+bTEZ@3$dxMN)b| z>Zyi3!JL)a;@N=M%9WQiyiGqXa4?nMt46^_+kKhb74w~;r)2NCKg;|vX;2iAz(eFb zkRnVx=adO4ZhQvgMl1I(iO8eqNPTQTZi##Gkrh!3|A1PkIq_RzuwsKr*7O~DZTI|G z#80M=dSiW7Km#^#aj7UF1;YRtF~|kH5N?opsL4c90w}>YH{uaWP zGRDrpxcu4k%)^E%WxT7ZyrpdSVBBS{xhHEMw9vd@vY_*UI|!&>kAd7?t;2oL{ON)l zEBW7iSsupJFvIn_=`B7w`&INmpfVMy)h#7EaB22-^aJ#8!%{Sf*SK&^9Qu<{*wT>* z5H+#2w$|6)e;<{#VESwopGy(2`9JWT69h=Se#^3qMH7|&+qk3Pqh~z{gim0z?^iY7YC_~{^;%+ zLvj=E81s|RwXBnrJ!>E?X9i*Hm*$iMs=)MC_jEWF2jflXdh|% zS_gh!WB|@2ROq730%Aa4Fu^#}lRsiBMU8zGx|*rHV$PAqVX%{&rdj_=#Q&#L2~>cQ zI=woKV|i~wWZWI6)NKIV&0ahwbJ@!ZxkR}F%FG_BS1PGSILOTrf{@*ir`?sGE26JNj`Jnq0 zOqL2Sv_q$%dKf0M85>7Goq%}&>UuQ9IHZ)XLU!jC5fVE1F}H8YtD6(hZ8OpZCazYg zs8PCr_fCDu6Bbm9l1+z`t3OzQzuy3wV*B;M=;qJi!Bf@6*9!WYQl5Hm2LL-d9Reyf zvfM1wh{^hC{K^RdvGA&%iF7p!;1NEWeoe?3W*aCwNRttI4s4WU(K03+2?*y-l+uV` zc%7$(if+!Rns5d7O9MGGHv$X(j=@Bi;GjjdkaGlDglAz!ND2RhR$Fxu)5z5 z*eSpkMMMIA{b_|N(c$r#M5FD#KGNbyS`7`knnCjkweTvLO8YE)r)ZQ6F|Hu_>h|Js ziZ8j#+>c;G1U;ib?sC8Ank9r_^mz-8aeG&mcXx}Nl0^0?fS`YbqZI-^YWxOY_VC3t z2+G9fmWokjIRqQh0Ck8jea?PiIw6dpzda~T7Kk8#Z?-2zu<86@0Kh03gcv8gDY>~q z4Jv0EBZkOg((=HU=_K0f7>YoeRc+AZlCiOk4{j3u&&bPL`)ef+o(Nih9}@s(1x1Ck zrI;QsP#VSAO<0l|=jOEDi*EdHU1&Y@M@;Inr6YRut}wvLNA?bralmfmDT9MBX>b!lRv2tNDQ@-k*&iHe6aa#lmA!^6-2R9|QuG_L=v7Xiq9pvc zCqgUW;#EN0_1_gO77DcXzw1~WK|cTQBB2G_{O{_a5czK(#Qko65C{I>74kYl@&7g)sO4j;n4-G8=j)S_CoHg{P;D!#?+?j zw7RAguLlR|Vr!27cVa&2sUwte@L<9Jo&`7>>+I~b<0t>N8uvPDC4{$%h=}Ofi)XDJ z9TBNehgYcrD6m4@Ov7yb%S#VJVp-vRq*x$Hr)`bfjWK@Ok{`n9*JX72a=xa}YS;5t z?{H*V!ssrqpDg0N64#AP6%~68EgTBWSk~`!BOl5z>c1Xm&^n}HmQOIfbmW_9o+ARS zL>7fHs(kcl(VYtg{e!w#&wWQ6lX^@jtWSvbSMRG4NzZGJHS?LCiCR7k9~rj32L!{_kfs7K!vA7u&H zSHD3xs?N0@cC@A1E`rn5vkC8-ULG`oOHsVtTyWCBaZ8lG-DdU9;0keR*_Y15dJ-Oa zGNJ*vZ+N3X2zC60IY>K<1%FyNf0+bY`yT z*6qWgH3OLm8j?3ZppJ$fe9vunea0`=QY44|pZyoUy z6IQLKMG67xbP5P%kBXSTS{j}HI(khnpI#dn{((ERM<`owOu5Po95fOTM=R8Q@Vbc zIfh2a_ayd9lemcWIK54ye})ao`h8W;<~xjl%eu!4JJ-*hB$d|E@Z5DL4v3?Jl<$t& zF8|y>__vuuvBhNYiN%B~1VOm>;sOA2<=yFc-6WtuUlKg$*DS=pUVE^CMM!Mo7>Ba3 ziL~+#Un#D3)`^2VG z(mL1mw`&`F#3>w{myW&d0UQbMR1e=T4&gH3^?oZZbWwQdrA{0yI0&74AH`20<}z+nUU-$HK)VryEZ#~+1|H(@B`YPT#RE$_v+d*KXN8xYosDZ@ zc(or3)ME9YyjVO~5gHQoDj<8{ayl=Xokqo*J16GCYP20p?xPt_y<+GYwsE;kVrA;4 zqA6thHHS^sfIgPfME&eFG=)y3bg-(I0wF>Y^Erom^?p&jig1M=$>4)n6tlnCbmnth zv6pJtZX^_Dsrh*73I~ITb?8SOJ*Ij+{-{w6sN*Lv6 znF28|MUR&posRtRMqdYLoAPfAUm_gnt91j*((dBOT>t!?VAT1Zr1kw8p5!d`HaTPM zQ157T!fcfud=46EmGUuvXKxE1IgX7xfma<&7O+$4TSl3MNobKian;%0e;Z{cc^Brb zGsJq9-uXwHn0#7vXGU6;-z5Uvv`S!I$h`!HpsX=$W5`7`Y;@ZCX*q`UIK`$K~T%5F?t%B!acnJU` zHbz0oG5{5N03BUbQ4wJw&bp0fT*%1Jj?ey!G_id~>(}&3!CtdM;lN8(pyT>Mu|X-b zHgF8$ZS`zNv#iQ~iUPMLceMS5`F_AOF$vBG0&L<}>#(A29+z+Eh=$25xQ4Yh)v)G^ zI6qyj((Z?pL9fD}v@7Ek-d;^e@g4<}s7$KIYeQ)$fc*f^14O5Ac^^Kea=>Hb7Oi1f z-}rJ{Y25oV3ZK{eamx9APucs`PWfwZuGhYxVv7!6rw?^pAFn5n65Rb*CHu-^f*DE5 zZwB^-_7%V8NhNy!vrXqE!K#s?soQCi6g9QiX>>~#>=@t0EW{9jXVyA16<_z|k0xA< z#S9isJGnkMUCe?EP^8&feu#}SZap#K+M*@G7s#u5-ph}bQjNsLj3 zl*izIyETSbY8DClb$)X@c6~#vUJ-N*3rjhEf{Wfnu^-zsfq_{Pp^elr0JdJ9F zQgmqP{l!1UNK(%eX$=IjibhfO)I=t+wifNKGw_HEw58<^)xUaiv{4X1gn^Oen7PdR zwegW+x0-lJZnC z-LGrbG$xl0_9UmnVhMQI@hUewA^7HbH+GPb zoZNSN)<5lS%k-)FzRIi7Eyp=)MSt4PmNd(wWq4V-kjs6W^LfbgTc|zRTG-Fuf9;k6PJSa~Rx-k4bJ9u=uG^s_#@oGBp^;H*0HWpQBd%G_FU7 z@{TiQu&aQWaJtck1~Y+C;jIY?!Pi2`wyFie4GN|*;|=5F3Cq^hc+@GE-M6q~oaLlz zX)p#IP%HQ46BBx;@`uuDn*+8K`lVJjL!iIWsh<-IS7NmBiP_g$yTHa#JLcOacV;me z;_kue`1-X%`D!IMND(`sPpP`_U`4!IiwFJeXXjX&cQnP@?9?MPD~1h)E|&UZ&BUou zje_^7Zl;VoYniF2&v!a>i=*_w}bhz|FCo?bYK0HiDdusGp|FqVY^wdXQ3yu5Z zR@}|X`!guMuoCsi?zw39tLQ>Qw>M#7l@72zqdraoV%3U-r{jevTwL7B%-lf$Xh(E* zcdxAPnmdN`)ac&5;7>Y#Y4y~}WMyGbxJ#JTvQOodh>h|tF<27MjA!#}^Ys}@kuo6} zP2}-y1K*|7mL|lRWJau|G$_`*8u>?Dsua95@bAM;)L1QyhI-_pKQoon<2sp(dhjlZ z3?A`&a^qmxwEUI#5)tvjL9b2!dNwKR`o_W7#%|e+<>?!?#3=8}m_MtOH37Ss^9`jY z<}QE|c-^n^%Ev~>nRsCs+Ve~wdwCjo*#@KN;YMG8Bf(6N5KYvwE<3BmXa?`yP(M)I zIZJ;??-w+J2Yj2zs~@tt-a?L=Xki@1%TO}DDUu4ByE<*1MzM#|AZ#FrHDk$bxjw(u zQ%3`Mxe$bi&*@tmw&;k_`u6_7YWrS|ZTu;myVZtY)tIVD(dnu>R zlLreD{oYK>K6fmsM8rw8&a}nXPGqZ>n2=?xWjAke3~%*I8s)-xgIw8>vz6MU%MDq` zpFe*-nBJN+vg^N6T)&g>jUo+WFInEYXL2~~96dcb#XlkQY9Z>|3a+e!)*Vk4h*aTE zMa&01q{N-X=EK1<-roZ=I?bzDc2RD2x{NEc`E+78PxVM6)A>J1nA2r>Ckt-qd3}+R zVYxQ)>DEls5|{e-w~`zqnc-qrTv~cCiT+*BW@F6Y>I<|^8`p4xpEOQo0+_#rPK{g#U=sYUGGw>F3dGPXW!d>L2%uTU=S zX}krq`^3dAZ&d>5Gs@l@azvuz>aQ3~`|Q32Pe7X7?H8+BDR2KQb)F~6%#}U9p}~Ha zNKXoJ^S5W#pW#f;)5egk4bLjrDA69M*V0Hl8?B9_GtGGC{1+~80vkABA*wa+rZ!I~ ze9@Y+r@gMPM2R9pq<2|(hvd3+r(IPrmZ&#>11Mz7hnq5tS&aQ``El;a!Q|BB_C}^0 zgu`Vd3dSxleK1-3O!u08Fvp8WwZ`auEK1aG_pMM!XJ_}3YE%uwLKSzDSoolEmTL32 z#>s39h=EU9`=vn6Xss7VeT^rHZSQO4aD3E$ooW-tXha zE7h2f@LBXIThK_mulY2dzr-+DtAih2oIhDif%_Ql_|4-Vrwmf(_KCXitNqa_<2>w-%A6;0&EtzdXU<)zwV3cg`fH1G)$>DL z>!CUvV)HM?|DqB@_6TYz*~uc$8fuNXov$XG;%Jq~n?{gEGJ6%10>z7_p&>JCC2+lu zJxsnv8WSu|3pPYNURbgm80((rcQ3&LgXZkw8K?iorvrYwp<>}{SyOws)7c7$ z{N5>yp2iuPQb}+3+zx-nOuj+>`Xb|NN5C-u162akaqYlyK47T zexJZHhb6>c*YV~R8>Y{4hcl!z#EhrnN>wWzu%CYkX~PDl`A6rK7z3uk zRwOoEf%6(6(!p7uM-?m+PhGB^B*i6G7kjfnpkgJw!f34k4~Cr_XPQMkR7|{641s3I zW&#Ien5%oCBWu%#N1Jq?{6#`0$GrC#INNoNP9ZrtN6cC}v2x+K=UO69^i5lvC5ULg zTYoOVy!_XP;ah7E?fNk0s26L7hLAIorEi^>uzKC^N?p8+;zVs-`D$M7JZbFCH=E?t zQ2V#$6YFiG7pTI7TI#&reO5be)n^d`gTo0FE`hUt{P&!RgWq*HGq$E!@rJQ?Gwi;p zs+i#2mL@hV+p4~kXJ%GnP`<9;AezdjIq>F)iDvCOnLf2!eU6s%wTar%Y8oAnW@Bez zc$Q+N${800l|6;+?)>}*-+<~R3R!oQ?RvW#KqIdbUS;riuCQd#&y*+b`P|n*Z#j7T zJYn$V+@V>O2iKLbi;|?RfdB`kcgo3rY1$_3srU?YWmq64%;O6`*XDA|F%}0KwM^ji zj6S`2^25B^et-GNoI`#2$)a`+)207iW)PRd5)l9f!ric(4FT|Pt~-C0GwXU@nqHtD z=o_yP>L#6@xUtDGLV(IXNxFvuMo7G#@MV{sbA`|KLzjb=seFT8&Rv3s`NurXFtdky z8B^CcH<5vIS(CcAjh>edYhVPu&<*WD1>f%kaxKTQ$Xq7=NE9y^4-K+d57C`Z)e4p! z;Qh23fAXnlp*&ENBMm_dtCr@n8^&bP3adL4hymR62x_F=*V>XWEm%@qm5NgpvkY|*Vb6RtV zyzCK4^wve!_8PCX0UohD(kll6ki$Qrfd(W*Df9h7&Nw_N zbAu0|S(VOL2Ox8uEFSFjw@1()*`6$pkvz)R$)|=Q+uJf+87HS<&-F?pJQ2D%Jlp&fgfO675R9Xt+C{F8Ztpr< z_>~)0=91nnzo&0)^L22CUgqmBDJJc0mO0u1+YG8L-w3klkQ97@Y6NPior7JrQJQF#_wIA##=Fh4D-TPI z>5ksYT>P7z*TxK&{-2>I^Ajt0m{5Gg-~e2ZMn2wNyYV?ZnaIdhbhNJC9b!!pwdE?=`?F%(bxqfEw>Nd8N{B-q#uy6QVUbUxpT%M2qZuEh&y%yxsqPzKrnRmQ}9`vrKmN(=MlZ-Hc~%JB_G*nA)zL9-0K$;2+tfI#k=e|~U?G1d#*A&bcpn+%WopIhu2wsb|Rpq1O% zIa9mneoz@o+DKP!t+Q0E_4d5#rwo+EmsGO+2%u|K-u#byVx337ESIo6Z)YIQ_k-H1 z5Rc)+kVpI-V}t&H-hj+}H0@e#@i9KU#9Tc@)itfHHB;;6S2MrqlC^t#`T>#Y!EMKn zpPuxMG08PK2Aqo*YH#jJc`m02;Li z|5c}l*7c!d6V3ivzq)WmzJ?mbAa12w<)|AK;c=eA6YITdy4o&w8#qH2>zv}CM2*4h z(I$_LxG+mz;&80k-G!Jk8~t=7pTK)&?k+Er%J~#3mj>@S3>z2O=FPi>G>$;8{7KLh z+?FJk{R{4PE-g_}o@t)A2SU>|Us87H>}6#C?wIfMVJh{PmvAw49fJpfT^`*@)7!(r zdjf_W#J(mam8DpR+o3G?W*c-)4@}XZHcap`*r@Vp&Rge?*p-sZ zHZ^Ks(ZCQamDap&f5BV zv!h3903F#xc}q@yoOg0@-n6DiW4{C79`upFodYxtv!0wBgB^#5vSIp}a4dA{h|>0X z;-g)u2Cr=Qsj#;RVN^qcEmh}3f4r<9^*VMLF24L`@Tf<|n&nK8H%LSfazEl$Lc{c` zK4diHb7ag`^^SXMvf=rP31Q$C{i<(>-jnyA{7)hAt!`POtHs_y=OW#?X$CB1q@2%QKdC`(7~9{8Y7&oO-MhS-LVcry>{gB^ z&q(;VQ!hj8@h{f%Z&FseO`~{-!4#l>KK)52jv;=J5#yl|15 z%JGC4!;tzOJsN9HZ@YJx9oh_g_up9P_?N;kdT->CNg$Egd$OI)3X=X+vr$Opt;zH=oaj}p3n^J2DK2S@&_1PBMv-xIQP~LOUnqG`k+ocyJV@Q$e)ULE5(m|pQSD;_ z)U@rb{bTo~wWV(vQ9Kanw2SR|_?g={h;g2V56YFT|3)Xwwd z()~{$5x+&Ho@8&zm`z4(mQh;`U$=>#j?z+Qrhbs2N|@jIl!sH@3EY!7L91DdH1Bbv z7p$4143GO-H$&-^1!Xtj2jLlba&JL{;(_fv{!9PI>CG^T|16F1TIJVQ+g~mhJ$ZG> zZJ!nKe~=2u$7*%hEcEuB*N zxCi$stsfqn^03mpa&U61PM{A;V|aWY@wLaoN$f`NmSoVx_~uAGfo}&i*fhu|VUJe8VW_K=FOn;Y zR{eviN5LSs^G14^nqbT*-i|q)v$(Spkw@kd^KKLGa*5pl@-8fLrL~#9rWDrvc-M4M z=q-{5Ml{(J4PH}4k-OX0%tbk#swh&2??kJo?uYYq5@}!U){ndWu9Q+V5fsdv#qe2mhl!9%(pom zxOYoxu8tNt@c(YapU%cMlfdYQIU1Ov5Jn-9c+$wHMO|0V2?)f3zF!d!XvZ^$!-5z4 z#5?%fn|12G87T8S69%vYn-t~Ytpm*}T{#mQrb!1jdOUTxce1*pJOUkQ4I6wsG8sJG ze_s;jea%FEaGT3SM0==0MlS-sSw#59I6is`F8q|#yskxa~zTuoLYT9S@ zna25k(>lMbi_5lhNqA|3O5k2Uhc)Hyu3Ea+@^e5x9h_yy^Kg6skM-Qk+|(!IY@(vI zH$S#Sdl*z|e7)%cD=!yZv8XD4bR)rd?@M#OKpdj73lH$J|7Ss~cN!4OgYCZhDrK^m zoS$)G^Ic$r;}Nn~gOlz~Alu`OKA|f*Hd|-s^~L)pPmXoP`@}1X6`$*{&msr&{r9)i zw(gFe$tDjMEW`H)YYh|o_w%#R*H`c$LNSbhGPij;Fx{md8^pCjyfH8KajN`bQ@-Bk5DD&ex%4uO_$VZ#aio+Kb+QCDHcwnQE@A1(JIr@Qi5-LvYxBI| z&J0iIjz93edSXIDb3U9V`71*w$Ka>+l;Fd#K#MgYql}E_NF@y)kE1yJe6L7tKm!`H zezu)I70pL^2)D1UrT_L2SRZUi4~V?x(khdkuIf|wjYr%qGboiEK4Y^34<{;5FWdq+ zbZB}F@BiPk0M^|$;8~;1O-EV#?-?&v7Vr5)br4TI^m-G$$91p~3+uSini(%g+nf9g z>+aus*qwJ$RhlMjJvw+y9D{`w`s?k~e28CYxwA`VNrSUXQGmc~b;Cq^>wfj41`VVy zpST0zr2g+-{0{r@ZsUZPx2MTu$nD^X-g=tDLe_TTJ?ly1wG`=e_!7z4pIE*N6|JVT zxk3UQ)KXrz`!>j1{)Uxa=6+RU2A!cLhmFL=CBK8-*voWoSSvd5l=IR zSoz-KCxXemPnMcQJB^LaGu{iqm~%(R5?6=Q&{(#y96dwb)?laum(x{6flb=mXDx{r zOL}v4?wAi8rGKrk@$mY`#4g-iFK|OUOAq9Th;d-^DU-n=k=MVYP+Y9-ABJ8o>l^&_ z@|5!xP#MZtx{4LkAzqoN?w@OM@ZZdP*9==~;;D1qB5JALk>H4G%f}*yoTXI#`Pxm= z=z0FV!=pvh3Ky54y*ljS_;@-Ntvl3}bZD?J;n3^uxS@sg;PW>(+YRcF=g-Z~uU41% zUcZ33T-@X@w$^HL`2_y*Ep55N#3SrLh!py|^4sH2uO2VasHmujov59?2$ z0ff$Q?L`aFlR{Yq>4@;M>ME(LvfA=Yk@y$?nN%r{y$881yCHpwltXKl}-Vs_uDC-kAr4S-%~xbS~)GKz+}d4e*ZXzXE;UI z9KH05*j$Nz=OqV4(#H)K98|1lWt&S*$2mD$tZ8@a%-4ROq%$@vI!>(>A?9JZue>wZ z%ar4O$Z6NrEG&PMOw>L!g-bf^za}U02UjloU05>I!(_xbyCAMB8{Pc)sA83t@ICGE zad)j#B>3kOE_?OdN`{%;dfdWf96;l-)QTg)E-xB$-na63P0mc^DgjKi?WDir&A!9# z>yxvUTS0MizT;c7-R8qWI6N$Z$?$YxX1h0b*KZ>0(J>2!2I}5cH*xhBBX)*w)DcT% z>U9shlCOb3tC?Byqcd+4uBii#dTf2nmD z$1BL813w1d$Fa1eCE!5xkE>=ZV$dI-x;cP>h9T}MB+JTR%{3cIrrL<~lFzw5(D!3% zlqcLT*X+IXdibN)NLyAcH+D>zQpE%zq=M~F^eUmZ!eL+sf~n0nAy#7p|yS&1Wm>(U=Q|cUGQ2tfDP#< z*RL4(VQCsrk|sL5aM9a)I8_V&<3b3F@6Au8TrER=B4?A!1s<}k!EmUbFR@I8udw6c z+$MFG&8YT(S!RrubN{Swe@;4;&*2kGN_oE6S_m-ri(+s@Ud{<-|EwVEuElD%jNu7H zaofr=)m4-F2|@vr5VG~PKMQx^ki#9`Wi=fINM74q0K!wi0AwjvyM?`s&JA!-r?Z_V zAp(M|DZ`q-a?LN^R=?1$e#BfeD~?72-q3(}2dS62$o$&A}Q4+oEo_rq;O#PyES=&{-idVq$HKTG#Lu5 zjFJGCdP><3!#_GW))s~#GSt02oD3n)9*ol*-K4Q!lgUjrQg>DFDuB77kc6_Vb&ULlRa|FF&|&ShR;A=phPl;0Rf7_%m{Hbs4ToUw$Y|wx`ESiglh3x1f(M92^T{@j z3lfs0HgiR(_zAi5^H(k>awAMy1Luo%QZh0oQ;>JhdRMXZOc7NqVV6De4Ttq_tC^x( zyM>tID!r=*fp`1cR}XL3)9&vAZ$k78Z?Bv#d$9Pp9Ad({S@Q@ujJAIv)Xo>L)`ntu zVhJG)Y-~ZgvC>jfc}M=>?U#3j`YVkZ`(q=8860U%zI+^n{KgMIKZA*gi7PjWBb^I) zi7l9pQK}X|A5`j_ZRd&Vw0b1S)|nE=ON7OQZ;j5CeF_e_X6M8b=i$jSGNQ^95GZp! z-@Q$unWa>xS6$Wa=)j&SM*1z|JbO4D+$1P?Yd!@r0P&gA;Rq$rY;3zf5Q-Lu2gCm| z40>FrWB6jG7!?IW44j4rg@wz82lW^%!GGFc?28SZ;<^7X>}N_;nlxf?=J9?LT86kK zbu4?g}fpDf^(UMmdS+2M($kDy)d$? zcKTE5=EA0c_=fhJij|jXvv|;bAcZjir$lA;^~2hs#SJt9;%@3g%E^IfyU-8WI`r2N)0N2ssTG`Rg^ zT7&m@G+_}@VBH79p9_?zvb>Olw8f$P6_xoUWn{X$2T9+oqGL&bY#p2x>9YW66n#H; zR_QZ44RPz>vO^N(G|)+Ncuo~~!tBf9j}*NTUZ`8kz>lzB^p1YhALi@uK1WNLpjk;x z*W5f*f&kOz@fubsPq%D`RkdO=S>K@2C*-LP%}{69Vsge?dqI-q>NGeLs4vwH_{>5v z*=*8gSISTeWc?qx#Ma#QU&If#sU8-YbJ!|Q)D34!`)+ZW*$X?88xf`1J#s6vr>gjf zo4oGkQ$nkM!wv($+t3srO;3Ox{ zOU&(J;b#14f%04UPDX)d+~>`9;lTg#v%bXKv38rxKtT5I`tVc@8ZK z@xH#+#AjP!qoEk_julAZ_0+P~n2a84V>m&?s(Z5UI36!U%xaxsx#25j&`!Vy2?{pf z9SIw&psh=%Y@3^&GfEXe60goWIU!h;Z_as3wfnjdbd_#ZAlE+<8;P3nzy1EY&r!9s!(8dx96BC^S? z)Oa8n5-Lurkm|u;db^L;?tbb?3`Y(t-RQyToc=|LqFc#bXG zY@vC}#l{_FRWOXPqc}_lZ>#xr;D z8%03Lv zx3@!9wnI-d14?8(z|5mY!SBKR*(=lnxK@N#Zm^v0 zU&6NXg1Xk!e80S!o1Lp^WJm>e%-QO}<|k)jidUb=zPbhJX7jsuCxjw#)?ero@Op%0 zCnL%|ol%`>L@5R9QvTW?9wUt;g6@$QsjQ@bSy`U`vvg(Scf7bea_aR*ps7)IM5)DE z&|=0m(o2QjNdI<&CbkR=CheRPVJZ349T!g~aJ>U2Hf-}U0)5f1ao6}Pn@>iw70t}c zYnjbjLPrSVq@x`RSY_t-_nTGAzWCP!fMeRN8{Y&#vQ{cW>e|?(x3x~M0@}qL3No@9 zhYkMI@`@6TJdxomAiAA4Z!Od=R)?B>^(%1+S&hWf-%83#n1DzxO{BilefX=OPgtTd zyblRQxpgSfheD>1Fmi>*&#T(zhe+7J-cL?8btOpQ<|gLhS$lFgr7{%rtXgVdTJG)0 zkR-*##9H7xS9;3ji#PF=WO91)d#*w?(1VqxPz5%$Z~+zk$Mm@Fizp=J=099%Wm8U} zGg_F+FZqQpYdI&Rug+nTS$;os8jtBuj~v15k(=hNbGni0VRg(}k^Q6>Br`KJanr}t zq9dpD>-t>~XuBf*iO5|0rlqpgji#99@*#@p;~l{sa&2rocAx^Bk$MT?;pW=*egwk!vg%7(>CYp+3-CPzl(F6 z@okZzz>u5nf9kw@sjSd|+;}!J{gJePK`2dQ1#}kBu-+NJP9GTZl)En?nt$C}(v4y3JRSP^Kr{IqF{y7oQwwq8id$sLc@5yl^Nc2A>6K7|A; z6uDbSrBF?5#+*^9RYG{c!>i!$s{?j|>y?#+jrDA;Xr&yl`$ys?dj0L`B^DVSo*Z>M zJ>|cJXsXhQ`D#Z}SVsq^R;Zr@AEP>piqh4Cl|DxR*3XUa>f-Gd;06g3VKY3T(~TPu zz{qRx%?t@&-5lt@UtWI#;w`n_o#_e&#@xdD3BmX1VBq)B43&0053xzT{5+l3zd-CY{DIdYa*)bE34hd~qOVGKH{HK`=?jXz(LJ!bSFJ+H4#^d6s; z`j^sb)Z!snqva$i$6-OtRB&w0yJhnEQc@8 z*V1YXKim2||1PVLy<;{p*U0}ucRVkut&AEh^mk^VN-<@`!PV_OFLtq+GrQ9_>sSNi zt=~C$wAjp~HxRqy^UMkyoXckq)(!$9mDK?3cR4-^4#D^s6w)*v)aFo`I(AG#691Cp z;wO7>YOq3oE&n;cQ#EAAf+)SqvND#dpW^(Scz3)nz#|7P?o&vq(CFNlgL^w$KfjEMvnzc?(p~nOFN;>?)RVG zvw4S=CM^VSPURmwrBbSTKv@|c_7A4R%rO%k?$4*1vI^lBc_*Yb^^@!iw>i6*|3vXWd08B!)<@Fg=p21BF7ow5?Ym0_+lnTP~w7!V#0 z-Plde14+VOYWnt1*i~=uSbT~OF4reo*GAXbKdPY~kY5zqb^7bVe91qb!YOulBA?qN z)#(ch3voIaFi;@k8;|dIQCS{H+~C|f&}plVJ|cf2F0PxJRuP!rYPvR+E^3-V^&3#0 znVT|rDD8Y@`jIgQ%5TzR{d1GD+bK8Js-T(yj|(-Pg|_Ev!s4XwKN`CGQU5h$@WeHqi{ z1Bz@3htY2DusA;+#;~)yb#ZZ!@62xA-8PE(469Hjl~mtlmqT|ZX_aY>0;eMjdL$}S z&c_X4|1LIBbPOi$Ki=I+8_|bI3GZ%p#RkWfnpTJobY`pfPmUue+6!GaxYoElP+$OX zgLy|qu-Y<|5c*Kfpc6C*RMOd;K_>jT5y{ljZMJXbK&$9$r#D?MW`TINCRirN*8)Tk z005O7tp9wL)l%w`lm@H;hcoM0BL{M*D0pGm0r3btcgMO>q43vvYbRsr}X72aDLO|DYwRtWZfhNsYcd=e^ zGVGPvwaIa3qTLNHu%VtvsZH$i^ULDPgc$AhTNtxI&&2s);mm7?8EOp-OufZSd~SBO zhVj;fyPT1^vB(W#mc@E5OHB0GLRo<79^>Ehnp{rKJVA@QE^E14TZ6{To{iQ+<4-ppE~i*D0?@~ z7K*s;+b-?zvpEHy4EvgQ_U`%y0t$yImPX#6D74Rm^$m#Flar&4=OTB?)Ni&A6+N3Q z`|1{|mb^E?jx8-yv2gNAa{vC(>*#zuJUo}1NTy@K6DLoauKXb!lPwSnd>shcd;s{O zs(%(uBXn3mY{SIAZ#I4wj_ill#+Bk`zU& z{-Pf!;TId8b}rg5DoI?_0210NqbiNdb} zUzpv7|ESz~u(CA}Q&Eux)Og$}^&^~*oM28UEf4NG0tFQ-YlG66^>vG7!-;aKx5+)I zE?%^b9DoQD9OU`c{VKE%@e%#E1&Du?)h#G#AJ}6p;Iqpf@(P^;{Nx#F% zt;`*Ih%2LdzZVe`F-ycx5v|=N3AY~n zscygSVhxm~jrCwg2t?g;k#CUx%+A7u1_7W z+SL~CU)j(2{*5&;*b~l2mp#~(aI*>`2g#>d{3**yV=Sl%kZ|_S5ahGE5ob`3#%gor!f$!%YC)dXaG3xLfw^TFEgcjqOEUgDQb7rMb;DPQ%YE}bT<7HS~3y}L=`$T zjBdl`7_V#rjkc=p&KK-~V)Fg7qY(JIeB2LD_<+djaJ3#I@<8SXDC!Ep7ja8u#5iIF zzz@0XvA}fG=YE_gPSM_E#)uD?>To(`o)ZLirMt&TkdErLf&)F~2gcgE2YchSkQ@c- zfQl|dGno(C($HFBwA-6cTm$MfJDJ{no26P)YyfPn8Fk`D4ZSxqNJ;*v(yJxwI$yHW zwKa$nMl|{Tz5t*2x_Oads|_V?_Wa5e0f53&m5OW_6WOEsJ0X)(9pqYC+%h2JgXF0C zv-y;1^wMhcdD3S$P_#+*4ECEB5;^=)sEaSpGdxb3sDP-DoZDrOq~{oxT&yqTlE*RL z98R-I=Zt=o@DGY)+vV}mz7gyOZ;X|gixKZ)p!ZBOE6=TTwM-S)P54T1=fbj~dwykB z#eHk7FA8|wHnl60;IvIU#nG+#%E(jd%uh8eU5u-Vk&k9i!LD}^HHj|$3_(xar5Mt-&_j; z=*~u`(!curHXikhb(6jr{82n!&XLNE4pZ5FI*kwR0SK7izZ=-TDk=hD{K-aER%^Bl zs(VncW6|%KmG1p9u*TDJ#=ejmy>212B4KBLz>Q6QgPdkw^Bx-zK9K$;_>Nr9ff*@xA; zxy6Iifr%4AEPnCU8vTc2FPe7iem&$sx5@FFXn0LS4#I3YyclAbo79rhEZkbWCXb3bgfrlC#ZF=Mu)o|O=D&*UA`}jhK`t=I|Ar1nVnztRZ4*^Z;Uqt#ElX*Vj$a^>ih;wfi90qwGt)k1wPNe z!*SsA!FHi$+Y2RbGQ;lal`vosP@Mc5uPMEtJWxS}&BV(7BWuCo{;e(zScrWR+Or%$k$i3+Ey_1gdZkGcV?h#Tock?H5^37)oXu2 zBWoT&{M27;#g~Se&uQH$&tZO?ICB~+88NVL^Z2x^p0##UrgA9KW`hqT(83o&BwqeN3Qf@EL zLK>gWRY%;>^w#W!efoXOJuo?TeJulVJE8)|{`gEX`RH;aMe&0$byLsHW zt+BoWrSy#Vfs8JGUitvM&{f$x=uyp9&ZfEUNTcY*Lf{7xL12gyLv+`I(6bx?&uAu$ z(GsiJs06x7Z;S33ydPmJR|9xW!X|?>d4jFAM)9AMY}GdzmJRLm`J;z9UJnBIm(?T+ zQI;9`WD1_n6rtSBq%Ox5eq6Xn=VIdl3ertnWdC#>(44M@W9>tHd{9tuZgKvN=-(;% zU~v&-90EB4U0wLwyE@uzx2>%pQ4xbg3M9{*j)YPv0nVk|0A`C^B-T&#q=a^G<>G1e z=5gNEB$J5EAvSPkpBowzV)OqB;coaukF_0{FZSAa~4f&PDPB8<7dZwgv5gI z*2qzXGa;T1q)cE?qfEe3_OBVT5d&8ah{(|quGr;NgXrko$qq9fkT+I%nAhR;`qj`D zqx2{2`r*B90O@e8kttnE>y+PvPrES~;}h~wjQralWASjgyGxH-?lGH$;9F6V_TpwA ze(!mLQE#)KmxN));*}he^TWoc9m=RO^{lFmxxEY|+~nEAvY>A{Eq_aay@XuetKN2- z&HgEsplM4>Thq~XwchK<(htl}Ul+?dgB>0n8dzC8&51zycUICjCMRpAAA*^jA!|#I z;3XXnLPRT$A7V`mkb-xcCQAN%rZkiGF2x)cff~y36X(J$5au>w71r!ELL)@Q2P+GB z^M@KHGiLrxGK`Jm#Bbq-XYy?a=7>eOWtaTCZh#E}RD7I4Ym6V8yE8d8Ral{op1wdW zT$f&Gv86=-lt}uK3lSorM6bKss3^_$`}L*0k4HK!?%9vdmQ64}iHvI`Vk0a6R|{ad ztYw)XclM}E%I%9xC(GtqaXPYx%h1|rt5h`OcP4&1N4Cpuk|01RxwA@~7#{6NX=Vx* z6zAEd?eFxxhr-9FQs~-DICaXQxo;7j} z+Zn_7->I~<;!3B{l}Jo^0*K)=V_jT!QE_!zB=jKtzs>Q1HExc1{#hNR!cSC>Pn|zb zs|8f>Lx5D5P?N6H=(9kLM5j$-q&jddUDLp9+WXV?7b2uug>!|@<6*oX;iIn|;kPB? zU3Nlhp^zen^IoPm))*gZf<5-SBiV*0g|9Un7MQ#~F}3bTp;WB7-d%N+DHHIYEh+q7 zgPA2~P)&ry9z~Cik7f!~Lr2|6p@m{t?QhYv*h)thsrwL2164}pjxMU9)(A`PGHppyTJ zR(~DT7NHSg92&QB`Z+s~RzVAtuu~C0PkG|H`h3bn^T_$qgOQ5#>Gx^{8QqHF`80EM z>>Ur77C4ClIf;8>;}{;}hb z;@LObiRf~$n@{#RIG=8PxcE~N9v=0R=+D0(Pc|LWJ4;ThQtr6b?uK;K%DD*hLyJEb zpzMM0bbKLORH)|sRrEcGOCVps(4xh-BK{u%J(s*#cuN`B2!iZ8QM^;E5Fq;nRE63VABnIuS{vu>pen zP%80e@`O$=AxP@<*$3NCb?x4us0j2j#g-x(0+M1wW9!mm;w;|#v;Ft9Ov*&>(=Yb^vnzf5zh$CSq(O0H15FfMl}M%eIv7UqyuxW{CH~A; ziqu>og_NkYM}VIBBj_9fOx97=O0m+C?hE)|LGaaf?`&L5kjZ@j46nOA(i^V_<+c($ zj~CP_EVqIF;8z-4ZQ2{n5^irWfRgrjyZ$_p&A-2tAkORY8usF5>u@n{b9;B^;7l!5 zQj3nwWy3E=-xojRv@tX1_I2x&(=AGwgZ}z>j0OVY^<(-Kj_$Pb^cp zJ|C<)zIvyi&B3?zf%Da|KOGXT&^>;|AFYN_yXgCCNqD?xOD~EQ3OA3BHu9nMQuiZ{xI{+ork1EW~ z4G6c;lYj^p`B_-V{KL?~B3OldfHw!7{Ry8^s}Z)p$W&v$uAeujq0~+>hRvhb&^uee z@U2r>V`*32g_A2grji~}j-HdVWwKtD;@(DAexXEIgT)ik^~qWVKl{=OXRS_m&K{Ly z4FFDcI-Y)F!y!5DPaCa&2{E!PmxeL3;%+JxX)7{O{xzdXFi?Rhlj9dxmm<)wINcXQ zUqL-pB$;L%-uq7?g+HrQ64dc(OsBw}e-A8f(C@ky{PaFoVlB?1A4B0tnx+$zBtHW{nQ5#aCZe;WU|L*K2#Xj#C`_ zr+$_wkQI`ywtAjZnN;e*90eoiPpc+9Mxy|Z8+C#KD^-EfYg3!^y^ClwtOPvql=+gK zRCeHjMcN)wd}Yr+Ow;eUe*f*z)ogwiDfvYWSeq@T zwBHJl7Xa$a;feWmd>j8H_3l9|{&27$#D+W1$YAj31XBe+;_}2oAK=aWNFcMUu}`1R z)IUy*#?-CAWM?{+f`U~h+NDIjH%&ELm-5I^c5c5eEGlAb*IYk*XP&0t7+D+`)Kd10 ztU?EG1LNPq`P_X4Pf@bPrnA5*_i3vu@>3tas8klvmg;g7*xYo{ zlPg`!vJp?@EkvDki+9te-H1M!);XC0+zCUAY*#vx@K2ugWkR_>O=Yw!s!P$KeH~mLjN+8He9Ou%h3T(q!g-yfPOkS)^-|!S(p5Cex}l*r93j<5ctn1XAE5%!%U+^NZVf!BYjLVB^vD5w&-Zq z%>Ow$Fcxv_3)h8b>!uICd!Ofw*6!gRpNl(0$f#h{Y`I4QDL_$&`tLkm@2;h-zdSqn^dt}#8nEV{*M49J42+#y7k zGtB!*F#&nS@0&poxZrpK!j(J2`WfRoFUw7JWH>hab(qujv@2N*@1)shBPHP-etep9 zTB*tND}?Ymhr~tut2WOsslo`6U?J*+h}SSc4}}g(iXv&Kr1Yy)_+uUW&m>6!hMH~l zBx$hI5H05rh%7SQ^0bayXsgDo-U}NnQ}X9Tx*sD!f1@q5x;`x_vfdftg=)6FwKF3s z`wS-yP5|%J(#1^7Rc=dc#N*x+yMZzi@U?JoE&vr?)b-w2#u+B@t{$Mj7&NK<{TTE~ zHT2v6gsjlHA#sW%6U*EPNd0<93ccaSOx}5EC(@9^t%3_yNov}0er_dsAQoKr#m&tI zudJ{v{lfzSw8GsTPy*Y12SW+pZ7ALOZx{gsx!_Pcm?7Q{J^nWnPE(>(l~RMQ!f7M( zG%P&2!}sp~%w1JKDy}WF&6kgno*$I_f4@n!eP}>XaZD6+%f+SfJGWAg4ftmEX;|b+ z`}_!v&KxGmF1?lOMa*ZAq6^+foL#cuu>7GIKsq10ckrpy8JyigI$QhuFZZpM&lafV zCODHAOp1#>4v0==Ea)sFbX7m@Bi}GZf|Mh$;C`=FPKjJQ01yNT*&V(WPIwSMVK=m_ipg+ANB|awzedI(Lm*TdiIB${l zl5H%mV<^;9A23tgrGEDrrbW2wYUd3V1+^H`p)^MHu?66M0br^WQ5e~m3ZS?PnA3pF z;Q0ISkA!My+{s$2YZ`!$a``=LwA#qw-V{;db~2TJG>-@-Hqm`s(ZZ zh1>&Rw#$wgKW(L;;GhKM)_mk8e_-&Y+hj?kmX4Lz*Mq*kwV_~NU3HWIjKE}}CO&{f zD0ci|!&Re|;o`WDU8jBD&~H|!k@J93w$<$tG6uFQ40$N7P2eK4xiwd9f;f=Fm-UGH z#`Hl^DCRfQ0spoyJB(Vmj|N-4ZyA34>!V1C##+8Rnsj3Z#_3>AIA)ogR}f7wr&F%M z>*ZKVdK)hMVy@GG^>Mt7Kim@p0t$CFZzXHjEu#SI3V*T!bmexcESSUxVoAF|schDhtW_a|e(~e_l$%z`94qhu4($4&JSYhE5gR7?t0w zIP8z(g19`x#UE^0yr#nv*NzD2#|Q#4iS|2R9}$@IHS26(?~uLZT$juIWUTKjzwR~yZN)h-T)Ckto-;Q>aE zM=->T70KLh^ze9GFo0&`+GZV%PBX*`^WDUJBc}f1{BO&P-ILde@BPJU&$P!3_=oOP z_}Tfl%Yj~I$BFm@NSL$QtF6@TjDdLDhZ7LAJZ`<+<1t1F<`?~WzWzkabBKLtVO7Qr zP3-gRsRiii%Zgpjy%O-jFpUh2wApbPiT0=IY9vE0jdWXG;2wP6z8P(AFN0yK;>TWE zF*33AY`g~pkrSjZ1&H(DWdT)#Sf$(cBtHrYDi)IoB6q9jhv4;fR1}~+s0w9g=Cc4r z4;moNYxRH^qfWTGO-Ms699GRUstRDWnF*haBa*{g9=MdOt6|Xj1TZ4N$BK+wBCNF< z5`%)=8!PnXy8aqbs&+Cvw%{OSG!a5DSqDB>gCQ{A9|;27k=E8M8A<;M^-jAKt3tUWol*i zUeiSp3uaHP#XQG3-Y4y__lgbi`UdQo-(8{+K6@aYxb<6oIir}**FL4WnG=xBOLN`x z^G01(y9IsveEE7MjF+jS#MeJn_(`21^Nsj{l9Lu~hc`s?C3-OTOHAs&sG7hu77tSN zvbi4KoICZEfqs!X`+d^;^FHB9t?=`9Xf#^$F64rXV&^+*tejc0vgnC z?xIL8C?E}POy(9EFRUn(ld<+rPv8V8*Fq9ev%3=3)C8wkuX!53t#{m$c-<#*R4GH1 z+;+)ee08J&TU{F3-a%(PWzQ!riCMfGHnx7&Nu{BPn}CQSZ2bpo-huM-Uu@m1Z8!{9WP z22H#KrFscYw08abc4lV)G0>d;3kqT&NYMQIBe#69E!al+bK<43rWf-V`Q;*(_ z03=feZ$76F*VXTC9W<3shdGfAOyqSmDz%^D4;Rdy21v7Pvkp{>?1j3328mcOfJ#3! z|#B(BDk%0^c)FCGMZcR+{9JlatRAFc^TA>RhL+HHmuYSfyMd_w|X1 zow2)TR%LF1VVB4^AmCG%*Ff*V1eYg>7iuCqdO<(CwKe#B_QS<4X?nHyxjXIRx z$1h&qOWdr08GC^qB<-TUCldukQ>7lKbi5%MQ(b+}W!p7ZcdWu`!2M?TA? zNnW0Pyy0V~uVh+;s&Y&gXMOC9IFdrC4p{^m&5(l~10WewBJes{E`0Zv=O^WI&F1uv z_TV?vkT$I-Ax+lDZR zNU|=pfT*T>a#LjhO%nF?Cj{um#kcIZ(w5bFJ>-q2soLoV$Jc6sZOpYe#?uLO%k%|4 z=R06Pq8g~MzGktxCB%+F1*H2qM#WP#W{%NuuDandG27jh747cNSb$6!4=)Rw(VN$a zJlgoqP`-55*#xjTe7FkeeZIV1S_wEFv67^`iaJ}vD}}S}d0nhlx=P`u$%M2_E$CMP zDxV*}i*vTzo9NfZtOrZI6&VWXcu##2b>!8BXU_(6dq@l?=anKSB8;KHIy*b{{4osx zGG~TB{e0B{DK?5VHY#Qw5dm&y*!sPvs+Und+uTNWOo9&|UtlxGKQ0cTTUm90?De^n z?N4_jf;1L~Pa04i2j#32SNTyY7qU7w5MHwi4F)YEtZK5`aRbJU$&Pmgi1laS0U~JW zO()km&n)ai(l?E|soa*xfM`Wy1;b$F86Ci{Qjg%&N#Z3A*ZrUYS*e&nEO-okAE(P~ zM*g~&M!PR2SgWT801yDCBT1i!jYEK+)6G~gnZ5+ETCHJka4LL14n?Z` z^2Deu$z`&0rlrvhFl@x}MYn#O)Q_cQP*Rnk(b4cF&7; zIK`b_kt+I_RBFE<+Zs^_?&xdtk#~{ z@^Y|F=ZWwyAt5Ct6gPJK5$}T>2V3IdGZRtw1=;rLVF zW^ah+msAoIk3tsR=jRmic6h(sQyL_eGH4B^3HA0sBfu%UUSG3+vbYWn$!nYL;sK>jFBb+10WKka264O<$u}bM7(%?Sd&9kI+?7h z#iVo~kyagBm}HOE9g5uzA%vPDZj-O`Sy?8X zfwg7-F@C5fs})NA7w7zHCnhnO?A~aagRU1}v_`tHhgFx+jm8L{cLHH?>5R?lNm>tw z7nZ3BqwZAf8|>K%bkq>{YUhLnOdGu}@0fsGrq-M>gSe5oK{MsejD99&IX+<_KCT}H z3f?;Rr127|tZr05^}7v74@aN8Q2U1@bIJ|OEP5m2Szv%&NVe|TPJ!n%jr61VN^TKf z!fyYU|POTpgZ!!j8L3HOk7xSe$rflWV8iNM)eIB$R9ZR5?Swye-+d0_O z@%Xh?PtQr3d{pn(ehbx2vY&(X-}KmEk(p(F6(E=kFNj)iW&Jb*0O56WlXcEMSDMcF zAeECcK8;!{-n(`d$h8*G7P5FP^{Wh|H=aM{oQFCP2$SFAjdrW8Y1FvEX~%|bJb}%f zu>SsW(+H?=)9`WCeH{1!SyE8YBP(HWyH&T0czS9*kd~ zaEu!bJ`(s%cA2Em5=uW726Z67PW%Cd{}#IF$x}!fnbv6}D-DPJ8E3_6shx5lPXI`i zFh1H=fkMC;s7Cu5=}I6AAV@^UgPgI%y*#&jgRiosh-3p;><;7r4mp~^9>T$(Y&T?k zuqolb$n)x>yigDo9<#l7L)f&<$w}!iDa=>xWM80svBA}B#MKF|%gQ1PbqEoF|IoF) zXuimv_2qPE?L6!kmi7Xfvj5#QN>G+2K&@j{6&MxhG=&P+H`XR_df;YO@H`j-f$`x> z?5m@gX|pkEEZ2)@ZS6;*B6^+(Ny_ARjT`Uq8z>a9n4M!wW%PoHkN+C=R>uJw>@G{S z%jrhQ?^Ra((MYfP?55nFie3)=?|8^9S&~_-wJ~L7%%QRHejT3j z6nXstr&;EZn4Dnk8TL{{ab(9#K%3Y9I)yIKH4vrEiiOXo7{|{CF!8>FF@)o1mQHVl z8U!GHz|8t$;ifwTv+)3g<`qzD12!b07h{_B)I&;Y%$fqt6S=4~CjGgl;|!CBi9n~e zW#e1#;jt2gumydAe1naXx&jQ3I|LQ{u(;)wZu?W3*zm}(s5yy~_EZ4x6{=Ki)RHtU z$L~d%fVsWc9>1YLQU&B6yI4TOk{4`6wT%*(R%=)w;S4ks=jUTqA7)h_9_20reW26> zz=dJ8%E}L>MpX{Ts8<1{fykZCr>8#UwD1=NZhF>yG{8@oDo;lA>9H&|9U-r2&F7l( zaRB{OV1g6O2Q5eKaJOwSPn$7RCysQ)R*Gs;mZah$A;gXghyhFIO62jIib@eh2gTS< z*!su+#!L3{a<;ock$a4}UtjVANJ(GA50+zg%m1DPqTQlH?#||VWoeA?-knufO&CY1`K2N4c$ zJg|X-sxBYA{w76IaBcZOx3os}pPevS+sf}+Zd(Z1Is-OJz}HpE4Nsh5VPRFbenJYE z0Ctaq>wpq2Jjn)}`}Xp;Z$fe@qCB$7(O|_(C#nj5&?VUgY7;8N&viT5i*|Pm5&&M5 zQP&Segw&e;@m*Sv*RyY_7jx-m0_7*95U^b#*(8I+95HMUQejD9Kzs?Y+GkpOd#XW+ zi2%+d+JfV1z?Asa1|OgM9!Fw85AtI+?vl!*prU_WuRaXuI)1Minlx zDd}k=QVZ}r#0WrTyU49k#vP?&u}lO!M}$tWW5cqP8%^->2$1$pwOG)lH=Rk%uxBa7 zFl6&#Z!9pRG23wof`dk}cv1ls9x5iTa5yviM*@@+?b4~O2Wq3A5U>BS&KaSG-1dFT zLFjnL3I{Bs^%%IA|EmSa)XgTlnoIe4G`|tXfIHdkB z%L9;sGYsdnu!*oh8sL(u=vU?1auJ*R0T^i{zr`!$S(CZi=w>+5G7$#=i-&ThBLd(@ zedBrWf2n44&1oM#TB&ldcg%HL>2JUDx)JpQY;!z^dUM*W4Qvr-FdFp)NQXbF z5@1bk@qOZbuzmUSJP#;23JxrFybUkU1Nk|C^Z`aO&PT*gOhw7h4;uborXVhVaBdfP z&Po8lHmZKKwaU~QgD^lT#91w^S!eSeGuF&*9nu(s$dJw3Gq}jA`q!jHCE~-@`x3ii z2Uxa#mlS4QHvp1eU`8u)SI@d&5(P>CiqvVIgG?A(RHU|?MMmZ`b+634lp;?-h9bG_BT2ejHza={NYcfr z+RgufF_0u>@@{s~xl>EM0TttiMeTL52Z4@ltJCbXPg%h_R&K4a)MSxnQ^&lbxx_d! zJ1zCLx9)Wj4Nd(G*W}{v71T_Kj7+f!-m+eW$*|4);FNQcSv>0Gu(lwp_=7k^OiU}( zQJs>S_TOZ+*oHLjrk2;L8Q6e7Z$;pzc5pu;)J3=gU879wNzwN=O8YHtrx?iX0Jeox z3TRHm)j#cZba8N*<(my(o)>fjX1YtLTM6OhLi=RfGBe|i?W}&Ni#kPUCno+Hs&yHF zWv+8HTQe_LafK!h`s1M0Xd`FqN)J4+%N#!fxnbJ>2NLkKfW8TR`GKLnBz8evF!Qa2 zh}m7z@oX3>?2MF*-Yel3i{4gdPt#9uj)uyEncW3b5n`*e_DF;U$dzuC@s z-12=6Y@5Bc~MPH9N~_p0IH z;Saf*X);nSCb`jwEy>`dX`bQnhJSJiFzyC0ktJb(DpsaSwC&d2ZSi>6EA|%u`pBd+ zCO-pSYwy|Svi&Q&wV3?S`0Y&2wvwpa*(@Ra0~6o^Y*vgE_&$?bGUs zlU$g3v+j}cXJ4a?X??le(H~)AuPnZ!n~{0&N3_dX!lPh`~c-)z-H(0@MYTo|b|3}qVhE>%)Z-WR(ND4?xOLwP$bW2HhcXvp) zfOPkvyIX-ncXxMp^KSI{{jclg3vkZfD`w4_HF3|ZBB7pmoEvzvzTS_McF~HfnmW)7 z;xB5ZP(FU3{S9Tyv$W~!u%iIbUqi;}^3h-h}^ z7T->Y+{m+?$*xAUTm zFn5t}t{$gwXM-ikTf8IILIhB$P~}QkPx&W>-093H!R!Kio;ev%sd8{Zdjfn#G1&+V zZ#csDiy5(}xO0e2LAk=^3SgQ`@QVnUkN_=~xS*2Euj{v`)I|z;m*v3y%f-IV{Sn%b zEN}RrpCJO@3{VFje9_5zel8Wfdip58?)Uw-f||Hk z)UeQ?;!mrxSd!QBa7O)7ihHqkAY?k-)p#Cj_UX{$&B@?x6mQV*#ZeFm_I$Zg+#qhk zPmyY?lg&KiO%@T_;wGZPLlnVx^7>aeXoK5^3$;i>?}&=xA_6c*-V+Yw77&nAsI$2@ zXmZYdmvYg%;#FutZgu?B>5&P_DK~prJO-O}m3pwSz@wo_AtdamE5oRdxX;hPc zrc@Q)K}adlM~&$ml}f%^)qC(mT;{vN)u-+)ILN$6PDo3i+N~Zwgv+nAbA636QpX^z zPudXb<&4|eeAnY4kw&1t+AXO0TX!^%6=R;K zdyni51uFhCd^Gfo-~Q~FqmHwPYPFZElkqpjG-Hsb_o87Jw1c>VTjc2YpZbu9{83m{ zqrPa)ww^(_a>8l(cI z@Yt^9~Sp>8UqaX8;f2!A3SQaYA!7V-DGYw)A$LOe#A`QKkl zFN0A*%N}s-$;M?JvW;V1 z{tlEjgTdlW$%0#GHqPjPRKY~0uKT+`1TVf;H{O9ctz-rD(uViH)@IdGl=d2dBir?H zYsH?q`W&2j{20zQ! zBRqnfsB-&RSw&JNlFp^6X@aBzirRQ_jsRbY^hYr@j-K2?xz%UKl7MITuTSLHaX8hI zFC*RJu?uTa`}LL3wc&XjNoxafM;Z$dbN3tXsS;>MM*2_g7Ee57AB z)FDn8Pcdz%rN6w4dizRHc}C0^T~95v&|`d0$k48FJ#9GQse61dJE2Og_XlQo(#n(w z@W&~bzd4EN>BVI|u91f+n?zsK;di;K(jmzGLh!dnXAk~9;YC5qiBGLAS^5W6FxbpkU$^`A z3o?_}s2@O|q8V?E<}aqE?ow*Nq{O?k{YjyYAGbE`WA?Sikwd2UsB!#eKmjjA+s&Q#P_gj)2{Yn|6KElRs_ z_dd?&yKS0za)?9?VwM>Y=q$Hb^4#9d-h#c3#2atZ#~9Jf{^F^hjx!Na(BrUSV1jso zd7|5qXwAr42<}*ZM%+J=jT#;*;@SHdn=*3Ux>mC6@IxsjGB!PWAaU(WN$SlTNZQ5X z13O8JS%TlG0o+XE%yt!UO2CE<;UsboQ1m z!M!x=+5K@y)>*UL93_UzrIZ7ST9KGEHMHKIwoI6_@MW1Wj)roXu7;o}$pk9XxbOyp zo*7PnHlZ*|Ef77ztVS&vbrt0O;`LMs2jcs*^}zLEZ>O+{98E<7MkLD1>YD!u3P*&+ za&s1hGb|sdZO@)Bi#>cv`G?cx@Qrz0s=+*aafA0Gjfbbk-fTXjlnzHKwYFnaxCS=|xdl@&8 zW4W^Do9BG7e>0zP7h`{Z)U5=-peTZqUsL4b;Bp$><_1105{9R-C+0MaZ;0^XM(wvF zi?9mQmeQM1FMPYtP|2?QZTVw9ux{2*z^pLw{+kn4e3-xn{P5GKBFckUjX5``P1g0n z&HYki?##N@G*~<;#}-y9%1ibseRcWh0cu;|S6?4j+;(RkGd7kII^={pkMku;Na^=b0zGs^WNzCl+vJDh+- z6i`H;K~&86xqoh{ETa|V1jR4IQ@izK1;RHt{tSq32h){bbVn)ioawPCvKoRol&0Tv za}5(*hvvsWsGBpV;+Hb87PhtF*6)X?Y;X6F<(Em13_iCoUHrF*#wG@0H(%1`T0CI& z)4c@_Z2TKSv$7z$(vYD1xSwy-iTryUxbOo5r=C}yZ6>d)Gcxa0eI#^Rl=gQH#0AM9 zMY8H_@!yt|>1gno?fhEtDZkXtj4zccu!tQKTjf(E>(+XzBwlS+U|h<{xTy2y_C^qt zlUCZvbgHIwf>&ZiQyALu<|3tWT4;?LTx_s=Mz~q@J!l0gKktwAP7qYgaQ#YQy4%Qe zrDiAKPe_kuZmP2Ho2~}ruvx&eP*bJr9a8BZlx)HL{CtBgBNZ=VBH_~5xRi*(!oAWK zLo73F@J#(_I?E&zCRQfaD5}lZBSnQK>^)(s;~dDM{Z9N^g#PJ+RBFt)QbbU{GUatNHL>H)|tFxLnj}2Ak z(vJNwFeURIzg;)0+e^wXEgVOKv`YtHbdTdfm*J^ai;L~dGh}i2ucL-0d?EjcgM;Qq zle+D@tQJHfjGb?kZ(v}kYo(`r*XNH?D%eVxN2v(zK;<_?4%1sRl`Cl4eYxJOj(Z zdCzSnTsk1&>zx{`B02^p=Gh=F0@KQq#a#Db$zmkvLD}%%-kY{L18z~zq7HH*m;emO z{?qfbPoz{O;}^HArakEq=G45hn$_Mr%4#ft7)F#A4G<1k+uA5}{FuER^0w)C-n(wJ z_|K-_-giz@pr=>^gYEH?wekr}nq~LT)ZFE?Gup`x|7~;SJzmnpp#cNRFwplmeri=a zn`;sy7vn-3NoU8T#3KRbp5nD49u_PUD_(ISH&IhuALK=kI9-D~GUL^Ph|Ka?bmXKw z_H7e<#K(4s(XI1g`!(sAxI);T_@$N{PZwhb4;Pm~4~kp=CZ-HWFdFi-5cxBWl3P1C z`46}p4lV&rT&zq(wibz2{j9(DjOR+gd;0mNemX&E<7lBajw0}89J$z(yGV z*H4cmf60W?GxKc6&)M5l5b4AHeGhZZ1Kisg&lN^t(~zd7oiVMhv`y_Ps9t<$PpPSo z;H#SgwSQ8eRL@XfT;L~tGuz<|Kv{{8IN0NP+9(4~N?!GykZEq`0|yxus|)lWy|oDe z&`8PE))dFiZ<&kxYW<|^eS0bIdnFMVfas+1qO=RO>2yy&7+7M!RJKCefJ>k)mS1gu zvSopLK|!H9rUq$OMnZaIyN$w2%GIMdrKsbz(PtMUZ4 z*AF~91a;QfTlJpHdS#_WwNGb-a7h5Fp2LFKedO-$^0DEvX5wk)Apxpu3~*?0$e{~&jSSkq=h^iE}|ID;&|k=%yVy{jauZ@Aw{v@@czO1^eMT~qjhs51&1Fn zR5hL+YYArGc;sGP1v5RvEcz5Z{do-ah9Kz}8JgoVJ<6b;f2g;)8N1Ieo=>trKz`SPlYIHj@|j|Sct*TR=(dp$~Zm0UVR2113XJ4akyX?49)zk+gGE_zyi8& zHv+a_75(Ptg{UKAa3vHexpuzvL*!c+$If;~ZnfL`p8k1}g9V!f;*kup%`Vi)1c)i@#L7A|$LSfTVvdWL0Akb33Wei!{qSVZ%wgrJA_#n<+lGSb_pNwIkF z+z!Q>Q!y_XkjC@EwfR}weoBj<7F21<#7oxMuF;cRw=wFEf z<9Wh**(Bm(;ozyhxWF<37CWbc$}n!}#TJvfSy*a&$ss;=A}i&^En$(CCq#fC1nJU? zR`;G;G5F}d*q}T#%B?^(2wvaymsuIenf?By54&$a=G=e6E7b|d7!`ftC76A?8HcO? z)&UCVO=E}A)9iVw@3Bu3-*JS1NF(Fb)md}4S4ulVa6{k1K?MiTU|6x^g8qG2%j_J8(X)_Xnaa_(BTAKO5Vxvaf!b{NqHlPxPRz zgQyaNO%IW=3pR$z@8De3TCMR4FMsx=l@CEcQ2e3GWavEj5&1EbUzSNHOu~f+o@OS> z6td(CnRF=l1LO+SOLHgf{aY6yem?ZL)W&y+c@{zglpW0=VNu%_u+PKQ42%t7#RnH; z7j5qq=RNRzb_{&pi<~*6f2LbOi*)yZ>hgHIk;1-K`Wu+6A_oC7--7$S#?Sw(2@>PWidjFB%F&U<+E$rL_4e+$G)3f+fYVAJ0zG8wj_phC&)#`dTU~ zcBxlICwH_+{+?ugNb=@iQ$seNjEzqh0+YzvLol=RZw=OGI56kV*S?jT;!DK!_6_AN zRFll|nKJ_Sizj?@7T!;tPxf+vVbNKQTn6` zklNKL+tJ zkG~${6#A5OpU8u;NP!qf6^`eJMAL01j!ZJ-u>fQR<~v~}Z~wiW7q~c|J-;HVLNQTA zjBaC4t{`90xE_X0Ds_Zlh5*_7?KkBR;vk8i-aZbQ7*u25QV)9tX))l1$Os?h`f`^+ zMd5lAa^k;??nm-Hsy0w{(w})ovf_;pMaV*&_IB(iR|aTwNU1_}gCs6_-d@78{c1ku zm+QlGFDpZVj8ok&&p6RiF+Mom>A4#8a+9iPkM*0vKN(GT;Y=edMgn`^EW{7~JXHn% z{+(^uLHx9nuF|(xoC<`Q0T;lu+2kMs0=2yi0goo2zk>AWhY|(EJUB95uBxZA6hlaK zDf;^Rw#8=q`en(5X)o`z`O(n`hb5WG6`hK#?oYr2_|Bqa6k5V$TmrnAvJ)@%-=3V! zby)UwpP3agL?Eg9DlxA{-|uJf;Ky-qp`WzN$^)fYV`dWcgMZ=ZXTG=wJttG2r#?3kY)K%8Vt~5D z`KZ<|SXTXuhUW?hWBny0$TF!6`?0Z-Zv;9sThY@8y6njl9X zaU{6ki=_8btKSh+RCqH5;8~NjF)gBSSe&*g7vkx^HrJ&8V$(hUflncrQ}6*3lk;Fh zP3FCngOAz**{gM}#tREuZ90eaxbR>G_^%BUlao$6tz%seqD?;x(;JK@!cA2P4Ww1s zCEN!bE`Jl<-Sl?0cBK|J{mO-(L31xy)Pc^$eT@$n)doUNNhQoxcBwZ!)vfM5iaVB`RsM;ClFW(2weIW>! zu=%j?-AW_-`3X7mK7Vs}$Ly_#OT_Gc7L_dK9ysCJsL=a3Qi2OxV?)bnsy(0Dkw@)i zDWXc!kt(J%N*7f?#>%Uh7k0#P#r{;|W+tD`O3%dPr-h{{=FXN?3c>6+Z@H&V4$N{y zD&C>R5Dv$@I-thlG&VGbiIS_>dJfb0kD933i6F{MMTJq3Ant7E_(aIV0MV< zlgRT7xG*vH&ci|-+~wlv7C;RYQW0c-U7@vV<(-x>SVp zF1SGHqN2pQh7&;%=<*0CEg{yKw@R!w6^n`M{SgOGZ0(RmdY_^X?5Dl8ZPOT$Kt{1Q zz%g}y&<-IeFa87iK0{kXDUc@;V1L}HnGN>I^IztcmF{2ocfcSrLgc-u6CovG3lSx0 zA2mL^(m=`-Da)b%VF9+91yv^b^^h1+sR!hGvZ>^r8+%f(K^Um)_Vf{r*9!O$l21>R z%8w@in~djDL^WP{b8|8c=;@TeIt7Cl;%>T{=2$k!)Gt8OczB-Ik-2J*s=``nm*bbr&39UnF*dbO$=p(gSz_4C~=S0$gK)o?ca zaJ2~G#7D*IF<3eAs7v!MC68s?r2&gcBsQ3tI|=HW=k<{as(4=w!7JW7s5n)e!WWJC z94bAWoEROL`;%_)`~f&(vi$@|Q?BJ2gZyhN+hXnV=HjL#Sk3_Om9oCs*BoC&X&c&s+JgtKowr-!0Z z`W6<7aRtHS-t^i5ryToP&Yy}PcXSnrvy+~$)<1>1d)Ace9Or3>%cM?5QMP+oAyTcO zh0k|)y{%=*Yhjn~gV&$Du3LGt41&Gx2IUu$TKqCoD-f?Lj8!tE?XXpt|T8Gp=xZyt0DLv}b!> zf^erhWc>EZTm`>?>Uao6_1J}=h~MA5Q2!bPlad*OmPv45pOfqtpPqxeuj`FpMBjy{ zm=QLvE;zUE-4<>JbI)y7ILHnPjnKQv#>sIKPzwH-Rafa5RMi)|4*S3iBEyb?$WUO5 z-td+1_t)AzZ1H;F{U%SiNlhJD2y`G_MqmVWTHA{^C}N!8-H#nLt#LbMLQLahkcBAz88r4X)o|i(&20RmP_Klz$?{O zAlEwl`KaVs0T)+uO38Z7_;mIJ!dDLWCFr0mQ$+CyGE3%!iu31)Dq3X_vK&edlOX1S zdJNwmtALIJ?f#H>^TC?88a?i#E27d?CGyl{bur&`HwgA=M_y7;CI{_ZAjtCjH+f0Y zGDAAEy?$_qABZZ%2^1P)Esyazvu^(VB%wP_+%23StLKw{)k2?^?B^m|GZn=NrKR^h zvNKUnU%lJ)K(Uaj0G()szSI11*R(Ti_lTks zV05mHqT{_s&I*WP=Y@65+o*6-omW#A=#MMa_@bl`{*^UON|aV| zE_|ME8aEE{^JG-5Fs-L)&WkI5kzSSu`lv1_6d8RIv!*>0jWCaOB4bAA<$`ZgAu1ex zdp}IgDkf|`VoST996>F+2z&98pKu1OBwsx`zMoY$dGw=bx_4CD`z|m;1jI+)Dle&_ zaK(7|nKd8UETmtsOjCXwKTacgC{9y8lsD4sRE-PUFnIrDOq5WYluBIjxfSyMz~&3t zm+RdV!@D#M5Ba`A3V9`>?9jfKEPoV0VmSx=_#xPH?GkVOeD{%p9fZ2$?=fhnRtx`zzF^}WuGzZ`;FlzWj!!&FD_hhGi}P>4ZwVRm(Q z_eWt8<@R`r$KD?N?3@I;d8Zo@Mgaj!w zv)^Hf6i}i9_>`Q4H=@h`?YW&j;C~>aWDIe;im0rt?^*7{6jgx*MhAF2yXOADVL4N1 z@T4*@{A#Z=vHW@RdGiHZa53Gk4@pi5(o%j3k6lfK=$MYJG z5LI6|Gpf_(`mobXH5Z+fErYS8s5ewf!Cw`C1*1E+xKA|8k7EHm$7Sr2%(bXf2hK<&FTq_z9 z+9Gz?t*XJBc4o02=&sMslhQEO;3~v})2zJZ^i%MiEk!F_>T16og1mD}$Uggd&b061 z&|lZ*dfD1S6ZYNRoqq;bl=DE3T9Ey~_%~(_`$HOISy7wq+wa5ap`Uok{Cbw9Pff>u zY}0Zv)MW5Wdn8YQY$uo`Ma{T@vd+LTP4|~de$O~^;iK-1pb=mSn(V>}%BJf4S=SWy z&j642{K#*6Z;H7qJ%Jfvf;sy^(j1|bUi=-A)h1I>LSXE)Z!q^UiWzJYaVb<|iuh}I zt4lL)W>trcenp--1|MMeR1TEieNb7c95%_n1`2O$;XC*Ih+#xV4W&Gj>%>4Q-CCC46h@#)s-yh=Oa&`GIGlh zGXfA<=PGi95JV5FLCU0SaQ7l3^>5I0wj7DyF)9N!9aeUuKK9lpr4WK2P;mcq*ayH^EJN|%DkFv7?%flmtzT006 z7}j(jE0kmy@_ed6Ix9oPv5VntuB$ZPF}glkj)<-#vTTMK&oaiHcQ`BX6^AMnDv8TA z4nnCf|B;Rl1cFI{kVkNHd`=ke*RNcFTy{~aA|auTeNspImP@Vokj$$>#!Z(VwzjzU z4kqf@dC8tsQ-X-`*sgD6M@D;f2+X(HZ8|b}Zy>r`X@lu;L$4*xyUSLAF0IZo&Xt?8 zzTArei{&19%qtEYiE2SYMsDQXGe^cd^H8Z=9?8m<3*+@KC|ce}_Q@<)V_YrG%J$m) z$q)C8R*%&Xsoc0d`B%l?+f6vNTh5@`N>e5r1z>+XK#PwczXovWUNx02%~D5W?`iru zLA^sVJk4?oQ8$+zMtB5-Jw_11g7V3>ogmY-OdVM{;fgoBPo1W^rt>p9pSSPtAJhZ1 z==$_=b4}H3VO)ez9q8o|82@& zBEutsp!KA4P>P_Wb~LvTSvvXtbkqMOIxennS{nD(x=(vB=?txu8+zQ+MwIKyDkwTy zeoK}dP_^F;O_khh0kKFuORnOerPv_B0X+VSjn7VzY>VtxC|qfgLfrjyNxZh7-s2Spi8rRg3-oU(qUq7axoWE9ZVaAPTd23DjOwoe zR_XG1Z7zA9Ex;d6mb2bdi{0mT=Nyiof0Y(Hp?aO;acRz_9m2!IheXMcvQuCI&P?Dn z3+?J41YN9w@MgRWVFkU=WM=k^+ubhpNEOK?SSMYe>&@-W<*|$W0;5xJ`{{Fo6_wcr z6aJg`xZpN)<0MvNNGMq3SYgrkx<*FB9i4w}ZHEP~(>TDV4+O>=*{$Ix4@_a+nyxdX zGywp3u1Sgth(WtAjAjMcre^_=?Ke4ssae?eWvx z#~Eg$xyn0;R6d-ZopNtkDFcl@PT=5jJ(}1#GoM*+nvs)}>mA7$G#uhj`3&IwIQfA6 zD&=AAA+8-NJy3UXLE-%~K1PeFC#MoO)<~)g+ztm(8GXiMe?rZ}dIyxWY7RkIlPA+( zfUaLpODb1b$>sZ&p4iiEPnC6XCaDei>e;Hz@5=#%3Bb3N?i;=_BduJ9`NmZS*Idd^ zt?@)R{JoFf$`UUTYQGm3f1r|p8T2+k3JS@bKOBDN;aydN;h7DBO>a_(Zvdj#D7C*{ zV}i$Kbx3S1muH)kSvU-swFJ z^Gs1Y`|9CwFLegbfIfZH@4lg|!GjxqwqPuN4;-J66aKzKR_)@uY&4gLz9>c(;BsVW z!U`$_qLr5wL>fyPSL@#eWPP#bcSje{CAaqYLOqwpW9f5`tv4!xd`v#9FBS%UC)5zL z?MDpSEq72_(o1(%dj%yNXA~1gvPA6M3H3eR4oYObC)&>ez%mZ1U1Rf?KE_xvY;<$h$C7 zY7$%@Kki699TR~CI;tdq*FGifCHdn4pq33-L5au&=Ym^1!<7{VgLqD~bZ}&?nwL$A0|y5yg!K5yIV=yTpPPvcj>VXKtx-`ig0Hv4EFI zboi$oXQ|qD`-g7dL!I#{7-1^LG$u=#APGPozckrPoak9_4aNjUzgn; zzoF^VD|zVw{3N)@_-gMiUa2|5Wr8Y#C+=bI{1&2CH-I~pb=xdw1OOl~N7cjI%=ik` zIkokrD%&JikH=&kJ-34eC4wdQH@NQR3Umdw7Au7@<%z&8wrQ4}btjXoEAHD}-|g(} z_Qu&kgSJpmP;DBwM8F0DWK9Ay#qc;A$7xZF`wH8Tjf3Or?sC=6skyq-UN}H4awBp; z$e7;AD;2_!zu%oL#s{Q#(Rt$tUSo$jvUf)?EP>&1_RF7YU6N9I6-B6#z=+$86GNWM zQS_GUQ$?BhA{2BWcrQFe&ii-SeXeO4*x1$q>EfaOV6!jx8JUl@WYZBKW=jh%N8QN> zeYw6!wlpXP2A0pvvbJ`c%4B5uKaVqQ<+^uk8UA@w_(dy|i%H|g1# znLpXK;z2bCR^8{&<{@DcpV*kvs1g{%G(&1X>l*6>&q2yn{*py%wu3(;ZDMSVsBn&ocqqn!ceHao#@+HJL1Z2IyDw0X`2ud8Qk3(ZJxI?&^Nr5 zlhCHKP46w1of8SSzX;rinFK}{;ps2I$j~!0Qn(Wb#S$uFF0Srqj(jlGxCnsAm23k{i`gbKtO6wM zW|rlF1$>`sB2<-DwvE~(9`-}W5(^qYou>Pq|4q%!3}y_q(uK`-=KR{cXSDa{v0qb# ztLqdCmFDvT=J6hX;;?43xu>ZhR9drBf%Xd#&7EW1EAA0c24G%wRS>$G<23~@t6wR# z^LSCtiqa6<)5626a6Z7a?kO(hz8# z!FQ$oF^m~JqGiv=PLHKlbFRCzPHR|K+KNKZT^8|hX0{Kcl|p)+oP6m@5>tL@tuzQP z69XU0?#TbtmNQacYRaGasl!$dN;Uv5(>{FUG)kRLAZ1}Wj|Ov890gD7{u}ClEFq!k2}bZCz$; zfo`L}1qqsBj{fSJ9l)Ej2lVEMj~9ob7X6_|AL^cz5f*w{I)4u;rkyld_~1;e5*w&zR}`Wm&Uv*l~Gy7}XA`dJ?s z#U)c8UScQTD_SARH?v-u_B4)O0N@Egegxx!G#}o<6BYe=&?o1UyZdg&AtWSZfmd>7 zHJ7O6=SOWZy0>Z(R2_s4C{rr4C@h(NuexCw*K*C*oR}aJAR6&{)c4N5GiHZ<v7g3{00sFbO?HM*5A@ z)>agS&Ozon3lm%R&8Ss_3kNDznpl6HV#`zV=_IZ`_Nh)rOe3NXO#b_Kh-M+N>T{L~ z?-3DC&1X_YhFT;d=Q!ukK2D+m>|DzKTLMtCUgZVXrOCHK*KN-Sb|@I+KiyN(2a;$w zaGb6`?%GH4=^h)uM?p(hQjSPKh%h^AyMUejv z_$PdMyy0xYVk56uXJ=jtc38kvv$t{Ia;LReYDuTXoiX(k7lcY-FEIkgm;SdHDgU6+ zj)LVxofbtuG{3jqZ&{PKa`?VCS2pt$Uq@1_cf&4kzvR&#(#m!G42T{72DrZtPk9Ol z+|dvr?#Sr4Xw5>`P1#1V9wl0nm{N-{vXFTK3I#d<^C8k-;);?$WST$g9Z$xi}vT=+zc)fkSOkOM&zcQheF)Y zT%$YTvqqyI1g3?BYT8X80W~wKu;DJu>!}7b240zG1zS;nKt9rgUg~(dg)Okp!!yJ(S5fB>d*_-y?)1BqXKQ zLz)E@6+b5^WGk#kkgiWw`i?CZDl4k;zt!`FdmKWy9nM1mEtdZI08C4m^}luK?&$DI z_w##&Ek?C--AATQbWMg6+#u$Hd7&VK!`gb6)6_ZbOpq_OpCu*(-aV?p8Mzo?svL~i0_YY?=%;Av9Cu-5Z!lCA()uQIh{-h3uFoO z@2c`VF1jf+!Z~Eo2?;%t>pKdP=%rgzCK?LaBa&IVfB#lEPzNQ2hdU4tr)>eZS`Pr{rp5K@LK%z&y$4we^5%Vc;H=#sE$$g7gM=m(}Aj z6xor$23ME0&BGr#Ip}m&$F)^dxtWdDpRQIh4%^b4>?rSyMwV^-m#hm@bMXFIeT@=*_vIbtqcl|%eq zO6u&r$v53UgFw=NUN&(^SO_s4SLn}C1~%(u0X+}Az65&U-jlGTBy2XPx`sw~9ZP*b zpyuf`Tb-)<0#IYvFE5UD8#8`(cK(fyiB52{=o}eA-Su^1iuxkM^xz<3%z$MrF1}+l zu_h`6WnZ6xQEqH{yo7`Vd|{z@Zhr31_;^7x2`0v|v0PHJ*tocWuoUV#CnOH8=JoYu z6f6~FxBFUf_Yf32GCf|JX6>^Hz})ZT5&rrMh0`WI<*?Fmxk8LrTmhKxzA64Nt*wJo z_t;#V@)4Joy_B08RUu_x*H>*)R?d);5=4cV*xvr$Pjob?gO`ec-%L#Q+rs*OXlQUm zBo(vh_}0Ltf$<6huvNYt9v+qBF+ixWBO@YCPEyx;Il;%u$W;6CnZ5o z*Es=fsAs7kmKRq$U9bGQ0iEIg@j&chn#I|*`CuDu(7WT6%5(H-hF9;T+V_n$Al+Q< zbrwX&973f~uYc5xyZ;q~ui(1ZMzTZy6OU6{=}34hSb}O2P5+c0!CPe5Uk4z#$odZhIefrMs`~eBU}~uUrgo7G>-J4iQFu1Q{{MRR`P=h5MMrS{ z*1wb_pb3`}hF>9ae_Q!?oDp3DrV?P`r)nPDm;yog*C~VE{_8pK5>V2I z2Aoas1ONZe9rmYQy&$C85@Y|hW}EVN#)7|Ib;q;^;ClBW&Hqi$D)#U6g6-V;6fj@= z4px)?OB7+w(jSjv5>jh`>lO%W{H1pNKN&B~2@?o&*;0>?l3@LcgnJ#aE$+WQ6f*%) zRaY__B2(}Rnf?1!*#Eu~4-;Wwuo+~BJ;oZsV?R&<#x#TdV*0~Nk8Zx8QIwNZEa|9T^*~aoL>9OQ<2gAulmJ0jvLZ( ze`KvY-~$=md@PQqTVJMo4#%I5b=FSmXfBspO$rKP$kazrrqxtD+M%bSF&JCS;Pkrf zplxTTB}*z0L0ZqaYd5n!_`3$+zntpd03(X!j&tW?r?Y^nN3wCU+vt72kG|}7CN4EJ zc)Oc2N3-Dc3xiwsbd0^4k(G6$6f0x1y?sZ;9PvPzM)#h(JSHy5Yv6>nOeNzwt$B@7 z>GcHmAN8Tpe;(HONhAC&e#=mSKoY$st*v4{XK`%=Z99#cBltM!l?jAR|8Z0q(Y^+Y!ORb-_X==zweU3extu&=opD8C^*Z3Hbe2W4k zCG#F*tDd_7Nz3gX+f5s*CwPdvC zDLq6w>`&wRaISd!dZ%1BG`XISw9NSjrT<>4CV~p&6>FC1G%S+`5Vb&8KxMMX+1<}L z4=l#Jz#sqt&qPi{7ntWgw6%z9|ftamTYeuXvW=WYK5xPyVB& z@ZUj>h=P+GKqQ!F?uG2!4!~oy#q4t6xgJPzS$+NTg-_FSsiL;pcxZWo{fQ4C+Cw?- zK$SP<=xj5PCOB`uKdbv&NS1N;IojiPY6L*m$0Q_3idw_EEi(TE$T2<(pbRkM@%TM_ zfODaG`$aT}Zs|MAjWH&Wcy_Ut#bS=_Q`}6U{n8X1e#w&laV+X$&G;Ssafn|Ix-Jm~ z9H*^6IUnt<`4!UsjCIFWm&>u~&QQ)sOYeZSG}NbtJnGVU=k4BSO^#+aot@R*5E9tF zfuXhP?-#qwCLbXJprKpX)==w8)WroBkXWqd!%1MTQ5@HJtfih(g9tp~eb7abn4C;x zf4JWmgY#=4-qs=L9gjy$-D3l}sJUg&jhwyCifOG*L{#R0hW~Sip6K}F->gg`Dji+J zVfB@^!5XZ#+=`0e$CgKKyXA)6&p@7OG7wd61pnbW|-(b*2pWoK^IJaVJaxxqv5gB?oUzP5hKVopZ3x?nz zB6}t-oEHRvb z#Q4@h+e)$RM2s-3fd0ug%MHROwtKz4%5LxBEcpo|vJfN)Arca;*#Ro_;_cdnZ-w^pa%$tW9mz~1B&8=;vhdYZSJP0vK>BhX9 zfHb}@0x6#WK9bvRh^}OOww$l_oKOo54V8#=XR~md!o!T8-UKfB#8HQ-%LBmTG8sG) zKv@R>YrW;+@;T_{2U(|yY}rgE1^p@FE82n-g<@Y?!upL@AXgINjCEWow> zc;@qi^=ABU|B}#6;T2;bxF{eH6z_36u*q#lhs*5_0D$4RJ(6&yb9&dM@q%QUqwz&J9G#LJ`TE<^9=+lz35tKL;=!o zaq{6{s`EPJps&LWV2Y57k2A(?=I!`^6iy#>*}t_LZZ%r3{ysC_5z^7 zdmmHM!qqbMhz;E#%S8J_gw_-6w{~AR2=fH`?G}!WoRI5F1-jS4-_m%X9gGJ{-3bDk|LgvFpVesBe)| zbj}{>%_L6V`cv>79E#h?CryD5!lA^w$TY>-bX2SJ7~c)L1ECR|I^o68e-iv$&Pzxf z6@-J^YaKB#^78&P7x5OFlKtn-llE}qO*)0Mmbg(H=S}KP+4O)s-H{kEF|k7B9wR0y zYBYSl>j6+vVD?)V-t3=>g(ij*ur((ddiPmQP~z260*}puG{CnE*F(r>R~N9WSrq9# za1`mDmjsb`kMDi-MUl=2b8d8;XR*&N6T^nr_B^-SZ_SCZs@)$jrO45z{+QdH)_V#c zEkO&|$`eiGdko4lw?SrBe0qs#e+d$87x2=eljUdng2;X9eF<7*3=LG@_}o%vwB4$tw8cz5o_H~~4su~fq||S>(ibqV1;{Th zHS3$Y%T1ZJ@EelTE;u#pB9_iO=Bi!UcqKoxUKn}&2?V)3^V%|vJYQ%X6V5`lQ$ZRS z8==V8LlR1*bu&jH>AxHw8P+vCaW}0jCfK6$Z~L+LFoBF0#+?YY`Im1^sDR$s_6<>i zL7DqcRAc;(;F6xx$>vXh|Est)@2ExGL4ShSj^`zjT6pawB1A@oVO>9(A5 zr&OD{9Mh*Ewj1-(J`l}l7cN74st5@V#V5w<4b}@%T-thRHixC0?8=C#p5|$v@*J{p zlkW`N@YfnvtLWA*?D3b}U<}X@6VG5_{B0QVXLL+oxUARxguWZnZvVIPsE(;aH(!+} z`$|TN#IrtG7VDMG&=1@|So-#U37f@4IcV9>U90YZE~7OR%{&AcGqGK#;vovKPKb9P46)@g-9x=iTa{Q`f*w3Ax=d&br>l+0{+HB~VhD z{hsbku`*>WrO}D*wv1+1XRGLEs?XV8)lG}sb_}#!EQS~T+DTX4Zy>fktNm-k*csOb z5*V6OD(5T)Uz{k-v8;)oRo`?@`88yT;vVC&zNIHDXVMoQ)*Olp`}bKn(~uaKspR3qUB7R>~g?iDVJ1s zLsEM4G~3RbY#Y@UpzVX~M{pQaUyg+vZ>c*jnPIzbgNMxTO$dQq`2BE&Kw3-Y%N{Gg z=aqTidn&S{j3zszn|KMiSUO)wgcQ&f4ffQ3F z%wVN%C6UGc4MF-bgj5O-;NKc4Py!nvHq;~AR>DbnSTPNge@lA!L`;0H@{fridU&5? zd0JbNDh*o4+1hMJTAokw0_8zfgm0Gn*YoxKo-doJ? zo%iRT_&z`I^5Thmo^xH-Irq7xj-2|Fv)hB3+z)5_a%C(hMb&D{9U&{Cg)aSLSK$gM z`-4YU#<7Ubi?f$D-lHJAeKvj$hy= zEVCg(?kntu?$G{K9g~i39I0H}-Tj24US*|bb+B1NUzN)%tdS$ya5$W$D>-H43!C|d zs-l#d>hUI{D{hc2`LmwNVF#vHhD)H7`_Y~2TvTXiJ`F}FPL zJh|y;fzvhaE4kTF>X>+tVU_-jJNic&Kg8XVxO^`+|&_Nc3R+=jdkt z<6|qTIu3FKNK4{Ew9P!02v*4krb*?&A`bWZlHiJuqla)>j!ml>HkX5HHa^ zbJD(mtW!nWfx=Qx_)GY83)^j7??k?kZzqp@(GTj&&@Rw=dU|-iw?%Xbb1by5h!h3m z;?6(k=`_DeY+5=)V`{xFpKUO`@F*6NbYuz&G9MhLXOe{7W|j4qa9svkYMYaraJqX) zSAdJEx>&osR?JyMy&aYl4xQ-wjbLdTXuh${6^?Q}=lwb^nv&Ou)~~aaQ4F|D059Jt zLRvP;gccFpYhx9cfO7#Am0k?USV4I7i|@wuC%0E?*a6n^%`cPlW)sB2=6uxv*z!7e zTm+fSWP8;bwg%Ns9Nq2OU!F4sSr=cMzYxXfnzyYj{ObE$pYATDha!W>X2gnim~wNM z&#Z2q>3{4-<&Xxyz%{U^EFzOS~;*Rdg0(pRC2oB%R4Wtf^*=3 z#`32m#Q2rWpsZ|;CkMq9tMop*%N*4X#%;1nr(_@M{^SU5$1dhk-Bv)9NUNx6nzte| z6L#C~G$qv0$zD8~52LJe;z#>U7(>hFkQ9?yo6PQWi>3iOT;bd8R~@OStKuCPM~8~u z7!Bz}8&KCJjI}-C_?e1RSTnO#_&ejYcS%>ksOzUzbLPN@A4EW!+a8l95vQw)51W9p z-+P$wVAuM|E>^(vqLAKo!x=ePgMLm%ev9bw8^ z?XT~6O6=3KXO&}R26t)i=|pDOLON8$Q@#3y{S3;|DzjX_hS9w~r(^rb6e`pIyYtG- z+M02W03SCAxc@apT{BxqSF`g0gs)eq@^}DU{^=E=#K198Wck<8QI8im6_-gaCGa1G?w4R0i&EeX-R`JIG4^MFJ!WS`RV(i6uhh-q7c(Nt8tPj z=uMAe7qt*UmpdI)4()Pkcm&B0p5Twr=|}d*0}baOH>hMvtu3dIUdT(#vrb%TYU4&R zH2NPY=?ThuP>EMDPSCfE6)4O?Qw!$f`+nWG$MBhbZTiDMSnIciyS9%KIN z!1U2SLtr(|oZNPu~7uED+tx0!0B3BU)O3i`iIe4-sa9>$-hPS7GZmz*42QC&Jq#DoBWtOE)B+GCeR$cx? zEvV*!RkzUA#>ur?h1WlHV-n%a=u2C_sRv?Y_^BCmqqhvJRMk|8Gs-eP%*zgBck;RtFJ`%D3nxgq|MS#-O%8kDE_C#fq+Vm_P?uMfl((01QACnHRd_-m#fxpV? z;S>FR-mz0dy)jaAU*q3X%aM4pqn+YS44N;PuMobqvo+fNVqoD%E@H+K*4jQCa5ask z0{{&C0QF{>OPfh`DYlR2B1HxE{Lx`(6C+XcBZxJEH?b~?KK)z)tfaHA`b{Qne31ip+sSP z%tYn0K0WFZXvMV0IJ%NY{!$1p4^+{$Hq=cUWK+P+?ypGR2JktKFX3(dm5vHLD$fAT zUyOApo>u8~1fhqaGF7BYzayrJdR31Pgk3&Y+GA&;cE61BqMHC#P0XDGWu|*7ll!3A zs8?Yu!5y|$PAfm3bjY6QC(hm2+4a9*`5w|VtoNl56A)4M&gZmI;3x&al77!|(hTJb z{)CA>pNkn@fCa7-UC5bkF#6J7h|Y%XnhantiK96oGgwvG3fQE?-SaZ054JOJJZB~b z_m-xz_r_#Cdz*`(ciNOuo>o794$XBmT-;%Hv?Y^!-*@|%VpRjDWk-Oc1XWiel-&dJ z&p%=3fP#Rpgk;kxQVd@14-c4TS#{=QTM)dkhleM)`1|I65*))mFQH zP$02uDlR>lBEssw;RCQUNScYpXf;mKn=VyefjgnKAI{Z5Nvw34(4w=c^^fQ~lX(%zh*q+BrK@=FZLaFFTIr zGd^P*F4;+KlwfE0i^INQHT^5Db6-}<(R48x`v<>pfU{gMufc9p>`5MGOqrrD-je=p z`;*GmaceQixlL75rM+*$_`FVT;L_V=@eH0s8-pHTQXdb-q+oqJZ(Ld_^Zb| zr;?dv82hXTR?{={VWrJlbrv^0JqV~k&QSY`-zAyaq$;s|;}fd6sB#0f@1gx{+KG8% zZ0uJpGPU*uCB6inO4m&JvbyAISZUzQqqI8bECmPsx5|djAANj=+n5Z?uDymz_D*!s zi5}p6Z}8w=-YHU2+zb_)@+T$Tzp48jZF7Ef`b8o|PvgmyL5|9O|8z&1^xXq1$O7>+ zsD_vH*MEHEz^8x|e^G4T-``Iw)q%hJhNkXrdwYiwO31~XK2<(Ta4?CP9rk$p^-jRR zx4V%yvTb^*|Jsdx8Se43#<$kFE)9K-_zjvQ!d(B3huduZSGH9@3Tt2Vwwx3!_m!vb zUvE@NH#UF(1cxLKO8uniEtcQt-0N2IG;l_5dhVL@a2x3xRINdd-JrjIJ=hIx>KkpK zM$CEm_b&1tLAfWJO#bFk!aLf_|2J}gNJh<+Nd^n*pri^(Pb3+c zl@$0*nxDzzm-_U8BFXbMZ;#iJ)l??=Tu30)Ct{F-oZG?ua0}P{gu3={CUUWHBg2fM z%x@S5%tl|uWz9mdtC7j)Ykm+^Hzhj;|IOsKT7(NO=^5R4KMQF5|V18+@XGW7x))d{(CuUv`4R zq4Jq1;O>(!Z0tmiq9f*6kxUT$RnjyeA%Kw$-QOvPOqiD|Fn~p0L0I&}WnNPU zOU*TvO@ZJ^89mrX?*@}0^?)(TsvuAWBEP<#ezITPy(oI-|8xK^vnu_dP?;HW`=^9q z&NN2^62*uyg@s~AHl(1gfQjS1O}yR_Uo7Fj5c=i+b4~ga8ezoy@(HaERb5y<*!+-U zlNwhYg3ka?3GGR>B8FFSA-S>bRUs6OY9&4lj%V=Ly1iC%`8R zXIPh%O*Cb&BWqc1xeNNi#rDSKp{FT6fY_87mE5jg!<*r6B7rc#Yk31`CtqDO!=p?57V-YrZkBFpqKGI&&4@$?dV_Q>q9UoM9Yt>2d;zl>a)S*2m6 zq`Ir#8l<2!g{RtWHUig%A9?^qw<404^+yC->?wo<(mfyO)gkV8iPe zFQPRjk}$ed#jJ`Upp@b>9YNz7wkIRyonSCQ31ncGTcPiv$kEl~3ivjs*WM;^OHzBX z)1lK`hmj<)6UPc2QL&`3?Bc0NYBI_huN$VoeE%`Wbky$O0={xW>Q1~|N}=4&;9ICzQXlXq4zjpN<6})gG3o%t z;4_6iG5GG)c&PL8V6)o3T60QZU`!_fyJ;l#6IDiL-^iAsb2m!%B5G1X(xFPhL})&zGn8tboh_m>72#4-O!!n1l4U zK2N!LXfNcN1sAOWGMJ}*PSk&XT#Z?``iDdY54fDRUo;)=$>_u^W17jd8U)9eu-w{g z>giHk=%2RU6b~O0Y+VjeK(nOOz=RKp7LUdKZ)q459}B-9`_7WAcmAvXL2}+*lh}oI z#`ptK`&YCdCXx@+T{b|Lr<0)gs~Ts8x=|5yI_Bep#l!Di2A$@bCe^1uhKa9VJAjNX z_>MVPAs=ytckJ%Sm!jb1BWn@A?u)aOHjRX6`7nXk4P1^PEHK^&-On$I0Yp_Ps$v2f&b&IrM3I9c+3{?h^nZ z>AQYCSsiV2o{43viJ`uD;R|MpaU~$yq&FN3$C>jGf%)0^r+wcl{>KeZ|RWEcTw$iF^dAwOT-iFT-S!fBIj6THr>1rR$F&4h59lO;ng_k z4@+ih0)4-$X#vEdw&zhyeCeX|rhJG6vvLmWxCAr}nP~mW+5|Z?ZUZ z%HiGCxlvRnC!m3pzuS)3cOdm>@7jUX@@3>Z3N3fAaOgIwJLECc@!rd{QFex_Niw|1 zL^p#{&!@*?4@RgMs)|gZ&DmY8Bf^qFvVIKUounGa=V#V#;Oyes^z~pQ?^Eb-MrEHh zs@!c@Qh-9@C~H6&%LHT@=A*WtNJBc^{=BL z+V#il4*>Z+?BdB%P5x@tA?t~|BPqI(nOAm=Rn}2r-&Wb_eS}$Qbl0%R!HlTjfy2qV zJbL0hY0yAHfw0f)rVi>GjSeAPVMPwWG-ZMX;ba9o2@E99nU$V}a!rNT+%aMxYLEAM zMp}-0|FvruZ+{@e;A`w(tEgag>s*qEh)hmZ;n`~ih4U8M@|_<&>5GQJeBs}p zBUA!_x)yjE`xM=G-fEXQspc}qOc+c`%KVppr}ic$CeU+*%?Ap1NW^de>0unG8m=$l9xvVOQ6dC@SR(`!km0vUzE#z?y zz^wt_NaqJS@IhU88*4Bb%$La#7Bfu_uPAhHk--c8R<49ympQ@~Iolsvrmz62_r7Hg2Qi2{}cD z;HOm?stPPznJAxp02}3xd}ns{&s@J%CovOYiSWQvVXjP)z%Jp^#zM?~yl76?Hgu#Z z=mv|njU~tY?j+K5c7A>!VRVYCNQp1=ez#Se(+Y><*@?~0&XDKAy7D#hTaP880_j?y zsNGo->LKzC*@{-{stF3QkbVw!e+^c31)O$jHFmAJ#HfD0Un2CrdeRJAYyhGrG5HFd zLjc=qHz2EQ-pZX+`92hpJF?z5VwIIE=sD}b*?2%DIx!j0*>S54JNjoypWU2DEixb; z(VXV>erk}pnhN_;4m%iRe1Gp7h=d@h;mN-Hai-N%^R&~eubT~+^)g9jG4{Hd~}t!iQE+j|-Ri(8XSCwKOfMyCT*@4p{ux}?de z@r5Qo1e}}mT)8+;^*WvIrV&YYc=*!Xl@n-~$E$qo@jB9|WY0cehjr(Hz~cpo6wbD% zhyUIF={oO`8C%uLU&!>^L)f3ONH&~L>BBSuPw%EN~6>&)ftZoebVJ*5VkP3zBonUIxw%nZEo=Ajuy@8-G zW91^jZfbs310$)f_1o576m;r~8=`YF`JHSby816$;`R&m%a6FK9rQH{!-|(DFRn<8 z|Gfc$Xn4`&ug!*aNT#9Pec&dV=c_FogvC4c-~YRA!fzc<|3;n!t#t$o@!vNb4(Y#d zo^U$uQ+-|$9@vwB$EF3d8CmYeMUoh28>BHP~|pGJ|K~m z3_js$sGTck3Z5MJD2)vY`q%b@YjfDjyB_Z4UWdPOwn53Q4282G4m)9M@0gUy znHE*{p1@GBpgg1a`ddjaOU;DXSA;pfYx)93{6}~G&gnyOCUVVGv5p`MWuP?NQ>IX) z^cMRXU0!~4bn>!P)nP?6JOi3}jiEBH@UoI3PQZXqH%BWix_lZj-~V7F>Q#$trchAV zypHGOgCG^f-xCvUXUX*%?4Q!Se7`vAE94D&Dm-cc(?1)|1v~w%3cf`-Nd?r7c~EJk za#kwQnF-rb-eq~_(I@fc%U6Q9no%%(t$^WEO?c_xwtzlRCTzI7LA)R`;kL52In(vr zu;|hczFi{yO8SydDj{O4;5Av{i>j>xvBbShE0rv&h%*4gPOUl*q zk%Wju1mRvvf{bGYan>k5BRr4-aV0s4>T4BK09!b!PBFz*}O)4V@Y)H?jl?G&BrHKkLgoY z3F-jJcBj_v`iW4qdl7csMBDu2&e3fZVZytf;9V8PS8Wr2&aBXV2Xcp42}x*UVLW7R zkDhvvTMtCRRSI3Tp13zE-ar3us1$%51@S@t0 zRq$OWM)Gk}zJ>xF5U-Usa(llpl%XhXY@8C>&k5RbrNXcX$RMp@x4bLBv2tUOi)z7L z9^-RMoxtg|6hi((NM!0Q8YR$@v+?oX8X8&y_7&0Mw@&im)lGboBErLY)L(OJ5QFp} z0Rea8*8tzWjrUS$k&V8IYPg=?_@IY>G|HtidyMM30x@G~|3h{f1OZ9$?OgV%t*e`y zon0Bqt?knOGu&sa-Ds`yYW-`Kv8`y)vRWq)5LKh1?l${P2BDw~J?8e%bUrA8sT%0vWHcwdVRmp>dAL1wC$wejBqc@qz5lEz3 zQlD-V1SJV8ZXbErFBJy67I-Bpr2i2ZyZ@I@?C=GNNs5Ku~dr@W35f*nte-pKx1ANfkLs zNeUM?XDeF=OAv@YB``%;woi;?z__biwj`evOv0%_hf+wDq!&r0f!HWfiB6jtB%T)a zc3?o&g@y0FUdPkLb#Ast^9`)17|E9$9K+)NSDHvX4l;qPHLbxVoyF5oY& zOH16>=#uKc6~D>DPA2pa-;}BO#oFb{%746<%|ZP+n=T0xrG)ikdsYTohEE({O*zrd zqdO~8PBUNStbqM`S{J+8q}In)C3gY_W-}xz*q`M^VT{agZDwUTQVO1nql155#BZk# zYeG>ELEEWN@&}!Z-mg~c=H$4fQOQwUn=O-5i}~Q&@Hjj^E;Q-WiER+JObfi|VQ2k) z+c7n@DI?}V+|qJtb4ljX3WF+gkV?s~7E;=!1G_e{|M{}Jn`N?n_jYb==m6$JNzL@T z?jA~Zy~1P9WbZqVyW6vDuV7J547Wr(@@#IvD)43sGGNdvZy~iOALSR9|slXq*Bk`ecIivS=VlUa)0dPJ?}Ix zHqSFZH`{2~-VTGJulnZ0$E57DWT*%j-jbN==jBE?GDW;+n37r?DqAEO# zF<||jrAIJi&yAHht-@xcT!GJ$Ic$cOh?@KB2Ip?8$c$~Z1?5neJaREx=4Pdc1Fk{Tp45ssW!W1V{3rU5s^m#2z8~hZnmFp_p<-NNWZm zsC#7u)lWXs#Tg6+UmMFSDoVJBA%Z9l9dW3r3o)^A!{^yxHtu@KYXhM-9yc3lyYCYF z2gf;QW~Q&DB_uY-t=HJRpwHh21D}5-#nXz}_ouC1o^~{`QSl*u)efauyN@PlPiIKL z!ZlR+s>ZR+@Bt-C8h(=lm@Ym1Ee*||MHupxj>AMj?X#`9l6E<_NTH>bk!vfR{bkF7 z(1K;C{KLmhI7w5ETYheX=0io+aPiFHf#c2vF0P?Z@}7@!gtvxjKwCN={>TWSNM7hi8jPN)kk*5GPRvr+?AWp)5QL)8}dqxJa2o z$rbkbQ!r(N8f2$ND#}xV4yb5T1@MAG4bz@KeE$1bNoV|N=bKZ%-ey!xjpxsccY%DU zl4z(#&>><4e^m-6qLHBP)%IEUGpLt-9!V7#v+HeDG<7Aw#KfGE>F_0u2wzSzpNI-~ zeRvo_>2a2jfoh>jqNW;=|2IJ4H-F#x^1AYp3Gz;6qvl!VRJ>K9-qC3d5^0E87B|hd zT=w&@7n;nx9n=aRRCQ$U2|$knu3)fkXe2JxL#ezr>lZdk5v%00`1yg)9$(pPSEgVX z%`^BCVq#+UD=X4w`26p&=2#c7NHOtz=@_ZL5{LM2@=+Dm_VgkH+P%Ew8DaK!Ia1Ah za6J0C^jyn-jh4Pi08s&RnUdPpI*ZJV{jPKDZyRoV^CI&hXhdv?Rgj!&#led;0b`se zX;`z???)3Dla09*^@@UUf8x6RTfJ)22U!|=wP`uEuJ?{oHYz$7nG;X&65$ky((Z;N z3gGKjKJ&Wha%sK#D`w0^EbYc11o+Qmn~6Cuhf!PEiO*` zQ-ZCnPfPPvlPGZ)2b9S7F=k(~1t%uvTPkC%`YD%NkVS z&6PjD9B1VYw@h3kV?;tA9!NDf2(P(`J%bSZXdYpd{gtNwzmKqJi}+WhDFhuXN@gyV z&mJ0Xe50W9!+^~4{ovpLH*LhyT4tQEe4-E;m|{nx^>XnSv%@IQ{P>z(rIMO6Jn<5i zDlH^hCo(3)EM2N!YgaXE$Z!WpfY()%Hn&rafxvZn-7VTZe0u{Aas`}r+7S{|dv%lK{5+{U~GGW+_m3`;R@CGwb(Mh;ubH-Ugrh)`z7tbAHVu69RDJ>$*Ti%1==RI6N?}p-S-f-KYJsHFI zo<)joy8!@{vH6B-T3kMm1{D>JySa9S?Q?q8b!HYF)oM2oOm9=^Z*E>l*jqD4x>bxv zF?W0{g-Zto`Rf3lhpyr{p}pZCDg(>@Lbt-mu92(R04dE)Tp(bGOmBJ)Hrw^>Ho zSE+vYz)Kd2GE@`O_gyOA_>Gg{52h$lxn=4 zLXUi5_Z!~tgC9QESkIr5dS9w#R4cUjQFE6N%=a+F_j(*Jc!jXk;Ya({5~KV3zG=9a z&_8gKBmrO(Q=x2TX^rI0qz+kvEMr>XaveJD1N=i*EI-vW8W4u60)Fxxw!+e$X5Ri( zVPHmaMGaIuB^~%gY3fq0M%!Vp98o{GMk4LL%jc^pS23m;I|DQz11FJD3z`m}F0was z_HD{lzQM+&CVMfu8hVXg$l~M32?aCu{9Jr4;MD55YQ{xGN0b{QUGznk@yY>jC3*fi zG~yUjB^(%{F&PQ*jY&!vY{=N+aJvP~IO?Lz9%)Lqzw$)yC1uw&HTTA4A8qVyv61Ck zZklgl;y+#l-#v~lvSV)Hq2l&|(+9rJS2Al7?eoLjpASUFxV`07rYLM6%GS?b)O@KR ziLU_to()R!;f=yoD8_w{W~Jn&N9DcMD%Wbp0+`pqAOOq;qDPTKce#M47x{IJ zOsXID>HzuKiM#HLfx4D z8i!uL5jH0dSN4t%biPCf%2YOz*0+!KSQlnZOlNVjyH7YfkE z+6}_}NpgzM#Xb$~?UzIVbLw2{Bpeh;=AJqUJ@w~$^M^YqQ|$8c9-B#e$kkuN3PwP{ z^1u;NW&_7J`y7STpS&FrK|TOUHg8iBw*ccXSB!yFhfCb~Iq1E7g2 zV@1<+xKVa9wPBTI&YSx+>VViFbZ1edY=bzCD+I}cy9@X>{3N@_vKAf>T+TD#?|}0A zuVwM8Z9e#1Q7mHMeSH9sqyfRSCRSnZ@SF&^Tk$P89qVoXu>CT%HBV~BRBZ{~qEsAB zz{I@8vkzKNa7BqKbUV2`Z}Y(aF5?FcnJ%<%S=nT#2hQcEeRMUGmw8`dHQ-_4^%a$2YPu3L%~HCHvTf7lRnzf7tqY(p`g&`#^)ub0w=rePe9}%^KYZ48WWP zC*fS49x|6g+6)J{E#K%UZ~F-E$q+?M9Hb1;h#i9frqCF*Cks2L^h$v560V!kQXT7g zgIL<%4UT_4symss^B|ogbAAf1cOUDV;rL};Gqg&K!EUA%>>$vNsy|-aJpr@)qdD*A zl%X~I?p;q-yDOFennU9Kf{)^{AaN{G?D+}@wtW)I;KwVbNx+;WOF_`fS|?DT{=t6J zzg0Ko+gj+KC)kFJWrXWrw#fmOjuiadNr-9c3P5&TQaN$;n zR*UBatJ1Mx=B>t>f@KxN)!tx}#>0{8VqNby-FkxSFQpC?uCle*lEh5e!CwL>J4R_C{?{?)_u0yvOl6GF#rm zpZ%KOw&b+og7ZI8-sFXN(!D^&th1PR0PwZW?o`@lX6fwgS6NUNPXH#s%LYGfDV)!% zDm#cYlQa^wnk`;8w)kEswLJ=byL!9wdA>ia`Zs%zJ3%~Saa#I=Jm4M5HO}RtdZU=C zZaGbB>zX zEOuayXfkE^306m)tf~U_RZBWcSlIYqm)%Jz02amXFiC&2klN0*Ldk0}Mj6__MoJg; zET7Bv4-NKB{-p4Pk0?liic&lXN6PI}Hy4MZ{J>;!@R6zUaEib@m^@)~N5tSswo)RI z6Yz8RS_SWmvt4R40ZkoQK2Ucrqxt93M1_~(bJtUs!QF3XX@Xra0vL8MKz*M-!ny<0U9X292rb{(qAD`-^8>3tJX^xE;&zY64mY^f5r)RPqb+AQk@PaK9ogbqVVJeuj@~gkw!q zIWxv1Hdcet_HeDR)a#TF(^4hhX3e9ygp(m*Od}@wcc)5_&m$5RcA5%fnkA#9p~B|U zqP`hHqymnj_v1@qG^jzahh){0(NL*kMpDNG7D8;-rtqZX1^z6Y{yy)#L4+hC-T732 zl<}{kqG%_}>j-;Ox6P4-#5q~2mf9f!B`EUjk+z)sL|*A#)_9BS8cOika>V7VLLaO0 zAdNVKg2U$4W+W=-=o~2I6WZ6_*`ic}pxRwSB1nA(N%dcSHO3q6Rk*`zxnP*VsyaQOZuaL~KNP$F-MAq_t1OX|Aq0kFmD%Tdwq?zs=LZy~{?F&K64 zeT7M~?hZe62jP?Akwlj4Hg0D#*@$@(mJTuEX?|PtB3vB7PkC9bxA?AzB$sxXivZJk z@7Vu&v$kl-h%&^>YoZ&^eA+@Z@vX0%3Q&XAoAl=0o8LLO%6{#S0FRP%zH#Mn&1&{V ze#!bqgwRYX2_#aHHtIJZAi)i;9<2ckK@`^;>Bi_6FiA{*)Ew@^i#zZoPSp={w1V0$GKQ%W1sx9c?uxr2Z(@D5jk6{WPUj% za&KgIQDct1ovfIZ!F|pa z7OhdeWKH)q;}AU_Z)&nt-?*B-r6QFJXaatJ5PTCsxKtOj%v7pU9RNV#zIm5C@>}Dx zk1c+?R#EVc47sA$J^5N&Dve5TisvNglIAdADQU@~ISYEBAMaUz5E5B3{|KxRKEf~? zdtxcqD}yF#aF+pUS@uSDai9^(O45GWBf^(AqEdq4FS9&D#zx*}Sqwg#DL0n-=65;> z54wg5SetoA^+0JEpb#13k(g#c>0P7pJwZ+J%~Tnafkb=8Wk%Y&ew><|AFNTFn)JR3 zJNDL`ti*cF=gy<#%{V~$=o4_qP>PYA*LK}kp##-$FpJam%z+F#pLv8!O--#k&)k8i z7}Py74t+XiZ+7{O?LdV&}l>PMtKqk9Ct-rI@VN@?J#s)e&Jp)_wE&&tS z9<2>w&=UQV){w+tFoM?;T0L}r9GaXpIAE?_Z~ZKu{?!1r+MgRL|I7Q|_M;O%#ALDj zMVZCCwt?i$Q8gc!wihnIj{zIGJ>E_&qS)B1z1H$P>QR(5Po#2nF9`b+_x%8AFzL--Q~nj`Hf^ny44f zUxa54sQq9Qe4UhC!jNNt{1(cFVmF4~8qzi)%BWVOqAyvb!3-qKsHY_IzV6QW{jM%( zN~<3{LsBNT~!+HJ{`1i~N4EflQ0 zWUlqDW<-)@uNb>pR$1?EF-FN>=^#i3#8>|i%|__Fc~wpa6t%z;GKlaaN+4sbMp(h` z7lcT)lr|h%NTDADn0T;Mnt;PztSy`RMsHqeb&$_8y5z1NAnj#P)ezv6_WBgTROVca zXZ{rM#NmEVWx(eJxAQe4tTNimHU$V4fDY7KT-w3X;V1u#|CR8^cVvY@%K|Eg!A59; zmUqRO$^n=F5$o@NOF&3W#X-rjdN=#3N%3(^2L%+9Jhrg4>&|ny0R@QXYyKjpQ|E>E zQM$UvZ=#y7{;rDsQw$iWxvebhOjlGBuvN$VOT@q$?b6|lXS@5@Q}{SmUIf>9%Ky4GD@GSsVPQ>Ig4MFJm%AD)rDYe#B_&d$lQLCakERr!Ycp zrGlIPoYVs>>dD|c8#2yVG|(aW;kO&j*vORc^_>V50)_2wV$f_$Fk{CIJVwY?jHA2D z>sy(|c_p6b8KVx)b(>Y_DT*XUcNC>8HOX9tM`=}x4)+XON0N+(@+D&L+&nObx)fAR!{ZNLY)m0&^OO5 z2zAkO^m`YsSKYH)B6zN!rweLO@GYcnSC8U{uP1An7=lu~8vloS_c~+B zfGN1#wY>wVCft=`6K#?oaUxIy_uOTZ7ppN`;*Z6D^<*qQZA;r6D9UVcd!P+_3O6-# zN-(pGw_0GJqdI08CvQdV!hZp>tc%A;;@4AL(WSlZ2Yxo79?sV?KjFkz^{_x4C+|R2 zN5R%8SCv+TnS2p7;19%psf7lCKJwxL6oe2UNl4UCd33esKzd29(m@64a!?3@QS!xQ z1>4uzYSh5+1TJBqTL_dQC>0vW008)GktgWmpLH+H?1$^Wh};Avm94Rvm^(+NW*=|9 zMfSyx`bAgUbNAOCsJTNWjr7xN421e7B;)Ef5Oah*FoD?%IC;SU9V2`h%TzK5m#!Ay zS=#wM(BUlHTWKBSVz0sCshtmbo8Jd&Duttbv5xWBZMho=vT?Guh>Z{voh7}(RDIoT zhAb4L%u}j!&o@DK2Bd1)ID6>EVQR*)Kw^TK3^JysF66xK_H}#Z2R5Cc0M>U9#JhHn zuytIZfOw4-ky3#P;Iw!)Bh54k62KpkZ+#uqy&0*fs+=OEH0^gEXj=zm`Ub)n!c zkdK8y4nS`?{ksdpn-r%?!|_IqL0TkWqp6saAvJYIhuT#xdgaaQeK3o0u`(S~10l*`6T%J&pWt&O!uJ`sYaqT|-5uhk!23l&#{M7hc zq4QfaSq79c-^bA=Ep7Dcpd9f-Px5kcX~6CS1CnmL5h{+p-vpp*7v|iFks5P{k9_SeCC2SrVd+QOP_qw^ zcJ(*u(*Q3l5Mi?fbs-5vl2uJ^ZKHMq4p~U+U6(pp-6^m{ZMfrStQ#M|m^`#q=`x{@ zi{*fa>F?ckzGo7ACeFa!0(4ndCsEvvjOQ{ul=KPm*~Na5WEPOiAhIg@P%m=IDJK`t zyE*)D$fI-~05&Jfc}mT4V%+c`!)Df}Al-$~K(-uM;TZ#gjZwllSE>9>!^K$2(Ks^dnKNfcc5_DI;q#p`jz~THh}PBts)}@TI#}>^$$p zU#NUlzsj59)T-b-z{|VR1KURwz}|c@7x&|tGp4z~!EJ{BLh~QKnw-`+ssZ_)*)bTJ zk@V=+NL=QsDpN;$HZqB8NtEk5NgJ0pVd2oOC!4w?q7B3;n19MwtKju+91wwu#0N33 zL{R%jy>lOufrvYyRD6bn<~wgAYx*HS>~kVh${s{Cw82&VdDn|;j+G8iw8(h;D4C-} zmz#3LRs&kHkLElsP6+3o_^L^RbbKkN(|>zMiUgJG-R5Xv@8JvlyHd*M8+a7QMh8s6 z0)J-Xt?!SLeffMX$bso2q)AsOwO!1P=KVbO7_`P7h+jILCp*?%jI3p?2L{1gPZj3( z{;PRD7sr%AWHVSm*F7AZdS#phIBxpY1(H&6yGK)9YF1Xn4NBAI{o2(&j|-}v$3AkT zgffYAaayjZZ%k|25C!)rKDH1-A7po0B98)yNMbyvp$lQ_DfbNM>WmxhZ2)u*^@DM1*!Yv^WIdGf^+8rr98E9Pp{4o zFN77l=I+TR?2DPlR=x-kDxtT~TW#arX@XD-KVC^{xxiu5J51&$AH^b=Tit(s>RCga~yN4uXUG-$zqj5|u93R2(_oEQxz$~_d7eE|$?3dD zEMpVty>MSc>U=T%B!rGMMHa_MK1b~p89`hKB&kbr69j#8kJ;Xa!dQ+%G((J45KAq#1< zDveq-H`_eYVonn+u#UE)y|Kp)Z!D?j<6Z7R3W|rJJ2m^Aq;U9MJNNC*FLPfSzPEOv zubs%r|u^yR9I9k9FjmgzOcG}lYlG&B}un+qR!v0Sn;pLfuZ_n=eoV+-{ z1z1>l(S=u{O-7z(%k{n|WziNYUmsts}NCI4q6Rh`97 zKk^^7k0uw!uW<}i4~pv27$LcGI06{9G^HL(x3iDu=T;!R>Ltq3n0>DIB(ua1r5yXf zK6%!IDFDl_1p)BzaVbPRZA~^RTEampfz+A3x0_jK;~mkp9L|fFPgiG+{b9b2=qte_ ze0;pTx7mEB{=Fw3W9f&cZMYX+?=U@%1mzV0z~7&9=1V#if`e@Kd3*#KlA&G61`QV~SqMVLPwiXoj z@#Hbalc-sCiqtM)sSMbNoMCFc$e(I$z!c#G?q z&q(Ad@=KryuFm?0ba7Zg?r`iL3z1rj*9#FX?hPQGs%>D&W}ctZ0P{Ta zv7MQn1I4cU+vHEI=K{Bt4X-U@o-pBSolP9?oIFlyCUjj_ny(jU-;^lMO@d4OCTj!z z_bT<9NA!JNJ_!ek2uDH->o@g32R`Cd3kiMwOhhaSt0F-nbIn5~vEDmQSxZ1$SZiWw z9{(Z#4Pf40Noo`OGm5C#eI}j7g>NAW*RGPN*SOYN<44CW?w-b6h74be*?D-o;U`YB zcsffCxYjq)YH(3lu!aH@@-#71hAq%gHNOUXu^2h<6}wx@zUBA&h#5#NFD$G`ETq$B zbF9OZQSe6!v;!P5GG+w2>~nLX(s%BK11YL*q# zu+5q6VHqS-Y5YT3nfGwv$ENrE&l`Qa^(ts5l3a@TxHwXkCh%I?{iPMc z9igh?{Wh|m0Um&`OC*B&8v}Z*jAVivT}CY_cRf?hSy6Hp`PDW}?Dc-Pepr=scXwBJ zRrGqC>5a=xDflDGM5oM4Z(Nm?MLmLrW%XEci ziP}&z+IMa>hJhbSTxtyU_MXNw6{k7tU>aZ5dX1Xi&8*{Vxk%g!4S_0&i%r_Z)s=g* z%x|;ZNu)bTt!_K;lbWLtmWqjakZ<6Jytp__O}t#2v%F|P`&gWLAs|DWW;T`XMKJw6 z>453A{4=gRAKsfGi%jLeN&5`jmlyGaiCmc2VBPmrDJHY&3?GqO1MpsoZh#YdkZT=D zUs!nYAhz#wR>Gn7ypE!pw4~qc;Q4DS@jZf!Uxvp>hW@B^v@%0g3;uW}8*A{XvzrK* zUb8r+e`=hSOw94$@g<<9p=Q3pxlE&e)t(PH?)F>%;ruh%LFY_h|H|EScKLpxeIS0> z!awtGGM?5oPMnwFC+h4`56Rs|5*7n6g+a|wM`c*k7H?)lN~GV!PX## zUC8YiH)#QhwNZi1ZZ;Gv(1D<{;bPu<7nqMu7%2d+vxBgoY{}&BLL)B(0 zj`qK4OD;!wwd{@_TpA3<-QOe*0`~e1v>t3sX&4xW*Lnt}QL!zV5bW)N1|q_LMG82* z4{gNJdbiV4mt2rTJkNQtN=a5$meH_XFNMa8wKAZg4htLK+RECJCMxz@Mq&vKR$40a z0N+3JsG@82iB)8I`0XJQ!vwOD3?suoARp_kYP&5jp9Oo`w>Ds2>HY!aZ`5PSkP{V1 zpjQ|!Rs$x_{@?OP3Yd4?4(1GlZ)PVxe?H@|F8R^QQt#@Dageyo_Rd0vOJ7Uze`U&x zf0wtND*sfG6-OY4xN)tLr?rQJk(_C<2I(7jW5}Vge^=q_$LQF4@$iA)OFLND>7Jzt zcF?cgG1CB#IAh)Ig#Uk~(JN_&1f}8MyJ@2+M#++_BIaUF4YR+;c{I(Z9R_BOj*f44 zyKw%$x4*=a@fdL5*x_>FV5i@#*z@&41hJ<39^Ai@`TvTk{jNV0nF{{ZIrB8hS^39Y z<__*FoqI<{^%iSSW~T}Oxo<0k3BfaC*VP3Xf_!3PEu1EI(IW24?pMAfG8Y6g<8z4r zGb;ahixLZC-dqk%|xTC&e*@0Dr&YGH? zD|}4%Jf`pXDPaC74iGX3H?jpP4F)-skdZ z3q0sGa7?siD=zUwzLAVFj!T1*H`y!`zv&_Hc*)}S9X|$$U~P!Bxx-j8+lV%&fjYk+?R3_U2(zCj#5Fu}Ts-DlU5601tOGZMg3iL! z2Hc9I<69T~$QYPMHEcA>if2kpPUQdpewVLj^Q~AQuN5=!0xX zF6q|-S#iP9b0PZD@1q_Q;rkCrd#Ww@ELN=JomaAzGXB~^z89rM1g|8_RUcUS;k!}A z>bIBSKsfL&^nY(DZM-k!7D<1Sh#~#Zhu6Sd(xtwFx`wX9>SnfyXG0q17t=@m`|j@% zE(nh%SQu*wT228Z3B??Ko7D$`rw$aS!xCcvnWu^Bi9H55W?G^C1FC1A5yFb}+f$yL z#@{@hq3ZbLg;p88W_;mEEKFusHNVXGe&w{@@Bbbzl9F;*@{~fgj@Q$S7b^<2y+Mw& zk8f2T=tVwAH6PBaqzP8iy-ic&<42(&Izwif%@0f}-e!~3PssilASgxB5cwh5^NStw z$#m|str?MxH?v#ht%#Zf+?&+hTPYjs*@cqWe-ce4m)XzrYWBUwswWn^MM8=XYqhl^ zE-u~MH-TCYVdu!Pl;{=R5_#!uHEQXU^v#M z?F~0?fpTC~09SnvQU#6EtiyxIZLG^j!@X?BNp1WavZ&4Ta8!eAyvebIVO-#rhLkS^ z#^ytsKxv342}J);^FdthZ7O_+K+7)f*5dHAVq|7eff*kRdDok5zX{GSRVEjNdntm% z6OuDkPPRf9zSO609)}@<6hgU>EQv7rP2#DjeS|9EL#1GAkos^qd21=h;6os;Z?On> zGEKr0LJC?*3G$M&vX~UHD$Mdpe#;^D&zD4hgvm#aHbl5Pm1O-?a?*_7_%x5gk<(N9 zC3C7kR|-&rofbnvWu6O}zL%l+qZMwyOA~|93=~o0AVprdr7FiA14ImWoTWm!KoFw^ zb5-$*7mNztL9;FyY(7?l;LCP&Q_Ce&!*_#(^FM@4a+Bi{!&RJyBuhI|a09UaOtC|M ziu-F9-@44YUG@3UY_%!=+<)SdC=p(#^?;d71gSdv z&~aY%UJ4;VUa`u;csmB}AqUw##yD(HC?qeO4wUuQCRy(e;7&dc*`hE-rizlwV#9=3 zO^!>al*Lx0iIy3*rF}d4uuxrr(12$5Hvn?LnQCNFNPxHYdhF9&DZRQ7MropL)LId* z$z#|2201}!@mro;q2O?sp|kv<^Tw8~lX&%I88OSOz%<(toQZG%7z^%xqTO2?4_@u_ z2e3G;PrO|HOptkx9}y+>4reQm>I6lc%5Vc_gR&9_b0@;(dAx^h5LeD=1?3j|18@8v*Z}>4J1ncf|JkR z3}b4cH&FA*L*R$&nI=zf)ybDi#6M2rY2Hfew^SgU1hPP%!!Cs-(S^L9jDxBp=-%I6 zcAez?ohaM+Wex@8+)we$$6IBPpe}Q3zPVEJGlTBCrfHj0wbs^sk1$`IJrmhlJ6fY2 znmQMXMy)yqssD9!tnKaVSW0?UO#VBly2c5b($0nJ3rXOxZ)^n$@P#)mmfZMw>m>7m zb=!Ba)P)=Gpa{6~iJ6(JIRd8`e{K`LGT*d$5$1lUD^$qqDZf=W$P!T?v1mQa`6Rk!Pl&JVPtE>$jBjZC7?&-whp9YMWX*CAHNs5cd`D zJzq1_thUodL4gJtald9VA-Jbqb2Td&b)v3!i#62fb3VNrkjI1&b0Pu&`+%kwn<0{^yj#bri;2IIxgN3xG>S+QR}&u>n=_gBeR zxqV(hfO{x1F0-5a&>iuj1S5; zF}1!Xbt=A&Wpb`X6(j%x^`N2NqUO8Xtq#-J2nMBY;->nRX^ z%{Tfzm(^qfSNzSp((P=mEh%Ka)Akgp#q}=coD}zi6AspK@0`~A&y7p3e3ikr{L4#OuoTaV-VaGQ zY?~B`a=l zB<+rqS@i2(ffM>ws^O8it8e1H^|^y9HnuUSz7%Oya#U&i8*~Mb!N@3s=U>s}?t1q& zP8|tEi}9mw9ioGijG`n8GI#f@w$AqAW!Ah0*fF0GwEV_HKg&qscrj3xa`HHtyvyMa zLaYuxjqNd~leDzR`4k_Q&_7cj?&5OCi^HUe_r(ug)_wbtbg7-xlZaRH3;bBW20WU6 zLtkr}jEl*)&`)Qtc5PoC#Bqo^$K-@nF{PCG9knXG~)h0#>~7TWcnCnVDoqH z`Ui&0S966oKX~9J5Qew=-xde0tklEUmnug`W)luRy$C89wo;KvfUh;zq$9TG8Wqm( z0vSrx28M?I98f!-OlKEkoQ;rTlP3Bb&0`X$b!Mmp1dx!e+=Sh&>c=GeznrSCVcCJ5qbHdpS-_d@<* zeg0fE3n)M=+$ijATph<|jZj=V!v3f+j9OOlOH%>`Lq%HICXUoBli1=u0|N zyBSt9c0|EP&%U}q{dx-S`#Lc7%?qxfS{pavqsWoJBR(%`~3 zD?%b|rqHBByTSowTp+Z7mDf$XquXVS#CzQsPIU(HJZaF1zUT>vu5Q5sz@xnk@4{w` z*s;VbcBN;jT9$*2ei}5N20)IdY`^U>JUymj>1|*=oxI`|t<_YN|Gl&HLN}HF*nW*s@7DuTvpeYlg8TNvwB% zq@AfeP5`9msH?msq3Z2p4{}kSVLpgq9kVae{1WHFwimIgFZ%jEq%8TzREe+S48vaf zL-1;Kq<@>4s_Z#&x-0jfvBF-FMS6i97xtlH{R7e-(FQk&!U)&khN2g?gI4@EOB94#&3&vd} z|1FKgg`^$=ZbVXHXz3onz8mQIXj-+TK(veljXi9X|!~k zB^=@4XI$j~krTt8p6%X!Iwcab($R;Hu4ffM0sML7cZ-3Q9$yjRa0|56!*6ylpSGU;A z*>8j)RwHnx>y}D)A53*L`au4{?S{-rNNbzAqEEpu)|4HuHrE58MUN56zo z4JZ`cec-T}tq=sxnh06H?@#4VQ*D-Gau9Sw5==6dD*8xTXERu<)3LpZ^^>6pVe?PN zvQFKh^OOGztCq7Usy(rwBEb?dTb?K!=yF6U1c)nAz?s4Y5gQyqW`{kDe$0KdC z-(N-5&X~Boos1$|OJ)%s%XF5P`bsPoXRJOtQ}<4N_YCwg?25SH!ZTZM71U{SeTS+k zO)%4e#+oOZae5Ea-n2P-5uU{vObA)x?drqkZDM7XpxmRx=vWcupt3Otw7*a!s`b=C zE^WJZll#0(KLgOG)kh4OXRB)9J02W$_6j6GMtP1>?ygCDSLe0If#b%S6ifYJHdM+} zF%+bj-$m}%P!kOjH6DF-hj|eW#z)a|mg=?_y|!BhnNl8TBfT_QkU1QaoOcy$O+M`9 zva@q5X3~uyJ)q}>JSj;*KmkB`tltL)9X@z-O}FHRyc(VIH_1tIm9CXZ<*MyuNl)$A z(RY?PEScGyxL@;!LNI-N&UgiQYGMI$H~L`i7^Ta7ZHcvseSUwce!T2Kz!k+THgp&` z`NILU{2a@36jix-|p}5%Thq*zE((#eG5} ziNR-xebN8FX90qO_UAi=NgNzp7Js3j%+*HE_eC%4R0LX6Gt$D73H$dxE^>L;uaQ_9 z8o0a+qABWqgU#&tQD-|&0-R&*?;qS8ZL~*3Mw1eAVUNS356Nob!x_%RsPsPWKB#=P zKC9{b+}Q?73a(TsQ!$WtjdIv0;B@%;0X-TG-ds|EKyonTPhV7@ixB}T!8Z$G3XZh2x}NtVb$;eebf2%`2%ez_g-J;8`8{wy2S3DGhlaM`GJ+kj ze!1+v-?!tv=JC1JrtfX@t4biX`O8bBydxK)ohvzAHVTERF6Tl%A@FTY{ z-rrUC<_h>G)f+sl8d4=-hnnCi0E;N1bF8qa@P57RWP9Wbeh4Fvo%?33WX{v0>{E^u z7u{x7U-n<7ccy3gN#4}7V}t7xna{bgnY_D9Dlx5HFR`ti${BfQ*g^W7$$p8Sx%@_c z-23(oPHa>pzf;&#Dlu<1T38fxIlXpe)TmKfmA1R8gpQ13_gfO0tJRa&u|Wh`D?i7# zw(_s~VRX-pqpMh6)Yo%t|2)77e$<XNFfjkczgYZ`Mq3LhW)(nRT-vjl0Wh8Z>FOXyN!)Ap$UkNV$U zQFz>Ls3!u&quWYrN;m#I+r#nNyc993;NU8kCyCS5X5qP2FinYIeMsVWjl;zQ|BQfO zaf9@cu)_WT>3CC<4=H$igQ*{6WTq(c;G9vN=Xoi?w*!ACp%PmMI!1<)`|xB+as_D@ zc;~+*4xeagX~nWB#Ck~`9gLAc4#cm(>?@F6IXI1yuKhALj=Eh4jR=un@WGde70}9M zIV(^w-`x;4)4K@as@!|!l zIO}^GU0Z#FiSz?#HKz8vhpLAJ6z^-jm+B)M8yoMJhEnNpA}wbO@2ZUW&Tr*er8b|A z@$QIAV(tB>5*b-M>laxoWQGzbt8kheo;bc&=$TCVLV7OcAwjGNJiu65TB4?N2^-ik z%cG(WYgPTcnyaXBest2BErA!>!)7SdDJKL|$r*O7w$)lLbG#q^?+Gu`EW&`uY-v$0HdCUc*AffP zmn+gP(|W!U?dsyL`342iRZ#$+YP(&w2)!FD&g}a$K&! zEua>DLr|pA#Ph`8;BdPwI91h_%<=zPT%IF$?bBzU+`h12!(U*9`<3W3#n=6(|AM%B}dX2p5D`4!gD(^K#0;N?CW{(cx)K)8W^?B2B0j9>b-B%PrI36zb369lk zB0TXiUEo6fxys}z({g{YL_f!<`3sTgS(!fCu+d#6{CxFf+^=I=p0%KgxTh z38t5P*-tAU=oFrx?;Blg0X(0taiXCJOY(N>FdV{uv#}-&iu+PG#~edLjk+InM!O>* zyLM0Zr*lyVhn@aE7gu|lEsj(SM>Ackvk4rL_!Id`UwIcP#}AxJSKHGK1Cg%&cp$EQ zbTVIvO_z+S^xTq=xRZL$Jog=ls}uDF5-#SQ5Ekm?r6~M~HK0$sySt?;O>uXFzc>8v zR=Wl#V>LyCQ-+@$$B3mdX)rQXSoRN3N6$5D=4)*;r3~}TPZqaF@&vvlrKWCe9?MSg z{FrxGho}ic>7vx;*xcN_H=1KSR4bF-@O%IyvYKy6yrWZ#0PNUvrRviM946DLb}!1% zcm;g+&IvsRc=ngd1igr2!&AsxU%^8N>we<)MJf--^CQW!LyNv{g7x+NTq3|4G1BGR zP%|D~Kj09A%Vcy*t+&F;#=*laa^9>yo`@Gj5)IZ?<@XUmuTd5I9nK#h;VpE)^Kz7% z^!3(hXlQ*cT7<7Xy2O&xs)xWN1wWC%i32CT8!wDZ4XH=&I+4V`-FD@C`D2+C(VW-` z3EKneoyLUIVE>dj424!=V86BeRmx6nHo5t0f{3vx@ml2ixqw2#NUCsK??~OZqx=zN_!~ajw(HlGJjY6KKK4IbXJ7+sr|&y zQ68t+4cM7`zY#&a{iijaK-tst!+@h*b-{}IY3)wNYb!+41 z{*Q5=iMZ-xy~|QXspM_G3N2pvU-yC%J@`LXwpF%vFoc=hTgPOW^VG@qRc$$&#Kx~| z=E(mTo0{%!@BiYrH`Or41snk)QndW(nrZbUe-$f$k2yw$l>-U)OZ88BddT+&RK=== zol${ERg)EYXDivzC>_tS6%vt~m-W_MygaP-oe<5a1QKHtpsL!+EUmzI?B$wqfRj-C z7(H+Z_aRt5V|#V0PlZ7}Dl>sTlq5PT^w02c1eSLFxoCyn#hvwdzS1Xv4JAj!+dG1s z<7v003{!V;1~Pd$MLQnD24oT^hr6IXw6)(X@5=qgY_5@|exc?A4o=@mFj`f8ZEjji z%PnO{MA+tSS1_fKI1;6Rp4VzMg*F;t=T-mvMRtqs807-j?X9jxZ!pBnQI8;X$kkjq zg_|Hm198%+bnG-ZmkMNEuj_j$gp$QN4)MkN)rNFcgvtHN+PZ9&iejdgos92*LUk8A z9}z(Y+~6bXohCiNDrNh6pj8T!m5@ijR2dBKV(WO_$@s$W;Jf8Rv^y;*iFoZJ5nRsC z(okuz-*|8h_Ga%xJ>I>q@_+teMiOo+w|RYG^lZC=I#Nbec5xD+&IFptRbIb)9~Z_qFCNRjstVWw+1!yKqRh`xZ5|Fai}i#vh|+@ZeJWaHFIgF{%+Gpw+$+b{QN#-D*-+^BEfej8ybfnlq&+JuXn3mpe1SUX!J zlsH;go@&atrcQNp&Sguhu77XyFOP!4o4o;l2v1Jz=UxSr9;P@+6-;W4fOqjF7Ym7A_s2s5 zuKdjkCGM*8ZPw64&9~ct)xq#xfkHiaQ@kfL@-u6xI4I$wM^6Fzk1+P^8FwN#UMOz) zW`ZgD*eRYZ0NK3^V;{^$~Isa9ga^)`#o7&pJ>Z}%)Nab&~mWkp)s20PRP z%iG#b9)2cMM%JM$IzNPUJ@F-f{swcUW+Pi2ixN`t=z+)Rc6kAwfAjot1E3F&w0PPa zK5(zZ{6Yn%#Gdn&K$SLsw7_c%e>j~l;qdcOez9Bm7d-faTskuG{L9vN=)MGtehps? zG+cj!{I4rUD*yfk^_yDb5nyNNY$#90zGPCPpqxzj1GdAuT~fj2{la%OTN)aTm*Amc z(GG^LY)}H@klh@egZ+P2R#s+xf4a3wV+)3Vx~p+o22R_`Noo?YZYjV*$Ly~6hX?$- zIHUDS%6_e5jJs&Z_L$)3Iaw0D_I6AfMc<7!cYA{ehtrcK$um@<#;bEuOPAB`#ld$t z-@jLXAL>2VYr{wvaB(RB`dCbr>;Ai(HkQKqeNFInedlOs7yM*C*J$rKqi6#(*VK+< zlZ+C(V`)`q;%P7wo-?M^WJS2SIZws$dvyMIP%)veYqO4_s7!#a84lR{36IY}AX~wF zEysqd%h|1*8hx5IJDc0tTyZ#HTWVn2@&S&~_Z#0L@TXHrI?iOAoG`PDhF$d89RJ$I zc}p+UEd7+n$)wk8@q7>JcuEPrJMA8;s;un>IHc8rwXE~}#AD?*W!et?aBH7()0bL% zU{7XupA!#}Wdef%xHu1!Ni|{Dng?q9I_|c+r?=W{xBI>+%RMvg@s9c?wf2j>qcpUr z^55}aI6hAcjV9W|BO{vZj$b{M#eI1>E2ffl@mOiPRgUVA!y;kZ=>DvG4aob4#+nSV+1=mY+wbWxK680N0zoPAQ$BCZ zdnt9Sccj&&S)B%6MJ4^eeu|J))>TQ{EU3S^OT&BYZ;A~zznL^+FOYB?wpRHeiW8Zl z0nPgyFPeB_L2#U`MFU~4>QLfdRWzD7K8JCCM1UpxSeEaRJ;oE|(iC>9=o`k-lYRrG?scw88Re>x=0D zXyOn{s6;twExdxmDJHCpu};W#&>Uva=Hvv3{3c@9cQc&+s;|D;Yvb7i@SY3xEJP`u zLZ__{emG22gEmG(N#;ndx6_HJkrvF3e-U=&@rDXUEF8B6#o;1w6@YeCH7;4Hl0Ksu zkBK&A_>~5|FuStEq7AbsfHzzpYWIiOe>g#hXg34DXQPTRxG2rse{!>Y;dl3~c=Wzo zCJzSmH-!V1NOm}@b~X|c4fC2DACvAC^XiuJW<-4IM=TPmC`xAtD! zU%F$9BmtWfWlgPX`q6*>>>9AKPH%R=((C&A5t>d$W;YsN+@GnvbE}VJp~Z>DBn%4# zMAM~<&GAfT)y{RI!Bu6^3mM)>*XfoRPMO_B-}6@ zM%gbKxLY`k-5egyi=WM*I?>wPtF{2bg+|yNq4t zcW@|u^A2m?`<^*340{J0mam-}1>Plyaz2wEYWKoB*znW-^HZnAF_`6Q?=1#-9TVhC z&I~n%UWCZIPK<_bk#bKYMam*lghPOQfrv;Q!-T-k#-MMA=(YB(7qXu&7Yg^O%M=P%J7YQX*<3A^p#|kty8(KP!N=}#%}7H-W3uhcMJyngW&5byxIqbJmnhu< z2!+{tWa1+`%7s^=1x#;lyGpfKqdU+}D9HFC-lFp-7Ue=srVXx#o2>VkG+GfrO}WOX zOY4}Bun33_R`}r7)pFP&c2>*&hK)SkTMzI7yIbS{6z-8p__mm%3ILhvvo%E@cOnv9 z1F-=R^K#dI=WVdn`^3nR;0cQ~^>UR_``Rk`%rOeL^_^iCy!d0G?gzB?+^fXuGwcN} zdz!qyYtpdwbn9pCv=4WGi@$<^Vn1R$JGaBG;b5!&LgTG^P7WC&5=yoZsrvGg3#AK& zoQ20NiS)v|vnup%<(02-GBe`2|SpXn2h9n z)_ySB!9YD*7?{23*x28eom*X-95E7l?ufQpu(GpVORmO}O^rZ`(VSmg+yHd8vnJ!t z?b75@!ff@7%=ql`H`nD|VfP!Z?2nh>EX`JIEFWy~dE24!dcYrkwvj)9lQVmiY7yhL zn6X?=%MzNxf*a9x$+4R-q%?wSa@gDHOAhB=ZwhocD>m`s3WU{B1TTn7xlC-#_egElX)bt z<1IbEM6@Z8k)mvq?q$;H2UwkiyX(1mjVX{(W_26Dzq^Pylcy~Ng;zZ0EQ04VB zsh9cerh8nX7Wf-XAXjwR$&p1(i}x*9I_eqP6YBnO56`{y7~TR}U1phz7dhh?Mw~?&PP##>@JpN>gsPOROw;=&8Al3Z(B>R|+7Fy$77U zp=3(Mz%NYkVO&X8E-CRQ`CAd_cv7Z3qx&wf(nEK(n+hVbxnx}CJ zxTuG%+4$5|irT6@sR5+tJPf)bEFhSKwyje&iFtmAW3U85)D6H{t0g3it)NhULPAB^ z(KPso>g*hvGJ%iu4)Kp5H3F4~hXYr>z9HL3>X6P~=ng#G8(9xQ_j1x(-O?N?KfB+X zp+1+8ykcquO8+YSp-EeHiFZyydvO8NA6{%l<0tLTM`>?J!E1;2L!ShMK?kv-Bo&+K z`~I;ixl_o1*+64!7GfsX8HaX486loI86skG3=dTnM&RRyi%jsYm}-5$hOM51n78Otvm1!H+LO(8SQo*RUEf`71lK4;crtXq^PVI8>mGUQvTjM>;B}jk=q zDm1NWQCh+w(3Tca4W(m-SCy*f@eV4wDO~wU88dDRn+45`9DjcKUTz!tYERTVoB8%Tb^{FANTk8L@7M9AYBwglP(GeQ%nl`oZsDeY>QKb zY=30|eBpkP zo!BZrzOZ!S#DBbCOZna^*0+wSh!KxWlSje}f7}fm5Cubvc$NKP?C0*Otr$I&#b{x! z#Ha$)Zd6gGU49jv@Xeg3^wyw`rLGUg|MaQL z_nm1)-Mr`cG*)BaBgdIQcz6w1Sii%-np)hL<`;U=_rPLS^MVe~CF=hf9{SkT(V4b< z)1gyP8*O)L?$m)s^`qJFh=&4|g%J%!C>H70igy+dit3#o>YgS46P)zxrjxPC@Kl+{ zXxlU@JDD}n^AkP`OO*l$1k!(WOea?h_|k?{_(G|w1XduQq5q$WN9s$+tvos$$&00E zHSZ=drqo^xMQh+6wDExPHd zQyqiuzc)o7?bhzrFPH+aUOlPq)k0o84r{a~@-d?FDy0UsvP+fzPtpMNinZ>MmeIDY zBOJm)l#uJYYfS{lM52gh$Jc{~MGXr8EdH;lEBN%tskv?i{>5FGz*Tb!`*fqADMPHU z+d|Z~Z9hrFJLPDgHqfIsJCXIr9~&KqMO(;IOQWJU9sMUZ`sPSd?Nw)<3$Y-gYhE5T z+=;fz#@g$tV$}v0>Wh~gfwv{m;~ozdLDW{}RD=0iP8dg;#j1cXz19!%!%=VnbZu24 z>Hq&U*V3tjgt}vn*J6%y)-deRo`fB*B-R`heN-)B$!H3l4zL*SNtyjt6k2+_-d`MO zZ*j@CB7?w@dT?-&%nDSt@0>yL1s90e0-W z-ZP7Ww%=`%<(1ZB4_U!_kl?zi#GAj}VL-7@$4V#7StG;=ivR}*C^dJrajE~;f4&H0 z^r)B503JRzRyImIed@6q5bM*-MMZT{6z~J;=`z~4C||vvl^-gpRm9eiE%*7FXqq!p-| zUZN@dH^ebm$Dv`N{Nv4if0Y+IOB|f5O(rYVboehnJzVi9RjeRuIj4?gbR~v{PXTkp znI}EUOS0lz>JE0=oIi58iLlY#m@qE5;m^W=8ZS6LBCYu|BV!c*w{I~Si}NL}-wb;O zCnuk(qNCsXzM1cXXJe}gUbO$8arEcHo$z}uSa*%SPFf{?G-`6>W;RtsI!(Vvyz9ln z6L4b_FGew%&yCb+PM|E4__{NQ!W5*i8ew@ug3J01dIUdVZ#6XY-ri2;vXRTIpraa) ztQ|ovRQq_ZUVNUj5FHuo$>+$Hkl;*p__K@mNkHwdk;T}I^rP__MiPL%7+>dlgzwXbCq<-=ftlRqEWR3|X0(36tS#c52t}rS$ zWFEJ|ljCshZ+yON$yO z5Q)pumP`t*?YZgh98cZW-Enu*>jive!yU=pkOGE`zi2euDkU zQxC?})X31#5Y=+ad?PRv%is)1`P8hq-J>^%1lF>UPA;bhqy2Cp7I1>NwJ`cWX$M9S zQ9}#WKs}fpj!zO^St$95v{<@DUnk<MXY88;SaKB-0G7+~q0BVyV`@Nrisehila(B&$6WmQL7d)E4&5s;sY4c5 zliFzgapKp`@uEcyEh#BO3R(_I)5!)w@O4-H!_|~`b{sl>TW`cgRA?j^FJ!Ve;0p>8 z5)!a99Z2=DsZ7P;o}+1ZGx=f?rpeG8Rjux4&sSkzFi zv9PyLW3XW*evcQ~pnwi3{0#;sFtt04(8|uan!?sb(yF@Pgf%Derb{8!(D}H-j+dFb zHI5@8;cKi0DJ7##?*J0cqZO}Hf3``R`(h=!x#`g#|A|-KCJe~oH>t`98RdZ*pGbx$ zY%M8ix6Gx_E#{sMcU_leLQQxL;rD&LOxJ8&Y#h8sdtk-TGs#~cO^RaiAG zbF=cUAF99CNNHft?J0hlpf3XLP+^shr4Y zZX}W$3jDp;33(zrb`@Sy^J@f&G3o$KLMKpG?7>&W!cX6zU&i7nGf zCHf}Ty{pGYtU?r?uQZ-+>#cv(cPQ3q zt`>U;7uCyyjSjcJpI<+TX*B~YZOSQfliU2XlSR__ZveDHxOfMo3JDq}9n#+a?f;bZ zl9mTgNZ_3I;9{i%VE9kp`FSqj@Cdo?`GX~rfB*>rWzc=)f!MDG+sVJi`b{5fT$D(;Up#Mi5d+4w0#{(>AA(@jEeNQbGYb5Y4!A~ zyAsFV19JL31f27YE=*p%ksz_D{0Y$RaMQug$;#l|+Nu<-4m1`Tnp|L3fFnXsRH>*R#>y^RepKh z=51n3{1hg3zU5g#=5YyLePi8GdHD^X*c!|;zu*Sx^T_9MOJJwXs}NFqyV3Xgeybrc z;>Em*gFN6%-ir2XuEhzhhF06;Qn2h^m+4u}w;QI2Bz8$a#@xxEnU3>sHFtGgA|Ws= zLKT1}At@{(ak(93pQgYb?4>#Bz>OLUPGt3acO^h^#Rw>Ac8^T=>f^AT_@a{yji0nu z%kw521i%DNwmcXlyi^4nQVf~_Ps0og3)_xETQj*|wxzy^O65q)E7}^qd%o-=UY7py zIU1em2#AS2XR=wsU1YmfC*%PR^qFo6aoaKR3&|N|30(pMrl;$c%Z?-0qG<-sC=v`@ z@RZkjSk_7~<3nJeWF6$HS?J>sBP@nUNW;%I5YdT8z42@}>EbVRGAxcVUEwO1urFUJ z5O1TLo%Rn#kEK2V=psftoWcGRm4Agz^{aS@lQVXq`^#WgoEsf7szwL6BYY7tdE!(w zaZ7rd5bxOSd~>n#ywIU_BljIZMG<0PYwv4WTx@lX_I8K$vpq^21<2AYiA%;+qK|<_q4VSt^+)Omfv!0iH7hR0{r(GrI5IMjOpf;&vM9i(L+Jn1 ziIKv|9`|JF5nZ@nYq{uu*etuf+K$m!=pN5bY~dhh*O9@@dF{^doGXcYgKE+W$-9Sg zbt>SWLM=l@0Jtank>i)llU{yYE}F>L{F?-GqP{Q3r!4~p&Vs5PrPb@_z4P>$_rn?i z#&gG7)XRH3EM>x^&_{kY%(u6%j<6t6%P~4jaI)0Zr&SMdg#=(M;zE?m!GK^*?g5{T z?Vu$IV@AL;8kLygA?Jr^k9!d_!4n=t8U3FI;+cq-x{C=y58R28xgP`UTmuj2HDXd zMhg_h&UOqNaE^J>S)VLOvqT~FZ|B{dM^-nEH$M??M43($KSdO%!{&FEQXgfGQU)!r4`|G2bCi#2L6@uSG z=|ig$yccqAkMsebQ{`dkHQcziZ~668_xI=VI>|cqc{xAD;xwXbswc>mLM(UO{6Ykq zZXBwG{Q>1G*SgZ2V3JnncA%Y^4kn#ZtJR}JQ@I(KSm&jaecgX43zj{;V*?J%G9B-y z0U#Z!(c_4RjS6|^^u#hEX(^Mehwb%}1PJnxyYbDSoY>jeER}M7uU2f^OG)PH!0>0= z13SOuc)8mi9${;Kj5TFFbP&I5wY!ONl$Bk^UUxCzM9)jk$R0q_2`QzMp;ChY2-HHm zDd$n5705fzwDxFIc){Uo7=3mARA2P@(Pq(dlYP2gSBvSR?Jxbrq$CaJ(uFP(nfi$= zDVfC|2U0+p$)W%;=0La2DN4{s`>)^1(*Af2#fYF~L|$H?vsONNC{~Q{0>r71M^<*a zb$w^?S@UD9wHl2_Nq&477UO!_?oKhl`xacT!(J)JYP@~YTg^$(F>P=N2pZru(5xu@ zkI%1Iw_zE@X5)80(7}-kXItr2#fbt3=-|Mk2-u9as9MG_>A!c_V-RYGN`75pkEwGN z^bU!>gKlj*wLiQkMFmc%om~8xke)hzS<_ohC&?2cIC^|fpj(QEB4{JkQ~bFmtDw=YHI- zx{|qJG`VYZu1<0Em&+OrruNNBjbfU%ip+%O<9x;%>MFr+YW*;AxbLS{lYz#2a%9>qph zGnBp-sNXm9BXhqI8e+t@WicQr2UOod*+s!;{?KXmR z`CyO4e83-wAjD7tDU$$nvaWr>b{wA*{yztxWYd8%^i7mY@Aa<5Kmt6j31~J)8r-+G z0{?4U=~8QS{=0pcxG4lea0W&WfmAO@zQE|VU)*W`0TEW`%KR-4D3$``ZQo?b^PH(> zx6#hT6Qma^n0osyhKzUigo1}BNKH2t@BnXZWB+TrSJQ5dh=ln4N?O7C3-H|iflvBs zV^7Kn5QwA!t_=SZpsKRcpug6;RKGe|Tm97H?TPdN{EdCWnL4F&dDnymAPdklJJr9; z0N&TtpG^n+&q(>G8KTBwxqu%)m%4oRA^}Kl?L$rzzy{Doa*_TeK>8~WIZC7s!YDM^ z?E~L_2)*a?9xQG;!s9xy)Pw@Q*sS~CYn>b^tu)zPvih8Gm!kmEg=R`JTtg-qmvwF6HYrl~OKp4zDf&&5rrgp=z{~Lz#|GgQVW;?ERGw1ik z^``wzVx2Lyi&Jzw)I^Cu-2-|=7JK?R!q+b z-i@G^i={-&W+33xt78d;+KlB@AuU;hFofu`-me#i;?=+kKrKV6%gL4yjH~b#eh+s? zbujOZ`-(?hKr}rB=Lt~@xjNA;hW(%FY+>0P+1gVl6HhE}?RJKLSZg4}h@Y(5{mSw+ zSoEIxN`OIDQiMz>yX%B;4hz>aCQo20O%gJ1YEo(U!zvOq~7a+nehQcRK_}VAagBnoZ6DWln%r8LQGMhC*}|Au>ZCz?eel zV&u118+NjGiW7YK>y$;rjBN)eosA)Y+c6cWtlFuh~M*-5g&j>JCKxCd>tS7*k!ii9K zoX=*56+e!PVg=tm*B9{_Rvtv_5U~TB07XONnC|aRsI&+%4jtckfcOj@kWFjz5?t&+ zlgg}WtQxEFUIgJ{_Tlh}ak=H;S6eEoB|xF*4bSqsVE9mSy%b#DETI<3m!z1id)}k4 zIL-%1Q`ltl};O z6zdh0;MqbQ&qhE#|CFwm9fxCI1Yz*%L$KtN3h{GVD~?51eSdW1jZ=!D_?Hdr*6!m; z#S;DQIaGa6;^sM~4KNF12XBEF`pl{nziOB-A=CYkT#y11Mwy`EFZiHhfz#~Ayfo1p zE0)6W;*SgORA2z0nKj+_k+AQ1F%0W->Fz@^Q z$1x-{bOwNhL$DCNgfBs{?{JicG~0z4Mv8turZsB$ z*eF8P5mE^jvzqWRtB@>(SQtKu`Eb+dgC0AqBpK8%C2j-{0xc|%iq*sM&Q*LClCnj* z8#85$KH}cbnU%5^XAa?JWL`5vdy9V_ATr07R?}92UK%PqOf@Cf-65;S{f9ZOmURGq zZo1qx-T_vv0bHE2&S&U-hQLLEXocDvJRG1cv+(plH7N3)9E&sDK-RVPVApk1X$Zm< z-wi-zAuD#75gOnh-imzj63ON-b~}+XV5>y>L9q6#l5arThqy?P({$g*FbTT2W6&Fe zI2?x5sO{-OLK$t*JH(aHc3To#UJ^zo*LwvKy~y(WV?$ZA z>>o*muUH2`?iqw446v-^@`JIJf%X(1bp*?;6_E_e1Ubothn0rBB1+`EVWe{&dvTdo zCW;|sp(L{0$8x8o!H6nQmK+uyyJwry4~3hOHR-ivuWdvL0Tjh<#r9 zTq(g2s1+rBYsDF6J|a%>LyIglo?amV>dieJwP}M0QC5zaExXCyW}y_}vNLjKpTfcZ zgccxEE1~G!-Cs&jSjo$?8^7|`E;S#U{W!znb1p(9sl-(Bw!o_Fzq_$7t%$TNj`}_e z?`F(62o%TmRpd!3vO26HwFa*dW&-AzD~M?qVzpmuvP0wGnPpyUs<@_4aB0oOumOey zWKNSs5^Z_W*KzvmXBHzN1JG@GUkf1VldHr0;@T4^ghX=Fp&zftWF_K93)?At%%Smv z-X-98mGcz{+J0j#W{1jALiq;ioRt8J6`q-03lAmbD!XW%KmuVSmG*eJ#ebdX?q10zL@vw=GTlrqX^cBaFK$lTkgl2>TMzmcb`BK0)X3<4p=1wT=6D2OF!Saaumc= z0SNb}wW{OXJ~#-IN&9V_T3h{rnw39Ld}_?06;H^nSI_7Tr&EKv*<(=ReaB7E6JZTC z3UWTg^DTy>fu` zmtgvKNH9BPRc|&r@}r41x}YcpVR2491GlS=x5}G6yH| zI@5Ry1W_yjiHCvrV?M5~Pwz8t%uC+b5jtMjzha89w8<)i)X=D0l7JI&Ggmr5#`-&< zGGVJ+C*Q%zKjPC_(g7p`@cN4X8@0pv60Bw}q9dP$V&FNfm9Sfh7fK~)G^*K+|A?D0 z*FTpSW1gIx%Hlf6`4*;F?%4vQFa&vj{&Z`(?ZTH4ya;$nK++%}E_$0HDKhP8o^18- z${m?9uKVE}&;Z&=0r#KMr!=Ms1McdLT#5a|8`icI5>?P7JXVsRO)}77zYuVPpgi;) z&+l>#5x{_fJOurJ%n`#Kgytu_V(E^KVT$fbB&2P+ksiqoq3 zNb-s!T%-W|IhlQ)S-q1MNREytv4uj|$B8Oo)y$y_+v#q6bsmQj{o*c^lzWs^(Gof} z*VD{#AtbZ1U!D=|AVOG^A+d#sI1xf)9y4pgf~2 ze-0CjCOC&QfaQaW$|8bUNL~&k2oy}jwH^`pt3eE`>cpV?JJ#f0{<*)N*%5gT|sQuPfj)&`JaZlxgEG! z?Q-DRf`I#`cWJn|7Ay+@4({I*r>0mSg|b?t^AFt9Y?TnBAVyqW46xT>x??IGHmJuj z5vKI(p%b7_lt_+^lt}hpVR+(xaUH-v=Vp?i&O$DF>4g)g6te5-u%X#W^RyUU6z-5Dvu%_y6Y9k%a6h%=XI;Xy`-QRpC# z?U-cY*(nGXuvQXNOZ+uLf_;SU*x$oV69Rj$eGHACllbS<`eSM5ioc{s3$E|Qguj`G zrQP$od3i>gaDyMw+vgz~)ZgS6!3QqBuaE`6N(ovFc!T%FDcj-q$;@bG@W{dxa>u@i z5=L3ERE~->gs_ULlsA6Momwi@e~8b;R1K#^S28~o$C_e~WB0(ut1S36SELEis2G~i zAO=DRf-s2lt{sn+?q<-;vNGd--zkNr9TWxxexT*v40zN`R7yj^5HC>fhAB?Ch&qz0W4H_ei8)2}Xt{(z;-dW|ch?RC?5-A?NB>K?{G6Ql& zacqa#Ak%w!9qOE05pt@SFh>4AG#sj~tv23|u*F+;L&hpj$OCTrS)?J0ce3YcC3##2 z_XksF8SBf0=n%>Q8TKdhQo+94q0+uRZrmP?^ap*A=e|^aUXh-k zUo`l>=PNVX3cZ{G!E@1{;Q({@hYc^qG%~&5QLBZPye5xpAx>0c_N~q zkn+)E>g$t0W!zug?|$<18b!y#8`uigb z!oms?HsfCS*kDzf)vrqGQHWSvuC#>;nqZJ|#pGXoSV3c&Rr z?_y3miB(?FjXV?)gOEBUrq8Z8nQSKAG3#1Qv5F5(aSoe(a7So<9>!Fs2W%)`>l@X2gzDiEcp+`f$R&!DOrtdT19Uv(jvoi|A{e@oLCuCaKGB9DldPf6T{a_cb-g1+3G<*Ze)&8gv z@$&88BR@{qcY6hjF@G@~e&mH%(iE#u%f|fe=}+uHce~u~`7S_PuvnNfe z!yjS4`+L4WLq`I4XKR0sF1$|ln}!5i#xBlZpj(MuV>;~g-r>**2{%AtGO9r|OqK(Q zx9D{L^xIt+yUTt&MzBl#LuzElF_uU!q^R;@07eJQitnGo-jb|hPI;;Tzko1Ik~k&h z1;u!VpuxoL!r?|~4)Q+O8PD5KDuJ>ez=-Skw4_y9K5ie2H}iOln~ljWn#LVeqGRpA zd}SVmOKAbcIV`|I9#ES*P-?~(UG-aO;tzM$Z~$b~rl$ImtOw(4)V!E}3-nteVxG^_Xpj zx9h7l`@t{e`||K-8g(TvaYChd`bb5@Uu%jpobXBwcl@+?myi~mIRK>$|6D9h&Mm1l z^XWmcVFh<4ZSDiLoXA&G3|gr;3@NaDi(+&G8{!O|4^YxRsaBTlw;M^|Exv3b{p|C!#*}N()~525kLl>Qm7GjzNTM z_MVJkw0^?S=y+F$51%r{J=mm-gRn?E{Bn{FV~s`%+|v3=3#sz1aIl2QOQ^?p@Hf3` z?{g*LfMJzlwF;he0EAu!owJB!v1-8SrH?aLZ?FG`++7Ac+(W&bYa7`NoHDsN zA`G!#*4+Ey(b(#Sn2vA^q+SAv8It1W?4Z_=JpQi-TaTN97x z%)y|37jrW&k0B<^tzIZ+cTyEAEa=zeCkdq-9QEMk-KI+~7>S8ZsPU`8=3Sx3?h7jB zceFChJgmno^0yb=`}tB#)4A&UXogmnZm>ogr|-&b(l2`U9W)(xo(O0HShDV5lEcvs z=FocX!Ot+Q;TJU#ZLyBsT%%2}z5lPcuYQQ?`Tj;x@eK+h2q>jA(#_H;4bsxxjWkOt zigY(DAt}v*bW1l#cP-uB@XX@p`!_uMgR646cVTmbUgVKr?>vLiZjB3 zJI-#zJ8UoNPH0J=Y#$BaE5AP_(Qm8plFR?i^i*T?r-RS)6-usUox1xBI;LG6^-E_C z3JsQJnXUAUR3T5h%i((KPMBNhKPAh|*)47;NKY;T6f(OLV!O2zX8ThPJZ~Ia4z*B1 zh%gP7=K{|!PtSw4*f1M%mviRQsGgD(MHd+VZfU1tHr}Ifb-!WkV_e@|xT*PGDcqM- zEhN+%fO}AP@uqGhp)EdgRkg;HoNl!s{VWdE+e@xor6KqmbePuZ$NYU zaj;aR&ItYfxTmn{XtEBd1IC~x5gEM)>wT_eh6C7Vw+^p zVuyK9SuhI^M zypqjl&Vx7v{`)l=fwsTxp9cMraxNT0Sz4`};H0SPMmot(@7&RXmN|Q6`-L8n18_Jz z_FS1BlX6~kTfqk0X`hO$Pp7R$C^_h=-b&QhwCv&&t3KD?)x1-9mi7#Z`$A8vbo+!r z&!5obNAtnjOp2S~=2=?}ZtF89YeRaEy=1gH*ox*Xf6O#@nwOOrjmV5LjCb{YbTd6N zRpfEn5?N|bSX_+om~9c1mW*bU%K2-wekP-`xU{5NY6r1jz3K`jua+;9F?v;2_%W$H z*T5!aHs|(p0=xUGGLru?B$M2Ru8&%zdQFE*3L!n6&Dm?`2d_OZF1QlPt)dEey?2B! zu1gGdResIxrQ9~W!ScZkinP#fqOohy@n`xICw1_)Eo|sLd zwv(Sn9&jxd9+tl*-Whs5SWxuz#OD9fr}UqThEvYq0*9JMFhd#3vH}j)IQ%j*_O*d| zC8bgy7o$xVSK%)Pj{VoS1X`~|3R?u)TtWz+MtqF*hI6bN1*L_Mi|S<&#@D#&Goy?5 z>*HG58Sveb8qRj%dhgf%9BTmwW8s!~GE*~q&RS4cQtMq|rZk^Hh*@}ysUn~^y<-G3 zJWK`-zcuSij9-nV&>(!_y_-8_Xk85odLqIU80h(gjX;v6UM4i5e70r9B_7u#2P>zG z>zDS5mX1iM79^28nn+M`g4ab}7-vuTQQ= z;6y?k=4tTzMCTQ2pJmd;^b9jZB$!h!PE*#_@_%-4KN=5+?Q&d=J4{3JSYxH0vFHrkH4ir`;ttlx#0XhbSqjY)x$c1%aBe~d18%hsQlE-1nrosNBTjwEd{~Xc zFXmWtZ(U@Fd7FQaES=eJka@o3BK%?_m>l$h5p88V2~(&enYsRAk5?vz+k)?&9MF@x zAI@#JIxw$+FO`Bsdx^|D{^f*$&N?{ zeM+K(xZat_z|x0s_+9dLZ&gp)B|}Kj`&BjS-o1DfvhAN9Nq#*?(A7!aMqQlZO!%}h zYT3ZXO~~{s*}02zIxbD% zPaSRrZvXdmwtu*D30fl}tIjv~eRbA#@#h2;E@@(8+&i3eUDNEK%fHun&_SB=m9FhL z>_Bd>cqX?{qutRw``232u=TSAy5BDK5~B1}Z)4=I+e(c_LV^gs#8yyWH|&T4VE7WA zS*pSUJ!4X4Zl_Cf4jqJcRh|=+pg`-bu8l8 zE!DFyo6|Bq&i1-bX}4AGe^C^ZJ+w}ob=VmDD#8|Pf3P=#RcFD8ZrB({!oAWbl5+dB zv$Dc#(ue{@u;`Uo{d*|eX!i%&CriC%5%SJ&UKQVhDrV>`m8*IvD_g~tP@;-@vEeBCinJC=V|iH(!GsM_ ztK{Y0c1p9_vm=ZT)Th+oy|Q~Yo^9tt>!mwS6veIsbabI@{g)BD_#tU}fjnEP4Zc1N zOlb3mq=Jjb$xNsNx0^fCY62%Aa}?~BYWTMsFXHwll^noOvz z^s6SoS5l-&Sdv5Pg#7JV3Amge z9DXz2TW|3MxM<+^_@^eH8|@H>7tuM!Jq|kKX{JrCyjHNls;m|dQJTF-gng~IQ3%eg zWV7f=rDm@nCZ^7uV^{{99f2#~4c)Gp11xZQ{;R6E^B`%tqd6_`Vw{%tR~iY<`E%cZ zqLjIcE@Y@*cOhd&?G#P@))Wa0yFs~76Y~kl>6?!!);$!2V!D|rEOq9sNATNn(OlaVu zQJ}(aCG2YVY3#t(_iM?_D76?Xt>mG+C?L;J7@)qjvO;BbGikKgn#2B6-I~~wL%+E= z(f!2Lp5n5C$Y>{`?8!?3^j-IgF*vzeq;vh1JC8g2VN4;pVYkGiS^*e_2`fHnX@pkd z3EbeCWOnO10OB|nOv>XkmRO&%f2?p_0WH6YrRbat1-ZR><^S@Hf(TgFhS(XpmtdE-x6sw@Yb28UDD@|fk$RgrfoSCfyTJcM<7wkvruwe13u8Zii_0ucg#m0x3r)90Gk zq1;?RvER7;`K8>z3ev8ue6O6KKktC&fUDAP&EkIdIz|U+cN=-=r6b$Bu8Y{}^a6Uf z@&^mc+Y*Mt?BUodQ_J^&ggS@c+c+_=F@r2{*g~03aatd4NVE@&ifd^~ZDu#sUahuz z3gT3U4mT+%d-cS;HIJ_#t#YegKr9L?=-XwT%`75di2~#`tZUm$pgMWWaYjI`w`r#R zy!O&mU~M8)WsbQzjN+13z4n1%Kdk*XDfQpY%s1}bxfnMiUGcH}a{t->7xa=lyzs+p zTkH0$&AqqElUz$msih0};}zQ|vkwpdI(BFbo2qgQJ&E8?^hQblWmzA39!1kr&Qas% z9{@BqIJE7gr-PD;!gi$v67qwD`fAP0(v0)(-@nBb6(+l*s6ZS^@)Lqn&Q0Sdh=7dmQ`ov8+uD_?SW!XR4W^RGqZ*1ON8T&}; zXX+VgTd>y6GXB?+%ZmS$G0Yc$7{dTajkcib-@ZP^CB-GTpVF-ne5b~Q^;dfuKW^D+ zx;@j}Ty(fWfNrfMHI~aOWQi3ksZg17Nd#$K?d3}M5$WN*Wx7~SCaA6_a8I8$A8C%g zb1sULeW+gJfOS7 z;(ogveXKxt#qW`_Q$VlNIgdNcX8L zFJzusY>YI`a7B7HTxIw1w$q*dA>>CB(s$QyR8_ zp1R!x6mnjr%reUzShs|Pj9zDcv?_hf?bZ5TSrTb$7cM2xcfQ(bnzcy5r97c<_k6YA z;ixr*n4=tY%54PCq1t00ii~L=yH#;0Bw5hHpX8D#IP0~rmE57 zAtPohTWqP9m>=Z2u{Wh8A*)wIvErs6Nr-GhYUU!?rS?3s=Uguw;y6dyLVeKNxVwJQ zsc~wAF(3$CuQQ=p(h1nMa4xkx$qjPZFA;%fPEJj^$Ho+sYXbebS_F@>6Z9jfmI9yOk|i@ol4NUJ9T4 zD*!j+Ln1r}z!1|mvy;wp`<+3+jvvArjv3K_!adOJRqev?b)dmup%R-_z-^X zSQb)Gu$>DGER5p=JZwo_D0@^F=eoRutKj`}7V&Is5awa67CcP};(sY|(X z1zPa#1nZEQh1dt#VxtkvfY{?!efYr9dK(d@KunIfp;Rf0?RdXx!6?0B11@%_+3y$P zAm7cC%cSpPlu-#$tG@{jFxb(2FuZ5be!$?) z+~`=Q_P}*uzggv;>D)O4bBm{AyeU$(#)Ta0ek|z@V2Ai7A>zk0hEkzcL!U_V)q_zd z%6{?~Jd(|t`-RoBjz&d~c%9j8T(FR7=M~MU5kme`tastk9E1Q%wW@EnSf4S%)N;@% zO)n>LuJ)Vx+OU%Te6gus(kOYH^vFT?YJk?kaxkgU^&aTTUFvu&1Jg=c2C!G2%}>$uG0oL~Lt!z8d+1@6Z*g8mjbopq;Xmojy9q zXH#}k8X{RT24|a++b7vG!ffvKC07d}T4}n@ViyYpivBL1=e0+vf0^#9LKbpA3@{Yn z8P|u;>(uc99MtW%%n;_=RrXff*|lv(F0)*)l&ZJ&O)P5WM`~t%nxVFy43W)j8l_4< z2qi>#@8)%uY|>SuF$xIpVEdSPmheb4OEB;>q1Tuk&=^XEYWqJ<1-}H0bcJ`b;A_v4 zMA4{3P0Li#;>#sWhALB2LVVb$`@)SsrptF%ADvxkp}$~wOmLbl%CMf1~n@H`c|&u1BG=$R>DV*p>^%um6Z_1s+e0r z%$5qNcZ+=<;P+5yXxm<=`(pnEQ_1_D^ZAN{54L7qd_nEeqCOi zrpQts?sA}7gKIFveeS{bW4-1%jrHMMuazi2Kk<-Muvy{PL2P=guZWJ@**-jo zmGz;~@+c$z=w_ZchDps-MuM~4a-UCNYczf^(u*s5CorRhqJBs4Yu~romXVne7#?6l zvF|pHsy+ZZy9Fc!<5hY^Ag)B87WKr@mcAUX;B6Kl7Lv3pf%+Sg({^^cpGfFDI^FCA zKcwBgoeNQKP!tTlNkaeBt#CQG@o!*1{|(IU)FFL!uP=y%`77X#>@Vu+R9L>u#aOPK z`F~CjR0tfQyX9mGx6~a8pyZpaq*_=oIk)Ph>iYBqxiNvc^~7lAZ?D5fQF=XK`b>}! zMHnIY3#c;4v0Bz{mf#bXD@PuyD?3dCMP8}xjSKE9hOcGIOgmn^P%ZHpp^F*Ff};NL zxcHABEm(JLEYjA#p=`;0w;a)8D2f#)wEER%erMfSCeY6hL4yR3epg~ewYPjAVD%){ z55rt_!MSvQm@vkMZhEbs5c0%{a$KL^)W~ZF_%Sm@({CvVKUiw_lxR-4d2(y{xX59v z4_qr>F*vf+LHa$Vy(hymQ(@Rlp&9lZEV3<8&pog`SH*vFa^ilre0k)uPL8ZV0dHCK zNRh-*6(pxgb@=TTg=#@K?+Zrj$FMFtLwYPV~E+|Y=y!ML*kLfD#;o*($ z*r*k|*)h9LP*Q-7P%3mC6}$HGwFRE_d`%U34=px!IFZC^P9Q?FT%<{Di5(|=OMm3* z`0C_3CFpty0|emSb+@c&9bTs-KNepb8kRT79NqY(`ujST+BFeh8guy$#+TSVk8H25 zNPF@zzs_G@s+QUPXQ6!aabSmk*lIYcUA_FL?NAIiq$VzQPEMD_8C8xh;s0s@c8)#x zA+$bV7jodI8}9~Jo{>yo$J5^j7HF<>lo16v21r3NLXSEhAsGz+X3wndcH-W3`lika zpA}gFsYB`%N-!h_!qH;0-npK$9AY@j;IPHGUqac2VyTnI+cT9vv$M1A|DA{CE7QHX z{5D<<<)13I(jA&~tc1EKWb5H*yKj4lj@Y<&w?X)GN7ls%9K%k>*$ z|B=zsfz>eRlMCxpfWfp(r+-rhCzd~$-^z)!oGAV=!^_3ColW{!&@)K#Q?49kF;8^f zaB8iNNLS2hbJIH}QtMCV=5O_WHhq7zYtp}CwKY>0nac0B&!Fx71suZj>y$gm%}ex6 zO4^FAG32Kk7Tj)@r>*z;@0~^r4o6T$*qhOXb?`s0K7E!_drE0=bLQ>tbC59PJhN^2 zu$tHw&ayRA6&-f%_TOyHH*&o^%sD2OF1610b5)s|(+JEx+$?=ylN{fs_)RPSq1xXVX>{kNV|6T2%NrKddNQYi2rU z>X$?oj%mbm9Pa0Me-^$q5%D{O39j}mBKDhM+^)DPTq*%)c&+VY_@r}-xD4trviepU zx#DN%hxn;-*eSjxQw7pbK4{QhWIM#;Eo^9E0US~(pKmA; zfXDO!9%1xrq!_|fHxaC0L>D&IIpN;v+jj&n1@Qh?z+2!%x%T$zs)2n$AM@QB6M}Hx zy^C|3Hp6SxyY8!OO>J^@|GtgO`(7V=6K-Np@6zmMh}&D(`~b6ZfqEbK&nhC@@KTe$ z`Nc(Vm03RA{r%RVZVW6eCce6Z`So}-CMKq7em+}A`*u265|k}^r?WHTS*pnPDMA&M z3TZeYIN*J`WZh+5ae-Q*H1pmw=s7D8U5l_7_RE(NO2d?CWaQxBc#5|Ntl6?jWCwqZ zz-|w%QU){hV1?YP=@0G!t<29q^?Zvu-`U08d`42~m%?Z*#Z;$GG4nYFUY6u>qB95b(J!F!rP(si z9nYWd@56<)>xJDd=}nq3vGH+4q#Y<&2IW!4|NiB+f6F(-!6+ge78e#q>qIWo*M~Sh zsee^0v_Mv`y0wM!`IH$QVr`x5;4uml&rys} zB$)ByOS*7wWGnK8Fo0nqw^zyR`d#olnlDM64TX2%?z|@4jMiAUF=4k|GtD^8ShDVc z(ahNt7#Bv?J`v$wFp<5+KWPw=|i^S4uW=KD$4zbY=Vv%|I%mH?cM1 zKyvg0LJ4j^CU+MTNZdA?RUpsZgQ5>d;}yx&6wD>xreDa;7t|LmE+^Sny!ZF>n?`>D zkel!hix{=oMbK7P#=;P{{UQv)5V3s4W6V?$?g2 zegf$SYc|r~J^JT(LaV(_FzHqb4Ek5DSE4#lPcOd<(#lbgh6+qHBHcoG;=efw9CR$n zgfh(1vz`nyV;X`xpj04_p^^x-P>ffmqgSjLiU*sUi5s3%zgzHj*RSp z>28!z6e6On^}|mp#_pGBAk#`5B0KFJ9KfMJGoszaf==B%UE|-=Q566p2drE20)KlL zEJ|M{aN1ru&lY`PlYeSS>viazGF1FqC$wK6QNv|g>c-NM0VCb10twMVbbq9=72(~= z+Xeo6d?R2l!>ufw-~juBVJ1UJoFbq@_(R7yR`#$UH0)BYfp| zhn7j1`pXe7KH};IVXz_4YlaNDPw!&8A*g0tHoZPr3fej(E@w}KFxaW|-slOCXz#Fl zoyh+aXC#slAyX5sk+DK1_$N{4Kt@{ji=Vl@Zhrd}rkGmi~tZMz2IuYzTt-e z#11W5o1f z=pxV*(mQ7q;lqF35CbN0*zH`}n}gP6T#EhM4ejtYuIWn9xY)7TJR&l(r@I6e6=hg% zqb_!~pT440j?8Uc8y`$MLqog;&i_dX<4*LQf9UZr_#0e8jY?IpfAf#SJYVBopYzHN z=e6Wnk?6^VIui^p5%bg?Q81XR=nkedB*xA;nkmyY+_Hyz)RZSG-oE8IV6~|m4^SKO zV$LqfZ1B_;2FyIzJBYt?8P5LyL<|EPRx>~9n?YP7=j7>w)fd&1UOz+?So13l#7OaA zi-%gA7VCa^2LAfe#03A5`#^XVTXrI;lL~_dv#%bi+jCNU1DR_&U(Jx=UsGXetx^e! z6MgGTCh-t6MsVYF1|hIYr(xe9uaO!hd;H_muep3Yq^g2U7xkG3h5Lt1s%D@VeE_lMQ)#}cF@7W))CB*|lZ7duGEJHT*w)YY6Z zhacZOA$hb+0Jv@fe%<>_b(c?k0`Qobkgl^v{LV4?wpj3Mr(Fi3s`1%FE}g%Z7%gLF z6%pEKJM~|&d$MyT277)GGkTkwK_~a?G4b4PZ(#wUf8}QZMLnbIuFU*%H|+M1@A7F% zYOmPM-YsP@upNN4j%*7`)n3ZtuBm3djHM;wbqHI`?Ak-vGcD<4@}Llo(VMg`R-yFV zE~$8*ZZDKM*W3$(!CKM;O-h_s@&1I8QrgttKU2vOopK`nY=4UKT7WUc45wW#Q`BmM z-TlcI(gs3_%*I!^j0#a|Lsa%OM<@a!fkTTqt2x@)_r^*lEb~?QVxQ*4*sdsXQ+CH> z_ksuGKZ}E}TulsJoo~wW?Qcn4)$PfiSk6fw^E`FQG>CPZg+xTf3H!G5EP=8dKr~ZN zadG`yCuxnZK=(Y0%as zv4lh?3Lr#FY&Joqv)j^E&gY28=d_zDov;fruj>=GL*w50Wv~b+5wP*<>o^5L$3cS&p%V-x% zk*{e=N0CmsV$geY`I2K@g~V!>XUUb^_sHM^VKY%>^g-0OD;PO9RNvs`_*fm(`L&;N z+Lqd(>yZtaAf-Zy6FBEpX3`KZYHBvUH8yEn`+ffFaw_QT>_hHIp=|uYiy@}YK8DSB z0WS{E=v=#->G+_ty5G=NCj~_X__&&QkZguq4QMSjB^Q9pWT=fqo6i0`NH9V2TqFQr z{~)5sY=7E-8)5hQYK`{9dIDINY=*OYWp#Bt>#gNZkC}^$>;AjjH}O22)hFi!yY3I; zuDhcwls^cSCA<`gKHBvNz*}`V^S4meDw2=HbH5rnlNNdLMg1wsz1If{!daDS@Q6&d zfVU`=-BH=KUMMCqNj-kOH_Kv@QH-(jO81bBgEVes;aOR%Hlsm7+(JXjvPe^OIyu38 zyO>DpTm;fAG_dIg*zc_Mq2M#mCh-@a!1#Aq9FbWspD?IYSwFB9(}vzcMUM&lZ<5}u zP@W2d_QNQCApz`0H)^Wl&+dIK`|Qw@L~Fe}F9ggpU!#iMfcKfu;9_Qxnh~&L3d+{C zzg6)8(RuQ4JC64!yKm^Z{cyyv?2qJVYRX?}_!N4C5YnnIUaUGg9?nZw%B%VZ1}`Jv zO+`56A*K|NRVH)L>P`-fxRYtJS> zi64*c@T1YT9_WPCZvwt@DS9liIP7}dvOKX51LC}bJH=Iy?(|KpLnZZ@zrR0sji%rl zUL(jc+4G{K5ES%IqBvHJ?V4GiDyXhnSX`Z5eL$iGI3(mr`3EmhUGiHauUMAE4i7dF zU5?j>+L!%tp19n?y8jH1NVtKq$bR8U4IE&t@A@6+JmSG5I|USF7m2wi zCZ=a^g~UgVUG06qBRBRw-Ek0)QkH9nPz z5~IJ^Uk+UtWOi!{&3}r%nyzQCcJAb+ZFTrK8G3+Bz9=PQ4UQ0VYx|>8AeZMDg7-P6 z0p{ch5ogR`%G1>C^bxj25Zv8QdJRF6TBL@|ogl5j)cyYAGP)YLSBVfRGOVAQ@ljBNEk2q7{y52giNxb~ALlzn12i0<8(#23AVlP+gZK`$ z0<{B5k9lW&cbD7w-r>o+>#IK8;brj{Jg{(0mrn7`$V=3`|6#L;=qFn@oQ#FmJ*vfR z$jk;CIzJ32i?Gj}u;c_(FZk>{Pwf5Csy~Rq7;Y)!^CFz6OCR1s-A0{ocO$!43Mz_) zu(IocbML&yO+t}K$WLX6_P}-~4%Gav@Zd@>>kZb89U4`oATw>j- zkPMpF)wfC^b+n~cjSX-q-Zk0yWt)QUH&?aER2bW(L+eH(s!F--Hzf@j+gFY) z=YEGc6*YXzRv`g!%4h*L=du&c^?0)Gn-RuJSllyN0DDZwe&U#wvm4*=`CAc=0r@uK zSWWGlf&{rRJ|d#$&qM=|j1Z@{{HX=fc<7S^d6!%V<-;9Zs%bx5IJ^NkUmJQ* zqI+V)HiBPNt_y-1ra-oR`-vs^f$ScRV3MVXN4zTN`(Wo-lKd-wd0Dc1P z{dAj{sx@_PKjK8yui3M;JVgWeYyjveW7EkG$99eW?goO!0gs4??b%LG&0dBQLNb_s zqMfihZIA{f-r#>|NN$2!0d(BC8kzfKmcCx&x?Z76(Kh=WKQE^hq5Yv&_ zjqxA)5M01~Aksfl(N^NbgL#hJwT;DJsLPK_-!(le@ z2rxJxIdg04i4d{(Ojqsv`(QL5Lr8Vc6Huu%I`D%stcR(Rz18_@^>Ki^RLaqwg4J8q zx3czsu(fk~lv*5WlU12+uoJ|X2ttTB@Dl(HJlQ#y$#UGcZF9>go==sw#l|kJjN|?y zZm%g^f)C&dLem%P5wSAJRCjpVr)7S-Y(PiB8_NJWg8&D2yIcvfe(|N{y9*4K1XnFg zN!i}ox=&zX-QF7#*1sJ!MRdPWzv{d@^}EZ`NnBJ^2VfoJ>&ckJ^$tQ9h$7ZNCbVs{v7E=*B_0kJ=TC4#lK;{wT&48W;s@HHl2ViM{ z5v^C>zED+ER4cPw;0W~wpyrbl{`}5@M2=m2sD;ex;)baH`#+AZ$0RjuL?dSVF=;(h`Ge`v z5ipB{4aZk;B$Q+k6FEyJL|^3YWDhWfU7=JtyT;-&M!2&5dtrpNOPUogz3S)Fy&eMn z#J=8{Z?;3XWI8wAwYQV@N_0E#A{!O*HCa;%uYU~O^!CM%>f~)p5&s(6V?k01p=0!7 zdDSL64X*$dNMtZ)>IW`=N=ffsIY5^Nsfk zE{d6%*@NAw*6`wI2d=Bx=DpRG)m0|y_BUH*0I7ATygn=*pZBQY#&!Jy*z*7K)nX9{ z3b3#lxHUeuspx}28ji?5em zU0C<=&5nh*YrUgPQqkVs6|fi-VA~i}gF)_X_CUZ*ka{_xV8uCA z-J4eq8V%nC{k{jgn9ZG#Q@z&~n2pcFYa{{lp$2zdxKOp1y9?eu$-VDJ5iL;gR+j7Mwmz-(y@K+cCztZ$zlWROD_!Uwn>07+>3)A&HJYfhO?Nh zAR(vUbOO7@tM1i;vP`OM8IDz9ic2JPK}oYR1IPDfF+mJvkXLmSOn^#;4BH2f+5Yd6 zzaQhdJp3bjvkE^USB00wd2;br<@lK`azZv>LgLlYaCzI+QTl0@N!T;wHC}jtvU980 zLObh3j^`s7Pt|DHZo?Vmlhz%isxSduwR{fkXwerJq$YC#`$DK=q%nz!fojcj?|&2L za@rYw$XPV0ml6}hIf}fKEVxr=i*cz3>q2SDb0Avqseoma3ipr2 z=5$K3)WuLPrh3%M;Gc{(+QX3Mu`rWO9Z)L{gsRo{l zP;UrIBqk)psd%Moj2s_w8{jH3g%KRH>vB<1+LwTX)CI`MesN?^d&$mvUoCLtj~rBbN5QtUU4s&jn4qo8DX#HU-0d=QeIwxf#y z6&NAww`~J#W~1z2$w-P7mb-2D8U>IAP2bQ?{W$#N;19jCCAH+RlakyUKApK zKei5#k#;znX=DZo?V2HuMHAcuBQV@>H0lF))=Q_YOu#-MD;JOipJ|o5QlVB83>XyR z?C0kn4mWS#P(m_kKexf~!(mcLfRgvuQh7=$&{K!xbgVy+tXBdX`Z?KiNq)3IwddsH zW)E%WJwW-kPcCCX%hS;<$WA4}t1m~^x3-3oc3r$eU_Xxxk_db*2s-Fv#{A3|Fp8X4yASm)ySZCP8j;#r15&F0vf~^TV z{1(yn{TushMiCKVk4Mt%t0KZkFE;feKE_z1)~dJgHu`{0Z&xF&c!wOU^{jW{8Je=u z=P_-teLb&}nE>v}R+{zExMe>1h;Swv{gYHY@~#xrR6}($je)DpOwxdNO^G`sBSXQn z7ZOI`jr1(XZSM9{o!aLqm8J#Ik*T$0mV(ZJt8SWFb!%J>xPY=ujP3Za175Un-ZtZn%f@lin2gV;;Pc_Ux!t2q;dn|;7g^-K7DwC}?RGzP0xgW3K zNx3y#MHa!qT?08#P2_12r93p-)rLVFBLbpPZJh>t)_rF({V;-Fa>j+HoGin$HxBz6 zV?Q5$W?{ks4}q}6-`G5{`i)`kf{kz2PNiL&I~(*#pZ+x?gM#vyQbPEh(wp{J^@33+pJSVG3nHkZK(5p_9oUuyUKE_Ff zgH+0v_EOtkyO_S)Z@m^Oa%D2FgEhv|P7FT5D^w!k9d%BNa+L)tmre4Z?*7Bq@ReG> zj$WiqJ4fbyOK%Rk?^8#JbzYyM0*)#KkLx5`)L$GwIwcupnC<#H^G+#O{o9X^>I3tG zD}ZHctJT~IUlPNFgA)gV(MS*M5TAW25O2mC$vJg-syQ>WF^*f)CHLIWcyqaaNd~vwdoY?OY@I#5;_KJJ9~BqY5=$CV zSM1TdBSlM_(~p9N)Xe`06Sr@d8pgUaVM<$jAMgmh-8~*p6dDqu%-|-3$ z!Eo&GvxiDQ-)EX5X&`q{9!sY-dBw~Eu{~l2IZ=ttn6mdbC+rd%Chb{mrQ8KyuV7a2 zCH9W=DGnY~4pbQaKK8O-%lzdRcMfn<(pQ(mSJ>ei$jYK=slE`IkR@1>z2KBlhlcfW z;^<#~F~l2OoEP0qb-s}2cHYGvV*l}e{vOvs07%l0>vk__73-gaUhmn$IVHFZ^5}t- zt5@Co>J+;f0ZjFd~O7m_VosveS>i3WS0K;;In=s%8As}Dtj#s4Sc3XLnEkm+A>~8?$fqJu1%gR@p>Vev^rtFs6shZelHAVNI z%+F0%6+sP+StFqoY!a|Ee~vV~cY&`NMj|F(Ibw^@h|uWczpqHcXv#DdX>D&yTl?e9 z8~t23|6>?>nOSN=IYj|S4@JDn3V(c6XcePmt7d>)L1)JGwel_M9AvZE`7DJk;ksh0 z8?o>$K5lw51b6)Fjorb-MMc9w`p9uvFm+<^O0z`PBpZrNl>Wm??EoMBg#~^1S+Kz= zDZ|X|@=~;UY|VrDf~>NcuH~Ew&E(|Xo?%Ny*dG}I&_=$ROdJ**&!iJS2$HBr&anfV2bMtata8i^|QX0ALcD|iZTV@upjUw#eeB9%hl&^9;ng~R;pk?sZ^Tt0KkD9G z2@XnO#b9Cb#%W}TEQ!~i>bfEp5d&nzry_PznLqGQDn340%BnObldfA$XQE?r!!-sk zMsX_N$itD`f^JiSrvzReKe9YRAtA=|SCW;D-@Z4Y$i%^&omfKMj8ekscvsTdhC_m` z&Vzq_VkF6~4EFWs&C$+=gshJ9qeqvDg)wYA0o*p$57d;*+4K;^n_@U8gKQ5-&P*7~fhv%49 zvggj1dxpIDw5-?@1f>sVDD1Z(U%0*jAr79X)a2<*S#vggZm%OXn>m~ab)0jZ`g!sM zrP1s6%yeITcCy9w26aRIRmBjKGU0I>xrAq!VHH2dp0Pqc3U6!jd-z_adFFv!l{!f+ z=!g1A@MfJi;1V;WY!wCNW_cN3yK^b@+Vcjh0rJV3v%G)I;#e&5iauHPQ6Ww%dNIz7 z@_H^`9O5h7KRhJVbMa&c%l)jb^~0{4NkD7g2|dalr$W)l;Z?jn^a4>TC3|xs8=To1 zt)4#W!rsXg0kh~wy=tIU{*sg*hA33XbkNCEsCTv~Jqd9pj-NgoNFv>GyWJK;zqENN z>ujTPNrW=bUw(#LoS}Npn#x0B zCtpd@8DFY3T_dp!HB+(*zT3FA0t}?Jvl;E;l$c0LMOLLNTES`erTB9uHqgh;V|a8XC-alU4}>EkB}Q*N7`_ln z06A`*P(fuQJ|t!KX-F;q4G&QkvoV1WXcUd0VfegX9aXl{t*I(dyZ41SV?ga!Y*!|o zS($pPzHL3&Hsz?$$U82$&^g>-;%B0QBq9#V*zTeHfBbw&oD{tG;5gfSJ2B<}ISvUC L8R5eBy59c}JXG!b literal 0 HcmV?d00001 From b0d1c433cdde475e2d463b0b84b94f56a8de891e Mon Sep 17 00:00:00 2001 From: Rey Lejano Date: Thu, 28 Apr 2022 08:24:50 -0700 Subject: [PATCH 051/119] follow-up PR to PR 33280 --- .../_posts/2022-04-29-kubernetes-1.23-release-interview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md index cc8cad881b..c2532638d0 100644 --- a/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md +++ b/content/en/blog/_posts/2022-04-29-kubernetes-1.23-release-interview.md @@ -34,7 +34,7 @@ REY LEJANO: I think it was mostly from family, since I do have a family history **CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?** -REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://en.wikipedia.org/wiki/ITIL). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually. +REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://www.axelos.com/certifications/itil-service-management/what-is-itil). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually. **CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.** @@ -158,7 +158,7 @@ REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23. -One of the other highlights of 1.23 is [pod security admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) going to Beta. I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running. +One of the other highlights of 1.23 is [pod security admission going to Beta](/blog/2021/12/09/pod-security-admission-beta/). I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running. There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed. From c88343d6b5a9ac5c826a085c247aa1e6c91c6e24 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Wed, 27 Apr 2022 06:10:35 +0800 Subject: [PATCH 052/119] [zh]Update content/zh/examples/application/mysql/mysql-statefulset.yaml [zh]Update content/zh/examples/application/mysql/mysql-statefulset.yaml Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng Update content/zh/examples/application/mysql/mysql-statefulset.yaml Co-authored-by: Qiming Teng --- .../application/mysql/mysql-statefulset.yaml | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/content/zh/examples/application/mysql/mysql-statefulset.yaml b/content/zh/examples/application/mysql/mysql-statefulset.yaml index b69af02c59..22aa944748 100644 --- a/content/zh/examples/application/mysql/mysql-statefulset.yaml +++ b/content/zh/examples/application/mysql/mysql-statefulset.yaml @@ -21,17 +21,18 @@ spec: - "-c" - | set -ex - # Generate mysql server-id from pod ordinal index. + # 基于 Pod 序号生成 MySQL 服务器的 ID。 [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} echo [mysqld] > /mnt/conf.d/server-id.cnf - # Add an offset to avoid reserved server-id=0 value. + # 添加偏移量以避免使用 server-id=0 这一保留值。 echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf # Copy appropriate conf.d files from config-map to emptyDir. + # 将合适的 conf.d 文件从 config-map 复制到 emptyDir。 if [[ $ordinal -eq 0 ]]; then - cp /mnt/config-map/master.cnf /mnt/conf.d/ + cp /mnt/config-map/primary.cnf /mnt/conf.d/ else - cp /mnt/config-map/slave.cnf /mnt/conf.d/ + cp /mnt/config-map/replica.cnf /mnt/conf.d/ fi volumeMounts: - name: conf @@ -45,15 +46,15 @@ spec: - "-c" - | set -ex - # Skip the clone if data already exists. + # 如果已有数据，则跳过克隆。 [[ -d /var/lib/mysql/mysql ]] && exit 0 - # Skip the clone on master (ordinal index 0). + # 跳过主实例（序号索引 0）的克隆。 [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} [[ $ordinal -eq 0 ]] && exit 0 - # Clone data from previous peer. + # 从原来的对等节点克隆数据。 ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql - # Prepare the backup. + # 准备备份。 xtrabackup --prepare --target-dir=/var/lib/mysql volumeMounts: - name: data @@ -88,7 +89,7 @@ spec: timeoutSeconds: 5 readinessProbe: exec: - # Check we can execute queries over TCP (skip-networking is off). + # 检查我们是否可以通过 TCP 执行查询（skip-networking 是关闭的）。 command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"] initialDelaySeconds: 5 periodSeconds: 2 @@ -105,22 +106,22 @@ spec: set -ex cd /var/lib/mysql - # Determine binlog position of cloned data, if any. + # 确定克隆数据的 binlog 位置（如果有的话）。 if [[ -f xtrabackup_slave_info && "x$( change_master_to.sql.in - # Ignore xtrabackup_binlog_info in this case (it's useless). + # 在这里要忽略 xtrabackup_binlog_info （它是没用的）。 rm -f xtrabackup_slave_info xtrabackup_binlog_info elif [[ -f xtrabackup_binlog_info ]]; then - # We're cloning directly from master. Parse binlog position. + # 我们直接从主实例进行克隆。解析 binlog 位置。 [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1 rm -f xtrabackup_binlog_info xtrabackup_slave_info echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\ MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in fi - # Check if we need to complete a clone by starting replication. + # 检查我们是否需要通过启动复制来完成克隆。 if [[ -f change_master_to.sql.in ]]; then echo "Waiting for mysqld to be ready (accepting connections)" until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done @@ -133,11 +134,11 @@ spec: MASTER_PASSWORD='', \ MASTER_CONNECT_RETRY=10; \ START SLAVE;" || exit 1 - # In case of container restart, attempt this at-most-once. + # 如果容器重新启动，最多尝试一次。 mv change_master_to.sql.in change_master_to.sql.orig fi - # Start a server to send backups when requested by peers. + # 当对等点请求时，启动服务器发送备份。 exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \ "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root" volumeMounts: From 1f9f0a29d0dd0f30b076e264b5b8baa7b180425b Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Wed, 27 Apr 2022 20:11:20 +0800 Subject: [PATCH 053/119] [zh]Update content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md Update content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md Co-authored-by: Qiming Teng Update content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md Co-authored-by: Qiming Teng Update content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md Co-authored-by: Qiming Teng --- .../custom-resource-definition-versioning.md | 28 +++++++++++++++---- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md b/content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md index 603274f5c2..38af138e61 100644 --- a/content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md +++ b/content/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning.md @@ -1384,8 +1384,8 @@ procedure. *选项 1：* 使用存储版本迁移程序（Storage Version Migrator） @@ -1409,13 +1409,29 @@ The following is an example procedure to upgrade from `v1beta1` to `v1`. 2. Write an upgrade procedure to list all existing objects and write them with the same content. This forces the backend to write objects in the current storage version, which is `v1`. -3. Update the CustomResourceDefinition `Status` by removing `v1beta1` from - `storedVersions` field. +3. Remove `v1beta1` from the CustomResourceDefinition `status.storedVersions` field. --> 1. 在 CustomResourceDefinition 文件中将 `v1` 设置为存储版本，并使用 kubectl 应用它。 `storedVersions`现在是`v1beta1, v1`。 2. 编写升级过程以列出所有现有对象并使用相同内容将其写回存储。 这会强制后端使用当前存储版本（即 `v1`）写入对象。 -3. 通过从 `storedVersions` 字段中删除 `v1beta1` 来更新 CustomResourceDefinition - 的`Status`。 +3. 从 CustomResourceDefinition 的 `status.storedVersions` 字段中删除 `v1beta1`。 + +{{< note >}} +`kubectl` 工具目前不能用于编辑或修补 CRD 上的 `status` 子资源：请参阅 +[kubectl Subresource Support KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cli/2590-kubectl-subresource) +了解更多细节。 + +从 CLI 给 `status` 子资源打补丁的更简单的方法是使用 `curl` 工具直接与 API 服务器交互，示例: +```bash +kubectl proxy & +curl --header "Content-Type: application/json-patch+json" \ + --request PATCH http://localhost:8001/apis/apiextensions.k8s.io/v1/customresourcedefinitions//status \ + --data '[{"op": "replace", "path": "/status/storedVersions", "value":["v1"]}]' +``` +{{< /note >}} \ No newline at end of file From d9a0e4123b2ead90e3aca65caa937d83acda155e Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Thu, 28 Apr 2022 01:45:04 +0800 Subject: [PATCH 054/119] [zh]Update content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md Update content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md Co-authored-by: Qiming Teng Update content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md Co-authored-by: Qiming Teng --- .../managing-secret-using-kubectl.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md b/content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md index 8e2dd7d202..86d0c9266b 100644 --- a/content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md +++ b/content/zh/docs/tasks/configmap-secret/managing-secret-using-kubectl.md @@ -184,6 +184,10 @@ Now you can decode the `password` data: 现在你可以解码 `password` 的数据： ```shell +# 这是一个用于文档说明的示例。 +# 如果你这样做，数据 'MWYyZDFlMmU2N2Rm' 可以存储在你的 shell 历史中。 +# 可以进入你电脑的人可以找到那个记住的命令并可以在你不知情的情况下 base-64 解码这个 Secret。 +# 通常最好将这些步骤结合起来，如页面后面所示。 echo 'MWYyZDFlMmU2N2Rm' | base64 --decode ``` @@ -194,6 +198,21 @@ echo 'MWYyZDFlMmU2N2Rm' | base64 --decode 1f2d1e2e67df ``` + +为了避免在 shell 历史记录中存储 Secret 的编码值，可以执行如下命令: + +```shell +kubectl get secret db-user-pass -o jsonpath='{.data.password}' | base64 --decode +``` + + +输出应与上述类似。 + ## 清理 {#clean-up} From 2d733bbb03511c07b7fc3d146b9d3b37254d761f Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Thu, 28 Apr 2022 01:55:08 +0800 Subject: [PATCH 055/119] [zh]Update content/zh/blog/_posts/2018-10-01-health-checking-grpc.md Update content/zh/blog/_posts/2018-10-01-health-checking-grpc.md Co-authored-by: Hao Yuan Update content/zh/blog/_posts/2018-10-01-health-checking-grpc.md Co-authored-by: Hao Yuan Update content/zh/blog/_posts/2018-10-01-health-checking-grpc.md Co-authored-by: Hao Yuan --- .../zh/blog/_posts/2018-10-01-health-checking-grpc.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/content/zh/blog/_posts/2018-10-01-health-checking-grpc.md b/content/zh/blog/_posts/2018-10-01-health-checking-grpc.md index f80633cf8a..98cf437a2b 100644 --- a/content/zh/blog/_posts/2018-10-01-health-checking-grpc.md +++ b/content/zh/blog/_posts/2018-10-01-health-checking-grpc.md @@ -16,6 +16,15 @@ date: 2018-10-01 ---> **作者**： [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google) + +**更新（2021 年 12 月）：** “Kubernetes 从 v1.23 开始具有内置 gRPC 健康探测。 +了解更多信息，请参阅[配置存活探针、就绪探针和启动探针](/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)。 +本文最初是为有关实现相同任务的外部工具所写。” + -* `type`：`io1`，`gp2`，`gp3`，`sc1`，`st1`。详细信息参见 - [AWS 文档](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)。默认值：`gp3`。 +* `type`：`io1`，`gp2`，`sc1`，`st1`。详细信息参见 + [AWS 文档](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)。默认值：`gp2`。 * `zone`(弃用)：AWS 区域。如果没有指定 `zone` 和 `zones`， 通常卷会在 Kubernetes 集群节点所在的活动区域中轮询调度分配。 `zone` 和 `zones` 参数不能同时使用。 From a571fef7c8061ae3fda39650a0a28295a42a2c15 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Fri, 29 Apr 2022 01:32:16 +0800 Subject: [PATCH 057/119] [zh]Sync content/zh/docs/tasks/configure-pod-container/security-context.md --- .../zh/docs/tasks/configure-pod-container/security-context.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/tasks/configure-pod-container/security-context.md b/content/zh/docs/tasks/configure-pod-container/security-context.md index 1cb617ae85..29e5a3afb3 100644 --- a/content/zh/docs/tasks/configure-pod-container/security-context.md +++ b/content/zh/docs/tasks/configure-pod-container/security-context.md @@ -631,7 +631,7 @@ in the `securityContext` section of your Pod or Container manifest. The [SeccompProfile](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#seccompprofile-v1-core) object consisting of `type` and `localhostProfile`. Valid options for `type` include `RuntimeDefault`, `Unconfined`, and -`Localhost`. `localhostProfile` must only be set set if `type: Localhost`. It +`Localhost`. `localhostProfile` must only be set if `type: Localhost`. It indicates the path of the pre-configured profile on the node, relative to the kubelet's configured Seccomp profile location (configured with the `-root-dir` flag). From 2e92ea8627a6086de622ee1aff522bad6567c62b Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Fri, 29 Apr 2022 01:47:29 +0800 Subject: [PATCH 058/119] [zh]Sync content/zh/case-studies/squarespace --- .../zh/case-studies/squarespace/squarespace_featured_logo.svg | 1 + 1 file changed, 1 insertion(+) create mode 100644 content/zh/case-studies/squarespace/squarespace_featured_logo.svg diff --git a/content/zh/case-studies/squarespace/squarespace_featured_logo.svg b/content/zh/case-studies/squarespace/squarespace_featured_logo.svg new file mode 100644 index 0000000000..a69d7ea5c8 --- /dev/null +++ b/content/zh/case-studies/squarespace/squarespace_featured_logo.svg @@ -0,0 +1 @@ +kubernetes.io-logos2 \ No newline at end of file From 1be866980854102455fc5d5fd6415807a03a13d3 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Fri, 29 Apr 2022 01:24:16 +0800 Subject: [PATCH 059/119] [zh]Update content/zh/docs/reference/labels-annotations-taints/audit-annotations.md [zh]Update content/zh/docs/reference/labels-annotations-taints/audit-annotations.md Update content/zh/docs/reference/labels-annotations-taints/audit-annotations.md Co-authored-by: Qiming Teng Update content/zh/docs/reference/labels-annotations-taints/audit-annotations.md Co-authored-by: Qiming Teng Update content/zh/docs/reference/labels-annotations-taints/audit-annotations.md Co-authored-by: Qiming Teng --- .../audit-annotations.md | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/reference/labels-annotations-taints/audit-annotations.md b/content/zh/docs/reference/labels-annotations-taints/audit-annotations.md index 2ab584e4ed..a90160888b 100644 --- a/content/zh/docs/reference/labels-annotations-taints/audit-annotations.md +++ b/content/zh/docs/reference/labels-annotations-taints/audit-annotations.md @@ -88,7 +88,7 @@ that was transgressed as well as the specific policies on the fields that were violated from the PodSecurity enforcement. See [Pod Security Standards](/docs/concepts/security/pod-security-standards/) -for more information +for more information. --> ## pod-security.kubernetes.io/audit-violations {#pod-security-kubernetes-io-audit-violations} @@ -99,4 +99,38 @@ PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container 注解值给出审计策略违规的详细说明，它包含所违反的 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)级别以及 PodSecurity 执行中违反的特定策略及对应字段。 -有关详细信息，请参阅 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)。 \ No newline at end of file +有关详细信息，请参阅 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)。 + + +## authorization.k8s.io/decision {#authorization-k8s-io-decision} + +例子：`authorization.k8s.io/decision: "forbid"` + +此注解在 Kubernetes 审计日志中表示请求是否获得授权。 + +有关详细信息，请参阅[审计](/zh/docs/tasks/debug-application-cluster/audit/)。 + + +## authorization.k8s.io/reason {#authorization-k8s-io-reason} + +例子：`authorization.k8s.io/reason: "Human-readable reason for the decision"` + +此注解给出了 Kubernetes 审计日志中 [decision](#authorization-k8s-io-decision) 的原因。 + +有关详细信息，请参阅[审计](/zh/docs/tasks/debug-application-cluster/audit/)。 \ No newline at end of file From 75a812ca5c04e7277ee9d506df3f44b73c43ee62 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Wed, 27 Apr 2022 23:10:54 +0800 Subject: [PATCH 060/119] [zh]Update content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md [zh]Sync x [zh]Update content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md --- .../windows/user-guide-windows-containers.md | 29 ++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md b/content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md index 88c23e548f..f13822c0bf 100644 --- a/content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md +++ b/content/zh/docs/setup/production-environment/windows/user-guide-windows-containers.md @@ -43,7 +43,7 @@ Windows 应用程序构成了许多组织中运行的服务和应用程序的很 control plane and a [worker node running Windows Server](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) * It is important to note that creating and deploying services and workloads on Kubernetes behaves in much the same way for Linux and Windows containers. -[Kubectl commands](/docs/reference/kubectl/overview/) to interface with the cluster are identical. +[Kubectl commands](/docs/reference/kubectl/) to interface with the cluster are identical. The example in the section below is provided to jumpstart your experience with Windows containers. --> ## 在你开始之前 @@ -278,7 +278,34 @@ with one of its main goals being that this approach should not break compatibili 目前，用户需要将 Linux 和 Windows 工作负载运行在各自特定的操作系统的节点上， 因而需要结合使用污点和节点选择算符。 这可能仅给 Windows 用户造成不便。 推荐的方法概述如下，其主要目标之一是该方法不应破坏与现有 Linux 工作负载的兼容性。 + + {{< note >}} +如果 `IdentifyPodOS` [特性门控](/zh/docs/reference/command-line-tools-reference/feature-gates/)是启用的， +你可以（并且应该）为 Pod 设置 `.spec.os.name` 以表明该 Pod +中的容器所针对的操作系统。 对于运行 Linux 容器的 Pod，设置 +`.spec.os.name` 为 `linux`。 对于运行 Windows 容器的 Pod，设置 `.spec.os.name` +为 `Windows`。 + +在将 Pod 分配给节点时，调度程序不使用 `.spec.os.name` 的值。你应该使用正常的 Kubernetes +机制[将 Pod 分配给节点](/zh/docs/concepts/scheduling-eviction/assign-pod-node/)， +确保集群的控制平面将 Pod 放置到适合运行的操作系统。 +对 Windows Pod 的调度没有影响，因此仍然需要污点、容忍度以及节点选择器， +以确保 Windows Pod 调度至合适的 Windows 节点。 + {{< /note >}} From e832559fa0e4452c1340c2cc194c914a4084d0ed Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Fri, 29 Apr 2022 12:11:43 +0800 Subject: [PATCH 061/119] [en] Update docs/doc-contributor-tools/linkchecker Signed-off-by: xin.li --- content/en/docs/doc-contributor-tools/linkchecker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/doc-contributor-tools/linkchecker/README.md b/content/en/docs/doc-contributor-tools/linkchecker/README.md index a575c4d1fe..6d4b714655 100644 --- a/content/en/docs/doc-contributor-tools/linkchecker/README.md +++ b/content/en/docs/doc-contributor-tools/linkchecker/README.md @@ -46,7 +46,7 @@ To run the link checker: 2. Run the following command: ``` - make docker-internal-linkcheck + make container-internal-linkcheck ``` ## Understanding the output From 9d98815cc55db2f761153c96a85ccd1b82294358 Mon Sep 17 00:00:00 2001 From: Kazuyuki Sakemi Date: Fri, 29 Apr 2022 15:10:44 +0900 Subject: [PATCH 062/119] current link is pointing 404 page. replacing relative link from "/docs/tasks/debug-application-cluster/debug-service/" to "/docs/tasks/debug/debug-application/debug-service/". --- .../docs/tasks/administer-cluster/dns-debugging-resolution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md index e9dd545955..55030539cb 100644 --- a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md +++ b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md @@ -176,7 +176,7 @@ The service name is `kube-dns` for both CoreDNS and kube-dns deployments. If you have created the Service or in the case it should be created by default but it does not appear, see -[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) for +[debugging Services](/docs/tasks/debug/debug-application/debug-service/) for more information. ### Are DNS endpoints exposed? @@ -193,7 +193,7 @@ kube-dns 10.180.3.17:53,10.180.3.17:53 1h ``` If you do not see the endpoints, see the endpoints section in the -[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) documentation. +[debugging Services](/docs/tasks/debug/debug-application/debug-service/) documentation. For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns) From 3ae803b70772054c8ba3662ec170d28a457667c0 Mon Sep 17 00:00:00 2001 From: Ed Date: Fri, 29 Apr 2022 11:53:13 +0200 Subject: [PATCH 063/119] Added link to Version Skew Policy in Upgrading Kubeadm Cluster (#33293) * Added link to Version Skew Policy in Upgrading Kubeadm Cluster * Update content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md Co-authored-by: Lubomir I. Ivanov Co-authored-by: Lubomir I. Ivanov --- .../en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 36e5a1a00c..8040e1185f 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -11,7 +11,7 @@ weight: 20 This page explains how to upgrade a Kubernetes cluster created with kubeadm from version {{< skew currentVersionAddMinor -1 >}}.x to version {{< skew currentVersion >}}.x, and from version {{< skew currentVersion >}}.x to {{< skew currentVersion >}}.y (where `y > x`). Skipping MINOR versions -when upgrading is unsupported. +when upgrading is unsupported. For more details, please visit [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/). To see information about upgrading clusters created using older versions of kubeadm, please refer to following pages instead: From 13c03b0d58798590e3ed0ba4c4e7a43ebfa6f4ee Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Fri, 29 Apr 2022 17:58:37 +0800 Subject: [PATCH 064/119] [zh] Update administer-cluster/kubeadm/kubeadm-upgrade.md Signed-off-by: xin.li --- .../docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index fdad20dfc4..9704cf3760 100644 --- a/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -18,12 +18,12 @@ min-kubernetes-server-version: 1.18 This page explains how to upgrade a Kubernetes cluster created with kubeadm from version {{< skew currentVersionAddMinor -1 >}}.x to version {{< skew currentVersion >}}.x, and from version {{< skew currentVersion >}}.x to {{< skew currentVersion >}}.y (where `y > x`). Skipping MINOR versions -when upgrading is unsupported. +when upgrading is unsupported. For more details, please visit [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/). --> 本页介绍如何将 `kubeadm` 创建的 Kubernetes 集群从 {{< skew currentVersionAddMinor -1 >}}.x 版本 升级到 {{< skew currentVersion >}}.x 版本以及从 {{< skew currentVersion >}}.x 升级到 {{< skew currentVersion >}}.y（其中 `y > x`）。略过次版本号的升级是 -不被支持的。 +不被支持的。更多详情请访问[版本倾斜政策](https://kubernetes.io/releases/version-skew-policy/)。 + +Starting with version 1.24, dockershim has been removed from Kubernetes. For more information, see [Dockershim FAQ](/dockershim). From c68c4e95187bdb91675335461ca26faae0fc35f7 Mon Sep 17 00:00:00 2001 From: Tej Singh Rana <58101587+Tej-Singh-Rana@users.noreply.github.com> Date: Fri, 29 Apr 2022 19:17:14 +0530 Subject: [PATCH 080/119] Added back the bracket --- .../index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md index dff7428219..25161b8c96 100644 --- a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md +++ b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md @@ -45,7 +45,7 @@ Linux containers (and underlying mechanisms such as kernel namespaces) work. You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article [What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/). (As you read that, bear in mind that Linux kernel namespaces are a different thing from -[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/). +[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/)). ## Skip the talk, what do I need to use this new approach? From 99c5658119b23eaa9c8d5875e836718149895c3c Mon Sep 17 00:00:00 2001 From: thomasd-ign <103668755+thomasd-ign@users.noreply.github.com> Date: Fri, 29 Apr 2022 17:11:14 +0200 Subject: [PATCH 081/119] Update content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md Co-authored-by: Rey Lejano --- .../index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md index 590aa0597d..25161b8c96 100644 --- a/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md +++ b/content/en/blog/_posts/2022-04-28-Increasing-the-security-bar-in-Ingress-NGINX/index.md @@ -45,7 +45,7 @@ Linux containers (and underlying mechanisms such as kernel namespaces) work. You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article [What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/). (As you read that, bear in mind that Linux kernel namespaces are a different thing from -[Kubernetes namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)). +[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/)). ## Skip the talk, what do I need to use this new approach? From e29fa4ea2e98c5733bab18bcfe0c04358e5d4705 Mon Sep 17 00:00:00 2001 From: huangminjie Date: Fri, 29 Apr 2022 23:12:34 +0800 Subject: [PATCH 082/119] update kubeadm_init_phase_certs_etcd-server.md Signed-off-by: huangminjie --- .../kubeadm_init_phase_certs_etcd-server.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md index 0f5fb4726e..c669e2d0ee 100644 --- a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md +++ b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md @@ -6,9 +6,9 @@ -生成用于提供 etcd 服务的证书，并将其保存到 etcd/server.cert 和 etcd/server.key 文件中。 +生成用于提供 etcd 服务的证书，并将其保存到 etcd/server.crt 和 etcd/server.key 文件中。 -保存和存储证书的路径。 +

保存和存储证书的路径。

@@ -64,9 +64,9 @@ The path where to save and store the certificates. -kubeadm 配置文件的路径。 +

kubeadm 配置文件的路径。

@@ -76,9 +76,9 @@ kubeadm 配置文件的路径。 -etcd-server 操作的帮助命令 +

etcd-server 操作的帮助命令

@@ -93,9 +93,9 @@ etcd-server 操作的帮助命令 -为控制平面指定特定的 Kubernetes 版本。 +

为控制平面指定特定的 Kubernetes 版本。

@@ -122,9 +122,9 @@ Choose a specific Kubernetes version for the control plane. -[实验] 到 '真实' 主机根文件系统的路径。 +

[实验] 到 '真实' 主机根文件系统的路径。

From 283448b92202c811ae0e1ba1eba299a0a31657f4 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Fri, 29 Apr 2022 23:38:12 +0800 Subject: [PATCH 083/119] [zh]Update content/zh/docs/concepts/cluster-administration/_index.md --- content/zh/docs/concepts/cluster-administration/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/concepts/cluster-administration/_index.md b/content/zh/docs/concepts/cluster-administration/_index.md index 36e27269f7..e1e929cfa6 100644 --- a/content/zh/docs/concepts/cluster-administration/_index.md +++ b/content/zh/docs/concepts/cluster-administration/_index.md @@ -98,7 +98,7 @@ Before choosing a guide, here are some considerations: * [Authorization](/docs/reference/access-authn-authz/authorization/) is separate from authentication, and controls how HTTP calls are handled. * [Using Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/) explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization. * [Using Sysctls in a Kubernetes Cluster](/docs/concepts/cluster-administration/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters . -* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs. +* [Auditing](/docs/tasks/debug/debug-cluster/audit/) describes how to interact with Kubernetes' audit logs. --> ## 保护集群 {#securing-a-cluster} @@ -116,7 +116,7 @@ Before choosing a guide, here are some considerations: 阐述了在认证和授权之后拦截到 Kubernetes API 服务的请求的插件。 * [在 Kubernetes 集群中使用 Sysctls](/zh/docs/tasks/administer-cluster/sysctl-cluster/) 描述了管理员如何使用 `sysctl` 命令行工具来设置内核参数。 -* [审计](/zh/docs/tasks/debug-application-cluster/audit/) +* [审计](/zh/docs/tasks/debug/debug-cluster/audit/) 描述了如何与 Kubernetes 的审计日志交互。 + + +处理 Kubernetes 证书的相关命令 -与处理 kubernetes 证书相关的命令 +处理 Kubernetes 证书相关的命令 -certs 命令的帮助 + +

certs 命令的帮助

@@ -48,8 +63,8 @@ Commands related to handling kubernetes certificates --rootfs string - -[实验] 到'真实'主机根文件系统的路径。 + +

[实验] 到'真实'主机根文件系统的路径。

From 86c0e4e5b9bb933c8f9c923b42549152b8973563 Mon Sep 17 00:00:00 2001 From: Gerald Mayr Date: Fri, 29 Apr 2022 20:27:22 +0200 Subject: [PATCH 087/119] Add missing verb --- .../migrating-telemetry-and-security-agents.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents.md b/content/en/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents.md index 13219bfd6d..3835057c91 100644 --- a/content/en/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents.md +++ b/content/en/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents.md @@ -13,7 +13,7 @@ Kubernetes' support for direct integration with Docker Engine is deprecated, and ## Telemetry and security agents Within a Kubernetes cluster there are a few different ways to run telemetry or security agents. -Some agents have a direct dependency on Docker Engine when they as DaemonSets or +Some agents have a direct dependency on Docker Engine when they run as DaemonSets or directly on nodes. ### Why do some telemetry agents communicate with Docker Engine? From d9690ad3141136bb641da68d905ee56edab58f1b Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 02:55:03 +0800 Subject: [PATCH 088/119] [en]Update content/en/docs/concepts/workloads/pods/init-containers.md --- content/en/docs/concepts/workloads/pods/init-containers.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/workloads/pods/init-containers.md b/content/en/docs/concepts/workloads/pods/init-containers.md index 43d23d3a0a..371ef58dac 100644 --- a/content/en/docs/concepts/workloads/pods/init-containers.md +++ b/content/en/docs/concepts/workloads/pods/init-containers.md @@ -332,5 +332,6 @@ Kubernetes, consult the documentation for the version you are using. ## {{% heading "whatsnext" %}} * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container) -* Learn how to [debug init containers](/docs/tasks/debug-application-cluster/debug-init-containers/) +* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/) + From be918fc2b3a358913d600f95c06fb269df3eef76 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 03:02:03 +0800 Subject: [PATCH 089/119] [zh]Sync content/zh/docs/tasks/debug/debug-cluster/audit.md --- .../docs/tasks/debug/debug-cluster/audit.md | 453 ++++++++++++++++++ 1 file changed, 453 insertions(+) create mode 100644 content/zh/docs/tasks/debug/debug-cluster/audit.md diff --git a/content/zh/docs/tasks/debug/debug-cluster/audit.md b/content/zh/docs/tasks/debug/debug-cluster/audit.md new file mode 100644 index 0000000000..b4b4c9ee66 --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-cluster/audit.md @@ -0,0 +1,453 @@ +--- +title: 审计 +content_type: concept +--- + + + +{{< feature-state state="beta" >}} + + +Kubernetes _审计（Auditing）_ 功能提供了与安全相关的、按时间顺序排列的记录集， +记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。 + +审计功能使得集群管理员能够回答以下问题： + + + - 发生了什么？ + - 什么时候发生的？ + - 谁触发的？ + - 活动发生在哪个（些）对象上？ + - 在哪观察到的？ + - 它从哪触发的？ + - 活动的后续处理行为是什么？ + + + + +审计记录最初产生于 +[kube-apiserver](/zh/docs/reference/command-line-tools-reference/kube-apiserver/) +内部。每个请求在不同执行阶段都会生成审计事件；这些审计事件会根据特定策略 +被预处理并写入后端。策略确定要记录的内容和用来存储记录的后端。 +当前的后端支持日志文件和 webhook。 + + +每个请求都可被记录其相关的 _阶段（stage）_。已定义的阶段有： + +- `RequestReceived` - 此阶段对应审计处理器接收到请求后，并且在委托给 + 其余处理器之前生成的事件。 +- `ResponseStarted` - 在响应消息的头部发送后，响应消息体发送前生成的事件。 + 只有长时间运行的请求（例如 watch）才会生成这个阶段。 +- `ResponseComplete` - 当响应消息体完成并且没有更多数据需要传输的时候。 +- `Panic` - 当 panic 发生时生成。 + + +{{< note >}} +[审计事件配置](/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event) +的配置与 [Event](/zh/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#event-v1-core) +API 对象不同。 +{{< /note >}} + + +审计日志记录功能会增加 API server 的内存消耗，因为需要为每个请求存储审计所需的某些上下文。 +此外，内存消耗取决于审计日志记录的配置。 + + +## 审计策略 {#audit-policy} + +审计政策定义了关于应记录哪些事件以及应包含哪些数据的规则。 +审计策略对象结构定义在 +[`audit.k8s.io` API 组](/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Policy) +处理事件时，将按顺序与规则列表进行比较。第一个匹配规则设置事件的 +_审计级别（Audit Level）_。已定义的审计级别有： + + +- `None` - 符合这条规则的日志将不会记录。 +- `Metadata` - 记录请求的元数据（请求的用户、时间戳、资源、动词等等）， + 但是不记录请求或者响应的消息体。 +- `Request` - 记录事件的元数据和请求的消息体，但是不记录响应的消息体。 + 这不适用于非资源类型的请求。 +- `RequestResponse` - 记录事件的元数据，请求和响应的消息体。这不适用于非资源类型的请求。 + + +你可以使用 `--audit-policy-file` 标志将包含策略的文件传递给 `kube-apiserver`。 +如果不设置该标志，则不记录事件。 +注意 `rules` 字段 __必须__ 在审计策略文件中提供。没有（0）规则的策略将被视为非法配置。 + +以下是一个审计策略文件的示例： + +{{< codenew file="audit/audit-policy.yaml" >}} + + +你可以使用最低限度的审计策略文件在 `Metadata` 级别记录所有请求： + +```yaml +# 在 Metadata 级别为所有请求生成日志 +apiVersion: audit.k8s.io/v1beta1 +kind: Policy +rules: +- level: Metadata +``` + + +如果你在打磨自己的审计配置文件，你可以使用为 Google Container-Optimized OS +设计的审计配置作为出发点。你可以参考 +[configure-helper.sh](https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh) +脚本，该脚本能够生成审计策略文件。你可以直接在脚本中看到审计策略的绝大部份内容。 + +你也可以参考 [`Policy` 配置参考](/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Policy) +以获取有关已定义字段的详细信息。 + + + +## 审计后端 {#audit-backends} + +审计后端实现将审计事件导出到外部存储。`Kube-apiserver` 默认提供两个后端： + +- Log 后端，将事件写入到文件系统 +- Webhook 后端，将事件发送到外部 HTTP API + +在这所有情况下，审计事件均遵循 Kubernetes API 在 +[`audit.k8s.io` API 组](/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event) +中定义的结构。 + + +{{< note >}} +对于 patch 请求，请求的消息体需要是设定 patch 操作的 JSON 所构成的一个串， +而不是一个完整的 Kubernetes API 对象 JSON 串。 +例如，以下的示例是一个合法的 patch 请求消息体，该请求对应 +`/apis/batch/v1/namespaces/some-namespace/jobs/some-job-name`。 + +```json +[ + { + "op": "replace", + "path": "/spec/parallelism", + "value": 0 + }, + { + "op": "remove", + "path": "/spec/template/spec/containers/0/terminationMessagePolicy" + } +] +``` +{{< /note >}} + + +### Log 后端 + +Log 后端将审计事件写入 [JSONlines](https://jsonlines.org/) 格式的文件。 +你可以使用以下 `kube-apiserver` 标志配置 Log 审计后端： + + +- `--audit-log-path` 指定用来写入审计事件的日志文件路径。不指定此标志会禁用日志后端。`-` 意味着标准化 +- `--audit-log-maxage` 定义保留旧审计日志文件的最大天数 +- `--audit-log-maxbackup` 定义要保留的审计日志文件的最大数量 +- `--audit-log-maxsize` 定义审计日志文件的最大大小（兆字节） + + +如果你的集群控制面以 Pod 的形式运行 kube-apiserver，记得要通过 `hostPath` +卷来访问策略文件和日志文件所在的目录，这样审计记录才会持久保存下来。例如： + +```shell + --audit-policy-file=/etc/kubernetes/audit-policy.yaml + --audit-log-path=/var/log/kubernetes/audit/audit.log +``` + +接下来挂载数据卷： + +```yaml +volumeMounts: + - mountPath: /etc/kubernetes/audit-policy.yaml + name: audit + readOnly: true + - mountPath: /var/log/kubernetes/audit/ + name: audit-log + readOnly: false +``` + + +最后配置 `hostPath`： + +```yaml +... +volumes: +- name: audit + hostPath: + path: /etc/kubernetes/audit-policy.yaml + type: File + +- name: audit-log + hostPath: + path: /var/log/kubernetes/audit/ + type: DirectoryOrCreate +``` + + +### Webhook 后端 {#webhook-backend} + +Webhook 后端将审计事件发送到远程 Web API，该远程 API 应该暴露与 `kube-apiserver` +形式相同的 API，包括其身份认证机制。你可以使用如下 kube-apiserver 标志来配置 +Webhook 审计后端： + + +- `--audit-webhook-config-file` 设置 Webhook 配置文件的路径。Webhook 配置文件实际上是一个 + [kubeconfig 文件](/zh/docs/concepts/configuration/organize-cluster-access-kubeconfig/)。 +- `--audit-webhook-initial-backoff` 指定在第一次失败后重发请求等待的时间。随后的请求将以指数退避重试。 + +Webhook 配置文件使用 kubeconfig 格式指定服务的远程地址和用于连接它的凭据。 + + +## 事件批处理 {#batching} + +日志和 Webhook 后端都支持批处理。以 Webhook 为例，以下是可用参数列表。要获取日志 +后端的同样参数，请在参数名称中将 `webhook` 替换为 `log`。 +默认情况下，在 `webhook` 中批处理是被启用的，在 `log` 中批处理是被禁用的。 +同样，默认情况下，在 `webhook` 中启用带宽限制，在 `log` 中禁用带宽限制。 + + +- `--audit-webhook-mode` 定义缓存策略，可选值如下： + - `batch` - 以批处理缓存事件和异步的过程。这是默认值。 + - `blocking` - 在 API 服务器处理每个单独事件时，阻塞其响应。 + - `blocking-strict` - 与 `blocking` 相同，不过当审计日志在 RequestReceived 阶段 + 失败时，整个 API 服务请求会失效。 + + +以下参数仅用于 `batch` 模式。 + +- `--audit-webhook-batch-buffer-size` 定义 batch 之前要缓存的事件数。 + 如果传入事件的速率溢出缓存区，则会丢弃事件。 +- `--audit-webhook-batch-max-size` 定义一个 batch 中的最大事件数。 +- `--audit-webhook-batch-max-wait` 无条件 batch 队列中的事件前等待的最大事件。 +- `--audit-webhook-batch-throttle-qps` 每秒生成的最大批次数。 +- `--audit-webhook-batch-throttle-burst` 在达到允许的 QPS 前，同一时刻允许存在的最大 batch 生成数。 + + +## 参数调整 {#parameter-tuning} + +需要设置参数以适应 API 服务器上的负载。 + +例如，如果 kube-apiserver 每秒收到 100 个请求，并且每个请求仅在 `ResponseStarted` +和 `ResponseComplete` 阶段进行审计，则应该考虑每秒生成约 200 个审计事件。 +假设批处理中最多有 100 个事件，则应将限制级别设置为每秒至少 2 个查询。 +假设后端最多需要 5 秒钟来写入事件，你应该设置缓冲区大小以容纳最多 5 秒的事件， +即 10 个 batch，即 1000 个事件。 + + +但是，在大多数情况下，默认参数应该足够了，你不必手动设置它们。 +你可以查看 kube-apiserver 公开的以下 Prometheus 指标，并在日志中监控审计子系统的状态。 + +- `apiserver_audit_event_total` 包含所有暴露的审计事件数量的指标。 +- `apiserver_audit_error_total` 在暴露时由于发生错误而被丢弃的事件的数量。 + + +### 日志条目截断 {#truncate} + +日志后端和 Webhook 后端都支持限制所输出的事件的尺寸。 +例如，下面是可以为日志后端配置的标志列表： + + +- `audit-log-truncate-enabled`：是否弃用事件和批次的截断处理。 +- `audit-log-truncate-max-batch-size`：向下层后端发送的各批次的最大尺寸字节数。 +- `audit-log-truncate-max-event-size`：向下层后端发送的审计事件的最大尺寸字节数。 + + +默认情况下，截断操作在 `webhook` 和 `log` 后端都是被禁用的，集群管理员需要设置 +`audit-log-truncate-enabled` 或 `audit-webhook-truncate-enabled` 标志来启用此操作。 + +## {{% heading "whatsnext" %}} + + +* 了解 [Mutating webhook 审计注解](/zh/docs/reference/access-authn-authz/extensible-admission-controllers/#mutating-webhook-auditing-annotations)。 + From 91754929a83f9a85c5cbda0c835cda471c782ce0 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 03:08:22 +0800 Subject: [PATCH 090/119] [zh]Update content/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md --- .../mysql-wordpress-persistent-volume.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md b/content/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md index 5d8ae8b190..8bd486d6d5 100644 --- a/content/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md +++ b/content/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume.md @@ -466,15 +466,15 @@ Do not leave your WordPress installation on this page. If another user finds it, -* 了解更多关于 [Introspection and Debugging](/zh/docs/tasks/debug-application-cluster/debug-application-introspection/) +* 了解更多关于 [Introspection and Debugging](/zh/docs/tasks/debug/debug-application) * 了解更多关于 [Jobs](/zh/docs/concepts/workloads/controllers/jobs-run-to-completion/) * 了解更多关于 [Port Forwarding](/zh/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) -* 了解如何 [Get a Shell to a Container](/zh/docs/tasks/debug-application-cluster/get-shell-running-container/) +* 了解如何 [Get a Shell to a Container](/zh/docs/tasks/debug/debug-application/get-shell-running-container/) From 64f58b2c755f761f2d6118e9cd32f8e0ef4ccf1d Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 03:11:53 +0800 Subject: [PATCH 091/119] [en]Update content/en/docs/reference/kubectl/docker-cli-to-kubectl.md --- content/en/docs/reference/kubectl/docker-cli-to-kubectl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/reference/kubectl/docker-cli-to-kubectl.md b/content/en/docs/reference/kubectl/docker-cli-to-kubectl.md index 9c148702b5..95e2e79c77 100644 --- a/content/en/docs/reference/kubectl/docker-cli-to-kubectl.md +++ b/content/en/docs/reference/kubectl/docker-cli-to-kubectl.md @@ -187,7 +187,7 @@ kubectl exec -ti nginx-app-5jyvm -- /bin/sh # exit ``` -For more information, see [Get a Shell to a Running Container](/docs/tasks/debug-application-cluster/get-shell-running-container/). +For more information, see [Get a Shell to a Running Container](/docs/tasks/debug/debug-application/get-shell-running-container/). ## docker logs From f19090942f8a80d17464e705b0102fadac220414 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 03:28:13 +0800 Subject: [PATCH 092/119] [zh]Update content/zh/docs/doc-contributor-tools/linkchecker/README.md --- content/zh/docs/doc-contributor-tools/linkchecker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/docs/doc-contributor-tools/linkchecker/README.md b/content/zh/docs/doc-contributor-tools/linkchecker/README.md index 016907c2eb..86d1c81683 100644 --- a/content/zh/docs/doc-contributor-tools/linkchecker/README.md +++ b/content/zh/docs/doc-contributor-tools/linkchecker/README.md @@ -106,7 +106,7 @@ To run the link checker: 2. 执行如下命令： ``` - make docker-internal-linkcheck + make container-internal-linkcheck ``` +kubeadm: 轻松创建一个安全的 Kubernetes 集群 ### 摘要 -kubeadm 操作的帮助信息 +

kubeadm 操作的帮助信息

@@ -102,9 +105,9 @@ kubeadm 操作的帮助信息 -[实验] 指向 '真实' 宿主机根文件系统的路径。 +

[实验] 指向 '真实' 宿主机根文件系统的路径。

From 5f5be17db90ba970daeb0139479d02165af49e2a Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sat, 30 Apr 2022 16:16:28 +0800 Subject: [PATCH 096/119] [en] Update tools/_index.md remove old helm definition --- content/en/docs/reference/tools/_index.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/docs/reference/tools/_index.md b/content/en/docs/reference/tools/_index.md index ad10ad1e7c..2ec317bd1f 100644 --- a/content/en/docs/reference/tools/_index.md +++ b/content/en/docs/reference/tools/_index.md @@ -30,9 +30,6 @@ resources itself. [Helm](https://helm.sh/) is a tool for managing packages of pre-configured Kubernetes resources. These packages are known as _Helm charts_. -Helm is a third party managed tool for managing packages of pre-configured -Kubernetes resources, aka Kubernetes charts. - Use Helm to: * Find and use popular software packaged as Kubernetes charts From bf84c6a5a0065e6f2a9e3640be685ff5c8655e94 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Fri, 29 Apr 2022 01:39:44 +0800 Subject: [PATCH 097/119] [zh]Sync content/zh/examples/policy/restricted-psp.yaml [zh]Sync content/zh/examples/policy/restricted-psp.yaml --- content/zh/examples/policy/restricted-psp.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/zh/examples/policy/restricted-psp.yaml b/content/zh/examples/policy/restricted-psp.yaml index 99c836b9ff..53dec3a1af 100644 --- a/content/zh/examples/policy/restricted-psp.yaml +++ b/content/zh/examples/policy/restricted-psp.yaml @@ -3,6 +3,7 @@ kind: PodSecurityPolicy metadata: name: restricted annotations: + # docker/default 标识 seccomp 的配置文件，但它与 Docker 运行时没有特别关联 seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' @@ -45,3 +46,4 @@ spec: - min: 1 max: 65535 readOnlyRootFilesystem: false + From 5810ecfd49e124d002062e985c538685f6cca2c8 Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Fri, 29 Apr 2022 20:46:29 +0800 Subject: [PATCH 098/119] [en] modify debug-cluster/crictl Signed-off-by: xin.li --- content/en/blog/_posts/2020-12-02-dockershim-faq.md | 2 +- content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md | 2 +- .../tools/kubeadm/troubleshooting-kubeadm.md | 2 +- .../debug-application-introspection.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/blog/_posts/2020-12-02-dockershim-faq.md b/content/en/blog/_posts/2020-12-02-dockershim-faq.md index ebf6511af4..607781bab6 100644 --- a/content/en/blog/_posts/2020-12-02-dockershim-faq.md +++ b/content/en/blog/_posts/2020-12-02-dockershim-faq.md @@ -155,7 +155,7 @@ runtime where possible. Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the -former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the +former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the latter you can use newer container build options like [img], [buildah], [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. diff --git a/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md index 6091090567..ece3efa66e 100644 --- a/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md +++ b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md @@ -169,7 +169,7 @@ runtime where possible. Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the -former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the +former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the latter you can use newer container build options like [img], [buildah], [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 9108ecafcd..cb0edc4b3b 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -93,7 +93,7 @@ This may be caused by a number of problems. The most common are: configure it properly see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/). - control plane containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`. For other container runtime see -[Debugging Kubernetes nodes with crictl](/docs/tasks/debug-application-cluster/crictl/). +[Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/). ## kubeadm blocks when removing managed containers diff --git a/content/zh/docs/tasks/debug-application-cluster/debug-application-introspection.md b/content/zh/docs/tasks/debug-application-cluster/debug-application-introspection.md index 7210356959..0c9350aee4 100644 --- a/content/zh/docs/tasks/debug-application-cluster/debug-application-introspection.md +++ b/content/zh/docs/tasks/debug-application-cluster/debug-application-introspection.md @@ -1,4 +1,4 @@ ---- +:--- reviewers: - janetkuo - thockin From fdcf1d8a4c591c0a1bfa526fd2fcdab642d582dd Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sat, 30 Apr 2022 22:46:12 +0800 Subject: [PATCH 099/119] [zh] sync content/zh/docs/tasks/debug/debug-cluster/local-debugging.md --- .../debug-cluster}/local-debugging.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename content/zh/docs/tasks/{debug-application-cluster => debug/debug-cluster}/local-debugging.md (100%) diff --git a/content/zh/docs/tasks/debug-application-cluster/local-debugging.md b/content/zh/docs/tasks/debug/debug-cluster/local-debugging.md similarity index 100% rename from content/zh/docs/tasks/debug-application-cluster/local-debugging.md rename to content/zh/docs/tasks/debug/debug-cluster/local-debugging.md From aaf7e99434e92ad05bdd36a16c5a7efa2e4334a7 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 03:16:38 +0800 Subject: [PATCH 100/119] [zh]Add content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md [zh]Add content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md [zh]Add content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md --- .../get-shell-running-container.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md diff --git a/content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md b/content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md new file mode 100644 index 0000000000..e67d6b1859 --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-application/get-shell-running-container.md @@ -0,0 +1,211 @@ +--- +title: 获取正在运行容器的 Shell +content_type: task +--- + + + + +本文介绍怎样使用 `kubectl exec` 命令获取正在运行容器的 Shell。 + +## {{% heading "prerequisites" %}} + +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} + + + + + +## 获取容器的 Shell + + + +在本练习中，你将创建包含一个容器的 Pod。容器运行 nginx 镜像。下面是 Pod 的配置文件： + +{{< codenew file="application/shell-demo.yaml" >}} + + + +创建 Pod： + +```shell +kubectl create -f https://k8s.io/examples/application/shell-demo.yaml +``` + + + +检查容器是否运行正常： + +```shell +kubectl get pod shell-demo +``` + + + +获取正在运行容器的 Shell： + +```shell +kubectl exec -it shell-demo -- /bin/bash +``` +{{< note >}} + + +双破折号 "--" 用于将要传递给命令的参数与 kubectl 的参数分开。 +{{< /note >}} + + + +在 shell 中，打印根目录： + +```shell +root@shell-demo:/# ls / +``` + + + +在 shell 中，实验其他命令。下面是一些示例： + +```shell +root@shell-demo:/# ls / +root@shell-demo:/# cat /proc/mounts +root@shell-demo:/# cat /proc/1/maps +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install -y tcpdump +root@shell-demo:/# tcpdump +root@shell-demo:/# apt-get install -y lsof +root@shell-demo:/# lsof +root@shell-demo:/# apt-get install -y procps +root@shell-demo:/# ps aux +root@shell-demo:/# ps aux | grep nginx +``` + + + +## 编写 nginx 的根页面 + + + +在看一下 Pod 的配置文件。该 Pod 有个 `emptyDir` 卷，容器将该卷挂载到了 `/usr/share/nginx/html`。 + + + +在 shell 中，在 `/usr/share/nginx/html` 目录创建一个 `index.html` 文件： + +```shell +root@shell-demo:/# echo Hello shell demo > /usr/share/nginx/html/index.html +``` + + + +在 shell 中，向 nginx 服务器发送 GET 请求： + +```shell +root@shell-demo:/# apt-get update +root@shell-demo:/# apt-get install curl +root@shell-demo:/# curl localhost +``` + + + +输出结果显示了你在 `index.html` 中写入的文本。 + +```shell +Hello shell demo +``` + + + +当用完 shell 后，输入 `exit` 退出。 + + + +## 在容器中运行单个命令 + + + +在普通的命令窗口（而不是 shell）中，打印环境运行容器中的变量： + +```shell +kubectl exec shell-demo env +``` + + + +实验运行其他命令。下面是一些示例： + +```shell +kubectl exec shell-demo ps aux +kubectl exec shell-demo ls / +kubectl exec shell-demo cat /proc/1/mounts +``` + + + + + +## 当 Pod 包含多个容器时打开 shell + + + +如果 Pod 有多个容器，`--container` 或者 `-c` 可以在 `kubectl exec` 命令中指定容器。 +例如，您有个名为 my-pod 的容器，该 Pod 有两个容器分别为 main-app 和 healper-app。 +下面的命令将会打开一个 shell 访问 main-app 容器。 + +```shell +kubectl exec -it my-pod --container main-app -- /bin/bash +``` + +## {{% heading "whatsnext" %}} + +* [kubectl exec](/docs/reference/generated/kubectl/kubectl-commands/#exec) From b761b7de9491ec6837bf62697221459bf14d78bc Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 23:45:05 +0800 Subject: [PATCH 101/119] [zh]Update content/zh/docs/concepts/configuration/manage-resources-containers.md --- .../manage-resources-containers.md | 8 +- .../resource-metrics-pipeline.md | 417 ++++++++++++++++++ .../resource-usage-monitoring.md | 113 +++++ 3 files changed, 534 insertions(+), 4 deletions(-) create mode 100644 content/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline.md create mode 100644 content/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring.md diff --git a/content/zh/docs/concepts/configuration/manage-resources-containers.md b/content/zh/docs/concepts/configuration/manage-resources-containers.md index 00c5eba261..c3d69cef95 100644 --- a/content/zh/docs/concepts/configuration/manage-resources-containers.md +++ b/content/zh/docs/concepts/configuration/manage-resources-containers.md @@ -411,9 +411,9 @@ see the [Troubleshooting](#troubleshooting) section. The kubelet reports the resource usage of a Pod as part of the Pod [`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status). -If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) +If optional [tools for monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) are available in your cluster, then Pod resource usage can be retrieved either -from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api) +from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api) directly or from your monitoring tools. --> ## 监控计算和内存资源用量 @@ -422,8 +422,8 @@ kubelet 会将 Pod 的资源使用情况作为 Pod [`status`](/zh/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status) 的一部分来报告的。 -如果为集群配置了可选的[监控工具](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/)， -则可以直接从[指标 API](/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api) +如果为集群配置了可选的[监控工具](/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)， +则可以直接从[指标 API](/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api) 或者监控工具获得 Pod 的资源使用情况。 + + + + + +对于 Kubernetes，_Metrics API_ 提供了一组基本的指标，以支持自动伸缩和类似的用例。 +该 API 提供有关节点和 Pod 的资源使用情况的信息， +包括 CPU 和内存的指标。如果将 Metrics API 部署到集群中， +那么 Kubernetes API 的客户端就可以查询这些信息，并且可以使用 Kubernetes 的访问控制机制来管理权限。 + +[HorizontalPodAutoscaler](/zh/docs/tasks/run-application/horizontal-pod-autoscale/) (HPA) 和 +[VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme) (VPA) +使用 metrics API 中的数据调整工作负载副本和资源，以满足客户需求。 + +你也可以通过 [`kubectl top`](/zh/docs/reference/generated/kubectl/kubectl-commands#top) 命令来查看资源指标。 + +{{< note >}} + +Metrics API 及其启用的指标管道仅提供最少的 CPU 和内存指标，以启用使用 HPA 和/或 VPA 的自动扩展。 +如果你想提供更完整的指标集，你可以通过部署使用 _Custom Metrics API_ 的第二个 +[指标管道](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline) 来作为简单的 Metrics API 的补充。 +{{< /note >}} + + +图 1 说明了资源指标管道的架构。 + +{{< mermaid >}} +flowchart RL +subgraph cluster[Cluster] +direction RL +S[

] +A[Metrics-
Server] +subgraph B[Nodes] +direction TB +D[cAdvisor] --> C[kubelet] +E[Container
runtime] --> D +E1[Container
runtime] --> D +P[pod data] -.- C +end +L[API
server] +W[HPA] +C ---->|Summary
API| A -->|metrics
API| L --> W +end +L ---> K[kubectl
top] +classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; +class W,B,P,K,cluster,D,E,E1 box +classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 +class S spacewhite +classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; +class A,L,C k8s +{{< /mermaid >}} + + +图 1. 资源指标管道 + +图中从右到左的架构组件包括以下内容： + +* [cAdvisor](https://github.com/google/cadvisor): 用于收集、聚合和公开 Kubelet 中包含的容器指标的守护程序。 +* [kubelet](/zh/docs/concepts/overview/components/#kubelet): 用于管理容器资源的节点代理。 + 可以使用 `/metrics/resource` 和 `/stats` kubelet API 端点访问资源指标。 +* [Summary API](#summary-api-source): kubelet 提供的 API，用于发现和检索可通过 `/stats` 端点获得的每个节点的汇总统计信息。 +* [metrics-server](#metrics-server): 集群插件组件，用于收集和聚合从每个 kubelet 中提取的资源指标。 + API 服务器提供 Metrics API 以供 HPA、VPA 和 `kubectl top` 命令使用。 Metrics Server 是 Metrics API 的参考实现。 +* [Metrics API](#metrics-api): Kubernetes API 支持访问用于工作负载自动缩放的 CPU 和内存。 + 要在你的集群中进行这项工作，你需要一个提供 Metrics API 的 API 扩展服务器。 + + + {{< note >}} + cAdvisor 支持从 cgroups 读取指标，它适用于 Linux 上的典型容器运行时。 + 如果你使用基于其他资源隔离机制的容器运行时，例如虚拟化，那么该容器运行时必须支持 + [CRI 容器指标](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md) + 以便 kubelet 可以使用指标。 + {{< /note >}} + + + + +## Metrics API {#the-metrics-api} + +{{< feature-state for_k8s_version="1.8" state="beta" >}} + +metrics-server 实现了 Metrics API。此 API 允许你访问集群中节点和 Pod 的 CPU 和内存使用情况。 +它的主要作用是将资源使用指标提供给 K8s 自动缩放器组件。 + +下面是一个 `minikube` 节点的 Metrics API 请求示例，通过 `jq` 管道处理以便于阅读： + +```shell +kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.' +``` + + +这是使用 `curl` 来执行的相同 API 调用： + +```shell +curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube +``` + + +响应示例： + +```json +{ + "kind": "NodeMetrics", + "apiVersion": "metrics.k8s.io/v1beta1", + "metadata": { + "name": "minikube", + "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube", + "creationTimestamp": "2022-01-27T18:48:43Z" + }, + "timestamp": "2022-01-27T18:48:33Z", + "window": "30s", + "usage": { + "cpu": "487558164n", + "memory": "732212Ki" + } +} +``` + + + +下面是一个 `kube-system` 命名空间中的 `kube-scheduler-minikube` Pod 的 Metrics API 请求示例， +通过 `jq` 管道处理以便于阅读： + +```shell +kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.' +``` + + +这是使用 `curl` 来完成的相同 API 调用： + +```shell +curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube +``` + + +响应示例： + +```json +{ + "kind": "PodMetrics", + "apiVersion": "metrics.k8s.io/v1beta1", + "metadata": { + "name": "kube-scheduler-minikube", + "namespace": "kube-system", + "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube", + "creationTimestamp": "2022-01-27T19:25:00Z" + }, + "timestamp": "2022-01-27T19:24:31Z", + "window": "30s", + "containers": [ + { + "name": "kube-scheduler", + "usage": { + "cpu": "9559630n", + "memory": "22244Ki" + } + } + ] +} +``` + + + +Metrics API 在 [k8s.io/metrics](https://github.com/kubernetes/metrics) 代码库中定义。 +你必须启用 [API 聚合层](/zh/docs/tasks/extend-kubernetes/configure-aggregation-layer/)并为 +`metrics.k8s.io` API 注册一个 [APIService](/zh/docs/reference/kubernetes-api/cluster-resources/api-service-v1/)。 + +要了解有关 Metrics API 的更多信息， +请参阅资源 [Resource Metrics API Design](https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md)、 +[metrics-server 代码库](https://github.com/kubernetes-sigs/metrics-server) 和 +[Resource Metrics API](https://github.com/kubernetes/metrics#resource-metrics-api)。 + + +{{< note >}} +你必须部署提供 Metrics API 服务的 metrics-server 或其他适配器才能访问它。 +{{< /note >}} + + +## 度量资源用量 {#measuring-resource-usage} + +### CPU + +CPU 报告为以 cpu 为单位测量的平均核心使用率。在 Kubernetes 中， +一个 cpu 相当于云提供商的 1 个 vCPU/Core，以及裸机 Intel 处理器上的 1 个超线程。 + +该值是通过对内核提供的累积 CPU 计数器（在 Linux 和 Windows 内核中）取一个速率得出的。 +用于计算 CPU 的时间窗口显示在 Metrics API 的窗口字段下。 + +要了解更多关于 Kubernetes 如何分配和测量 CPU 资源的信息，请参阅 +[CPU 的含义](/zh/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu)。 + + +### 内存 {#memory} + +内存报告为在收集度量标准的那一刻的工作集大小，以字节为单位。 + +在理想情况下，“工作集”是在内存压力下无法释放的正在使用的内存量。 +然而，工作集的计算因主机操作系统而异，并且通常大量使用启发式算法来产生估计。 + +Kubernetes 模型中，容器工作集是由容器运行时计算的与相关容器关联的匿名内存。 +工作集指标通常还包括一些缓存（文件支持）内存，因为主机操作系统不能总是回收页面。 + +要了解有关 Kubernetes 如何分配和测量内存资源的更多信息， +请参阅[内存的含义](/zh/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory)。 + + +## Metrics 服务器 {#metrics-server} + +metrics-server 从 kubelet 中获取资源指标，并通过 Metrics API 在 Kubernetes API 服务器中公开它们，以供 HPA 和 VPA 使用。 +你还可以使用 `kubectl top` 命令查看这些指标。 + +metrics-server 使用 Kubernetes API 来跟踪集群中的节点和 Pod。metrics-server 服务器通过 HTTP 查询每个节点以获取指标。 +metrics-server 还构建了 Pod 元数据的内部视图，并维护 Pod 健康状况的缓存。 +缓存的 Pod 健康信息可通过 metrics-server 提供的扩展 API 获得。 + +例如，对于 HPA 查询，metrics-server 需要确定哪些 Pod 满足 Deployment 中的标签选择器。 + + +metrics-server 调用 [kubelet](/zh/docs/reference/command-line-tools-reference/kubelet/) API +从每个节点收集指标。根据它使用的度量服务器版本： + +* 版本 v0.6.0+ 中，使用指标资源端点 `/metrics/resource` +* 旧版本中使用 Summary API 端点 `/stats/summary` + + + +了解更多 metrics-server，参阅 [metrics-server 代码库](https://github.com/kubernetes-sigs/metrics-server)。 + +你还可以查看以下内容： + +* [metrics-server 设计](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md) +* [metrics-server FAQ](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md) +* [metrics-server known issues](https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md) +* [metrics-server releases](https://github.com/kubernetes-sigs/metrics-server/releases) +* [Horizontal Pod Autoscaling](/zh/docs/tasks/run-application/horizontal-pod-autoscale/) + + + +### Summary API 来源 + +[Kubelet](/zh/docs/reference/command-line-tools-reference/kubelet/) 在节点、卷、Pod 和容器级别收集统计信息， +并在[Summary API](https://github.com/kubernetes/kubernetes/blob/7d309e0104fedb57280b261e5677d919cb2a0e2d/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go) +中提供它们的统计信息供消费者阅读。 + + + +下面是一个 `minikube` 节点的 Summary API 请求示例： + +```shell +kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary" +``` + + +这是使用 `curl` 来执行的相同 API 调用： + +```shell +curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary +``` + +{{< note >}} + +从 metrics-server 0.6.x 开始，Summary API `/stats/summary` 端点被 `/metrics/resource` 端点替换。 +{{< /note >}} diff --git a/content/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring.md b/content/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring.md new file mode 100644 index 0000000000..86345a09a8 --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring.md @@ -0,0 +1,113 @@ +--- +content_type: concept +title: 资源监控工具 +--- + + + + + +要扩展应用程序并提供可靠的服务，你需要了解应用程序在部署时的行为。 +你可以通过检测容器检查 Kubernetes 集群中的应用程序性能， +[Pods](/zh/docs/concepts/workloads/pods), +[服务](/zh/docs/concepts/services-networking/service/) +和整个集群的特征。 +Kubernetes 在每个级别上提供有关应用程序资源使用情况的详细信息。 +此信息使你可以评估应用程序的性能，以及在何处可以消除瓶颈以提高整体性能。 + + + + +在 Kubernetes 中，应用程序监控不依赖单个监控解决方案。 +在新集群上，你可以使用[资源度量](#resource-metrics-pipeline)或 +[完整度量](#full-metrics-pipeline)管道来收集监视统计信息。 + + +## 资源度量管道 {#resource-metrics-pipeline} + +资源指标管道提供了一组与集群组件，例如 +[Horizontal Pod Autoscaler](/zh/docs/tasks/run-application/horizontal-pod-autoscale/) +控制器以及 `kubectl top` 实用程序相关的有限度量。 +这些指标是由轻量级的、短期、内存存储的 +[metrics-server](https://github.com/kubernetes-sigs/metrics-server) 收集的， +通过 `metrics.k8s.io` 公开。 + + +度量服务器发现集群中的所有节点，并且查询每个节点的 +[kubelet](/zh/docs/reference/command-line-tools-reference/kubelet/) +以获取 CPU 和内存使用情况。 +Kubelet 充当 Kubernetes 主节点与节点之间的桥梁，管理机器上运行的 Pod 和容器。 +kubelet 将每个 Pod 转换为其组成的容器，并在容器运行时通过容器运行时接口 +获取各个容器使用情况统计信息。 +kubelet 从集成的 cAdvisor 获取此信息，以进行旧式 Docker 集成。 +然后，它通过 metrics-server Resource Metrics API 公开聚合的 pod 资源使用情况统计信息。 +该 API 在 kubelet 的经过身份验证和只读的端口上的 `/metrics/resource/v1beta1` 中提供。 + + +## 完整度量管道 {#full-metrics-pipeline} + +一个完整度量管道可以让你访问更丰富的度量。 +Kubernetes 还可以根据集群的当前状态，使用 Pod 水平自动扩缩器等机制， +通过自动调用扩展或调整集群来响应这些度量。 +监控管道从 kubelet 获取度量值，然后通过适配器将它们公开给 Kubernetes， +方法是实现 `custom.metrics.k8s.io` 或 `external.metrics.k8s.io` API。 + + +[Prometheus](https://prometheus.io) 是一个 CNCF 项目，可以原生监控 Kubernetes、 +节点和 Prometheus 本身。 +完整度量管道项目不属于 CNCF 的一部分，不在 Kubernetes 文档的范围之内。 + From 59b33cc694fef415b28b22e3cf9c808382fd9d66 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sat, 30 Apr 2022 23:52:09 +0800 Subject: [PATCH 102/119] [zh]Update content/zh/docs/concepts/overview/components.md --- content/zh/docs/concepts/overview/components.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/zh/docs/concepts/overview/components.md b/content/zh/docs/concepts/overview/components.md index fd359fc1ce..f67ee07b6e 100644 --- a/content/zh/docs/concepts/overview/components.md +++ b/content/zh/docs/concepts/overview/components.md @@ -210,12 +210,12 @@ Kubernetes 启动的容器自动将此 DNS 服务器包含在其 DNS 搜索列 ### 容器资源监控 -[容器资源监控](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/) +[容器资源监控](/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) 将关于容器的一些常见的时间序列度量值保存到一个集中的数据库中，并提供用于浏览这些数据的界面。 From f0990a0f299db8dd7b5947b9d5c6fe2acadd50be Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sun, 1 May 2022 00:10:45 +0800 Subject: [PATCH 104/119] update --- content/zh/docs/tasks/debug/debug-cluster/local-debugging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/tasks/debug/debug-cluster/local-debugging.md b/content/zh/docs/tasks/debug/debug-cluster/local-debugging.md index aca2f2e7e8..c296d44cdf 100644 --- a/content/zh/docs/tasks/debug/debug-cluster/local-debugging.md +++ b/content/zh/docs/tasks/debug/debug-cluster/local-debugging.md @@ -1,10 +1,10 @@ --- -title: 在本地开发和调试服务 +title: 使用 telepresence 在本地开发和调试服务 content_type: task --- From e6c738a96b493e92774e27e6875db41951bfd97c Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sun, 1 May 2022 00:34:09 +0800 Subject: [PATCH 105/119] [zh] update content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md --- .../kubeadm/generated/kubeadm_upgrade.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md index d03885437f..b1a6499f94 100644 --- a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md +++ b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md @@ -1,3 +1,8 @@ + + +此命令能将集群平滑升级到新版本 -upgrade 操作的帮助命令 +

upgrade 操作的帮助命令

@@ -62,9 +67,9 @@ upgrade 操作的帮助命令 -[实验] 指向 '真实' 宿主机根文件系统的路径。 +

[实验] 指向 '真实' 宿主机根文件系统的路径。

From 708924d9d5b5f231b6205aa5803ce60728435e50 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 00:57:38 +0800 Subject: [PATCH 106/119] [en]Modify /docs/tasks/debug/debug-application/debug-running-pod/ --- content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md | 2 +- content/en/docs/concepts/workloads/pods/ephemeral-containers.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md b/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md index 8eb382cac1..295c8d7d36 100644 --- a/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md +++ b/content/en/blog/_posts/2020-12-08-kubernetes-release-1.20.md @@ -32,7 +32,7 @@ The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin. -Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/). +Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/). ### Beta: API Priority and Fairness diff --git a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md index d32511da28..0a70fedd6f 100644 --- a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md +++ b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md @@ -70,5 +70,5 @@ you can view processes in other containers. ## {{% heading "whatsnext" %}} -* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container). +* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container). From cd693ade42ff8c809ce9c799c65d3a7b6e4de160 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 01:11:32 +0800 Subject: [PATCH 107/119] [zh]Update content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew.md --- .../kubeadm/generated/kubeadm_certs_renew.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew.md b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew.md index 58fff42d32..d8102a19ce 100644 --- a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew.md +++ b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew.md @@ -1,3 +1,18 @@ + + + +为 Kubernetes 集群更新证书 renew 操作的帮助命令 +

@@ -58,10 +75,12 @@ renew 操作的帮助命令 +

[实验] 到 '真实' 主机根文件系统的路径。 +

From 9716dd08cf5eda366258597b5f1c4acfecbc79f8 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 01:34:21 +0800 Subject: [PATCH 108/119] [zh]Sync content/zh/docs/tasks/debug/debug-cluster/crictl.md --- .../docs/tasks/debug/debug-cluster/crictl.md | 566 ++++++++++++++++++ 1 file changed, 566 insertions(+) create mode 100644 content/zh/docs/tasks/debug/debug-cluster/crictl.md diff --git a/content/zh/docs/tasks/debug/debug-cluster/crictl.md b/content/zh/docs/tasks/debug/debug-cluster/crictl.md new file mode 100644 index 0000000000..3eb5cf4955 --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-cluster/crictl.md @@ -0,0 +1,566 @@ +--- +reviewers: +- Random-Liu +- feiskyer +- mrunalp +title: 使用 crictl 对 Kubernetes 节点进行调试 +content_type: task +--- + + + + + +{{< feature-state for_k8s_version="v1.11" state="stable" >}} + + + +`crictl` 是 CRI 兼容的容器运行时命令行接口。 +你可以使用它来检查和调试 Kubernetes 节点上的容器运行时和应用程序。 +`crictl` 和它的源代码在 +[cri-tools](https://github.com/kubernetes-sigs/cri-tools) 代码库。 + +## {{% heading "prerequisites" %}} + + +`crictl` 需要带有 CRI 运行时的 Linux 操作系统。 + + + + +## 安装 crictl + +你可以从 cri-tools [发布页面](https://github.com/kubernetes-sigs/cri-tools/releases) +下载一个压缩的 `crictl` 归档文件，用于几种不同的架构。 +下载与你的 kubernetes 版本相对应的版本。 +提取它并将其移动到系统路径上的某个位置，例如`/usr/local/bin/`。 + + +## 一般用法 + +`crictl` 命令有几个子命令和运行时参数。 +有关详细信息，请使用 `crictl help` 或 `crictl help` 获取帮助信息。 + + +你可以用以下方法之一来为 `crictl` 设置端点： + + +- 设置参数 `--runtime-endpoint` 和 `--image-endpoint`。 +- 设置环境变量 `CONTAINER_RUNTIME_ENDPOINT` 和 `IMAGE_SERVICE_ENDPOINT`。 +- 在配置文件 `--config=/etc/crictl.yaml` 中设置端点。 + 要设置不同的文件，可以在运行 `crictl` 时使用 `--config=PATH_TO_FILE` 标志。 + + +你还可以在连接到服务器并启用或禁用调试时指定超时值，方法是在配置文件中指定 +`timeout` 或 `debug` 值，或者使用 `--timeout` 和 `--debug` 命令行参数。 + + +要查看或编辑当前配置，请查看或编辑 `/etc/crictl.yaml` 的内容。 +例如，使用 `containerd` 容器运行时的配置会类似于这样： + +``` +runtime-endpoint: unix:///var/run/containerd/containerd.sock +image-endpoint: unix:///var/run/containerd/containerd.sock +timeout: 10 +debug: true +``` + + +要进一步了解 `crictl`，参阅 +[`crictl` 文档](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)。 + + +## crictl 命令示例 + +{{< warning >}} + +如果使用 `crictl` 在正在运行的 Kubernetes 集群上创建 Pod 沙盒或容器， +kubelet 最终将删除它们。 +`crictl` 不是一个通用的工作流工具，而是一个对调试有用的工具。 +{{< /warning >}} + + +### 打印 Pod 清单 + +打印所有 Pod 的清单： + +```shell +crictl pods +``` + + +输出类似于： + +``` +POD ID CREATED STATE NAME NAMESPACE ATTEMPT +926f1b5a1d33a About a minute ago Ready sh-84d7dcf559-4r2gq default 0 +4dccb216c4adb About a minute ago Ready nginx-65899c769f-wv2gp default 0 +a86316e96fa89 17 hours ago Ready kube-proxy-gblk4 kube-system 0 +919630b8f81f1 17 hours ago Ready nvidia-device-plugin-zgbbv kube-system 0 +``` + + +根据名称打印 Pod 清单： + +```shell +crictl pods --name nginx-65899c769f-wv2gp +``` + + +输出类似于这样： + +``` +POD ID CREATED STATE NAME NAMESPACE ATTEMPT +4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0 +``` + + +根据标签打印 Pod 清单： + +```shell +crictl pods --label run=nginx +``` + + +输出类似于这样： + +```none +POD ID CREATED STATE NAME NAMESPACE ATTEMPT +4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0 +``` + + +### 打印镜像清单 + +打印所有镜像清单： + +```shell +crictl images +``` + + +输出类似于这样： + +```none +IMAGE TAG IMAGE ID SIZE +busybox latest 8c811b4aec35f 1.15MB +k8s-gcrio.azureedge.net/hyperkube-amd64 v1.10.3 e179bbfe5d238 665MB +k8s-gcrio.azureedge.net/pause-amd64 3.1 da86e6ba6ca19 742kB +nginx latest cd5239a0906a6 109MB +``` + + +根据仓库打印镜像清单： + +```shell +crictl images nginx +``` + + +输出类似于这样： + +```none +IMAGE TAG IMAGE ID SIZE +nginx latest cd5239a0906a6 109MB +``` + + +只打印镜像 ID： + +```shell +crictl images -q +``` + + +输出类似于这样： + +```none +sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a +sha256:e179bbfe5d238de6069f3b03fccbecc3fb4f2019af741bfff1233c4d7b2970c5 +sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e +sha256:cd5239a0906a6ccf0562354852fae04bc5b52d72a2aff9a871ddb6bd57553569 +``` + + +### 打印容器清单 + +打印所有容器清单： + +```shell +crictl ps -a +``` + + +输出类似于这样： + +```none +CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT +1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 7 minutes ago Running sh 1 +9c5951df22c78 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 8 minutes ago Exited sh 0 +87d3992f84f74 nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 8 minutes ago Running nginx 0 +1941fb4da154f k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a 18 hours ago Running kube-proxy 0 +``` + + +打印正在运行的容器清单： + +```shell +crictl ps +``` + + +输出类似于这样： + +```none +CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT +1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 6 minutes ago Running sh 1 +87d3992f84f74 nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 7 minutes ago Running nginx 0 +1941fb4da154f k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a 17 hours ago Running kube-proxy 0 +``` + + +### 在正在运行的容器上执行命令 + +```shell +crictl exec -i -t 1f73f2d81bf98 ls +``` + + +输出类似于这样： + +```none +bin dev etc home proc root sys tmp usr var +``` + + +### 获取容器日志 + +获取容器的所有日志： + +```shell +crictl logs 87d3992f84f74 +``` + + +输出类似于这样： + +```none +10.240.0.96 - - [06/Jun/2018:02:45:49 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" +10.240.0.96 - - [06/Jun/2018:02:45:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" +10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" +``` + + +获取最近的 `N` 行日志： + +```shell +crictl logs --tail=1 87d3992f84f74 +``` + + +输出类似于这样： + +```none +10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-" +``` + + +### 运行 Pod 沙盒 + +用 `crictl` 运行 Pod 沙盒对容器运行时排错很有帮助。 +在运行的 Kubernetes 集群中，沙盒会随机地被 kubelet 停止和删除。 + + +1. 编写下面的 JSON 文件： + + ```json + { + "metadata": { + "name": "nginx-sandbox", + "namespace": "default", + "attempt": 1, + "uid": "hdishd83djaidwnduwk28bcsb" + }, + "logDirectory": "/tmp", + "linux": { + } + } + ``` + + +2. 使用 `crictl runp` 命令应用 JSON 文件并运行沙盒。 + + ```shell + crictl runp pod-config.json + ``` + + + 返回了沙盒的 ID。 + + +### 创建容器 + +用 `crictl` 创建容器对容器运行时排错很有帮助。 +在运行的 Kubernetes 集群中，沙盒会随机的被 kubelet 停止和删除。 + + +1. 拉取 busybox 镜像 + + ```shell + crictl pull busybox + ``` + ```none + Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 + ``` + + +2. 创建 Pod 和容器的配置： + + + **Pod 配置**： + + ```json + { + "metadata": { + "name": "nginx-sandbox", + "namespace": "default", + "attempt": 1, + "uid": "hdishd83djaidwnduwk28bcsb" + }, + "log_directory": "/tmp", + "linux": { + } + } + ``` + + + **容器配置**： + + ```json + { + "metadata": { + "name": "busybox" + }, + "image":{ + "image": "busybox" + }, + "command": [ + "top" + ], + "log_path":"busybox.log", + "linux": { + } + } + ``` + + +3. 创建容器，传递先前创建的 Pod 的 ID、容器配置文件和 Pod 配置文件。返回容器的 ID。 + + ```bash + crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json + ``` + + +4. 查询所有容器并确认新创建的容器状态为 `Created`。 + + ```bash + crictl ps -a + ``` + + 输出类似于这样： + + ```none + CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT + 3e025dd50a72d busybox 32 seconds ago Created busybox 0 + ``` + + +### 启动容器 + +要启动容器，要将容器 ID 传给 `crictl start`： + +```shell +crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60 +``` + + +输出类似于这样： + +``` +3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60 +``` + + +确认容器的状态为 `Running`。 + +```shell +crictl ps +``` + + +输出类似于这样： + +``` +CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT +3e025dd50a72d busybox About a minute ago Running busybox 0 +``` + +## {{% heading "whatsnext" %}} + + +* [进一步了解 `crictl`](https://github.com/kubernetes-sigs/cri-tools). +* [将 `docker` CLI 命令映射到 `crictl`](/zh/docs/reference/tools/map-crictl-dockercli/). From b008adb1e678ce9e14a72ef8af3db842356b92fa Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 02:01:28 +0800 Subject: [PATCH 109/119] [zh]Sync content/zh/docs/tasks/debug/debug-application/debug-service.md --- .../debug/debug-application/debug-service.md | 1116 +++++++++++++++++ 1 file changed, 1116 insertions(+) create mode 100644 content/zh/docs/tasks/debug/debug-application/debug-service.md diff --git a/content/zh/docs/tasks/debug/debug-application/debug-service.md b/content/zh/docs/tasks/debug/debug-application/debug-service.md new file mode 100644 index 0000000000..f6e9b6401d --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-application/debug-service.md @@ -0,0 +1,1116 @@ +--- +content_type: concept +title: 调试 Service +--- + + + + + +对于新安装的 Kubernetes，经常出现的问题是 Service 无法正常运行。 你已经通过 +Deployment（或其他工作负载控制器）运行了 Pod，并创建 Service ，但是 +当你尝试访问它时，没有任何响应。此文档有望对你有所帮助并找出问题所在。 + + + +## 在 Pod 中运行命令 + +对于这里的许多步骤，你可能希望知道运行在集群中的 Pod 看起来是什么样的。 +最简单的方法是运行一个交互式的 busybox Pod： + +```none +kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh +``` + + +{{< note >}} +如果没有看到命令提示符，请按回车。 +{{< /note >}} + + +如果你已经有了你想使用的正在运行的 Pod，则可以运行以下命令去进入： + +```shell +kubectl exec -c -- +``` + + +## 设置 {#setup} + +为了完成本次实践的任务，我们先运行几个 Pod。 +由于你可能正在调试自己的 Service，所以，你可以使用自己的信息进行替换， +或者你也可以跟着教程并开始下面的步骤来获得第二个数据点。 + +```shell +kubectl create deployment hostnames --image=k8s.gcr.io/serve_hostname +``` + +```none +deployment.apps/hostnames created +``` + + +`kubectl` 命令将打印创建或变更的资源的类型和名称，它们可以在后续命令中使用。 +让我们将这个 deployment 的副本数扩至 3。 + +```shell +kubectl scale deployment hostnames --replicas=3 +``` + +```none +deployment.apps/hostnames scaled +``` + + +请注意这与你使用以下 YAML 方式启动 Deployment 类似： + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: hostnames + name: hostnames +spec: + selector: + matchLabels: + app: hostnames + replicas: 3 + template: + metadata: + labels: + app: hostnames + spec: + containers: + - name: hostnames + image: k8s.gcr.io/serve_hostname +``` + + +"app" 标签是 `kubectl create deployment` 根据 Deployment 名称自动设置的。 + +确认你的 Pods 是运行状态: + +```shell +kubectl get pods -l app=hostnames +``` + +```none +NAME READY STATUS RESTARTS AGE +hostnames-632524106-bbpiw 1/1 Running 0 2m +hostnames-632524106-ly40y 1/1 Running 0 2m +hostnames-632524106-tlaok 1/1 Running 0 2m +``` + + +你还可以确认你的 Pod 是否正在提供服务。你可以获取 Pod IP 地址列表并直接对其进行测试。 + +```shell +kubectl get pods -l app=hostnames \ + -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}' +``` + +```none +10.244.0.5 +10.244.0.6 +10.244.0.7 +``` + + +用于本教程的示例容器通过 HTTP 在端口 9376 上提供其自己的主机名， +但是如果要调试自己的应用程序，则需要使用你的 Pod 正在侦听的端口号。 + +在 Pod 内运行： + +```shell +for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do + wget -qO- $ep +done +``` + + +输出类似这样： + +``` +hostnames-632524106-bbpiw +hostnames-632524106-ly40y +hostnames-632524106-tlaok +``` + + +如果此时你没有收到期望的响应，则你的 Pod 状态可能不健康，或者可能没有在你认为正确的端口上进行监听。 +你可能会发现 `kubectl logs` 命令对于查看正在发生的事情很有用， +或者你可能需要通过`kubectl exec` 直接进入 Pod 中并从那里进行调试。 + +假设到目前为止一切都已按计划进行，那么你可以开始调查为何你的 Service 无法正常工作。 + + +## Service 是否存在？ + +细心的读者会注意到我们实际上尚未创建 Service -这是有意而为之。 这一步有时会被遗忘，这是首先要检查的步骤。 + +那么，如果我尝试访问不存在的 Service 会怎样？ 假设你有另一个 Pod 通过名称匹配到 Service ，你将得到类似结果： + +```shell +wget -O- hostnames +``` + +```none +Resolving hostnames (hostnames)... failed: Name or service not known. +wget: unable to resolve host address 'hostnames' +``` + + +首先要检查的是该 Service 是否真实存在： + +```shell +kubectl get svc hostnames +``` + +```none +No resources found. +Error from server (NotFound): services "hostnames" not found +``` + + +让我们创建 Service。 和以前一样，在这次实践中 - 你可以在此处使用自己的 Service 的内容。 + +```shell +kubectl expose deployment hostnames --port=80 --target-port=9376 +``` + +```none +service/hostnames exposed +``` + + +重新运行查询命令： + +```shell +kubectl get svc hostnames +``` + +```none +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +hostnames ClusterIP 10.0.1.175 80/TCP 5s +``` + + +现在你知道了 Service 确实存在。 + +同前，此步骤效果与通过 YAML 方式启动 'Service' 一样： + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: hostnames +spec: + selector: + app: hostnames + ports: + - name: default + protocol: TCP + port: 80 + targetPort: 9376 +``` + + +为了突出配置范围的完整性，你在此处创建的 Service 使用的端口号与 Pods 不同。 +对于许多真实的 Service，这些值可以是相同的。 + + + +## 是否存在影响目标 Pod 的网络策略入站规则？ + +如果你部署了任何可能影响到 `hostnames-*` Pod 的传入流量的网络策略入站规则， +则需要对其进行检查。 + +详细信息，请参阅[网络策略](/zh/docs/concepts/services-networking/network-policies/)。 + + +## Service 是否可通过 DNS 名字访问？ + +通常客户端通过 DNS 名称来匹配到 Service。 + +从相同命名空间下的 Pod 中运行以下命令： + +```shell +nslookup hostnames +``` + +```none +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +Name: hostnames +Address 1: 10.0.1.175 hostnames.default.svc.cluster.local +``` + + +如果失败，那么你的 Pod 和 Service 可能位于不同的命名空间中， +请尝试使用限定命名空间的名称（同样在 Pod 内运行）： + +```shell +nslookup hostnames.default +``` + +```none +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +Name: hostnames.default +Address 1: 10.0.1.175 hostnames.default.svc.cluster.local +``` + + +如果成功，那么需要调整你的应用，使用跨命名空间的名称去访问它， +或者在相同的命名空间中运行应用和 Service。如果仍然失败，请尝试一个完全限定的名称： + +```shell +nslookup hostnames.default.svc.cluster.local +``` + +```none +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +Name: hostnames.default.svc.cluster.local +Address 1: 10.0.1.175 hostnames.default.svc.cluster.local +``` + + +注意这里的后缀："default.svc.cluster.local"。"default" 是我们正在操作的命名空间。 +"svc" 表示这是一个 Service。"cluster.local" 是你的集群域，在你自己的集群中可能会有所不同。 + +你也可以在集群中的节点上尝试此操作： + +{{< note >}} +10.0.0.10 是集群的 DNS 服务 IP，你的可能有所不同。 +{{< /note >}} + +```shell +nslookup hostnames.default.svc.cluster.local 10.0.0.10 +``` + +```none +Server: 10.0.0.10 +Address: 10.0.0.10#53 + +Name: hostnames.default.svc.cluster.local +Address: 10.0.1.175 +``` + + +如果你能够使用完全限定的名称查找，但不能使用相对名称，则需要检查你 Pod 中的 +`/etc/resolv.conf` 文件是否正确。在 Pod 中运行以下命令： + +```shell +cat /etc/resolv.conf +``` + + +你应该可以看到类似这样的输出： + +``` +nameserver 10.0.0.10 +search default.svc.cluster.local svc.cluster.local cluster.local example.com +options ndots:5 +``` + + +`nameserver` 行必须指示你的集群的 DNS Service， +它是通过 `--cluster-dns` 标志传递到 kubelet 的。 + +`search` 行必须包含一个适当的后缀，以便查找 Service 名称。 +在本例中，它查找本地命名空间（`default.svc.cluster.local`）中的服务和 +所有命名空间（`svc.cluster.local`）中的服务，最后在集群（`cluster.local`）中查找 +服务的名称。根据你自己的安装情况，可能会有额外的记录（最多 6 条）。 +集群后缀是通过 `--cluster-domain` 标志传递给 `kubelet` 的。 +本文中，我们假定后缀是 “cluster.local”。 +你的集群配置可能不同，这种情况下，你应该在上面的所有命令中更改它。 + +`options` 行必须设置足够高的 `ndots`，以便 DNS 客户端库考虑搜索路径。 +在默认情况下，Kubernetes 将这个值设置为 5，这个值足够高，足以覆盖它生成的所有 DNS 名称。 + + +### 是否存在 Service 能通过 DNS 名称访问？{#does-any-service-exist-in-dns} + +如果上面的方式仍然失败，DNS 查找不到你需要的 Service ，你可以后退一步， +看看还有什么其它东西没有正常工作。 +Kubernetes 主 Service 应该一直是工作的。在 Pod 中运行如下命令： + +```shell +nslookup kubernetes.default +``` +```none +Server: 10.0.0.10 +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +Name: kubernetes.default +Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local +``` + + +如果失败，你可能需要转到本文的 [kube-proxy](#is-the-kube-proxy-working) 节， +或者甚至回到文档的顶部重新开始，但不是调试你自己的 Service ，而是调试 DNS Service。 + + +### Service 能够通过 IP 访问么？ + +假设你已经确认 DNS 工作正常，那么接下来要测试的是你的 Service 能否通过它的 IP 正常访问。 +从集群中的一个 Pod，尝试访问 Service 的 IP（从上面的 `kubectl get` 命令获取）。 + +```shell +for i in $(seq 1 3); do + wget -qO- 10.0.1.175:80 +done +``` + + +输出应该类似这样： + +``` +hostnames-632524106-bbpiw +hostnames-632524106-ly40y +hostnames-632524106-tlaok +``` + + +如果 Service 状态是正常的，你应该得到正确的响应。如果没有，有很多可能出错的地方，请继续阅读。 + + +## Service 的配置是否正确？ + +这听起来可能很愚蠢，但你应该两次甚至三次检查你的 Service 配置是否正确，并且与你的 Pod 匹配。 +查看你的 Service 配置并验证它： + +```shell +kubectl get service hostnames -o json +``` + +```json +{ + "kind": "Service", + "apiVersion": "v1", + "metadata": { + "name": "hostnames", + "namespace": "default", + "uid": "428c8b6c-24bc-11e5-936d-42010af0a9bc", + "resourceVersion": "347189", + "creationTimestamp": "2015-07-07T15:24:29Z", + "labels": { + "app": "hostnames" + } + }, + "spec": { + "ports": [ + { + "name": "default", + "protocol": "TCP", + "port": 80, + "targetPort": 9376, + "nodePort": 0 + } + ], + "selector": { + "app": "hostnames" + }, + "clusterIP": "10.0.1.175", + "type": "ClusterIP", + "sessionAffinity": "None" + }, + "status": { + "loadBalancer": {} + } +} +``` + +* 你想要访问的 Service 端口是否在 `spec.ports[]` 中列出？ +* `targetPort` 对你的 Pod 来说正确吗（许多 Pod 使用与 Service 不同的端口）？ +* 如果你想使用数值型端口，那么它的类型是一个数值（9376）还是字符串 “9376”？ +* 如果你想使用名称型端口，那么你的 Pod 是否暴露了一个同名端口？ +* 端口的 `protocol` 和 Pod 的是否对应？ + + +## Service 有 Endpoints 吗？ + +如果你已经走到了这一步，你已经确认你的 Service 被正确定义，并能通过 DNS 解析。 +现在，让我们检查一下，你运行的 Pod 确实是被 Service 选中的。 + +早些时候，我们已经看到 Pod 是运行状态。我们可以再检查一下： + +```shell +kubectl get pods -l app=hostnames +``` +```none +NAME READY STATUS RESTARTS AGE +hostnames-632524106-bbpiw 1/1 Running 0 1h +hostnames-632524106-ly40y 1/1 Running 0 1h +hostnames-632524106-tlaok 1/1 Running 0 1h +``` + +`-l app=hostnames` 参数是在 Service 上配置的标签选择器。 + +"AGE" 列表明这些 Pod 已经启动一个小时了，这意味着它们运行良好，而未崩溃。 + +"RESTARTS" 列表明 Pod 没有经常崩溃或重启。经常性崩溃可能导致间歇性连接问题。 +如果重启次数过大，通过[调试 pod](/zh/docs/tasks/debug-application-cluster/debug-application/#debugging-pods) +了解相关技术。 + +在 Kubernetes 系统中有一个控制回路，它评估每个 Service 的选择算符，并将结果保存到 Endpoints 对象中。 + +```shell +kubectl get endpoints hostnames +``` +``` +NAME ENDPOINTS +hostnames 10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376 +``` + + +这证实 Endpoints 控制器已经为你的 Service 找到了正确的 Pods。 +如果 `ENDPOINTS` 列的值为 ``，则应检查 Service 的 `spec.selector` 字段， +以及你实际想选择的 Pod 的 `metadata.labels` 的值。 +常见的错误是输入错误或其他错误，例如 Service 想选择 `app=hostnames`，但是 +Deployment 指定的是 `run=hostnames`。在 1.18之前的版本中 `kubectl run` +也可以被用来创建 Deployment。 + + +## Pod 正常工作吗？ + +至此，你知道你的 Service 已存在，并且已匹配到你的Pod。在本实验的开始，你已经检查了 Pod 本身。 +让我们再次检查 Pod 是否确实在工作 - 你可以绕过 Service 机制并直接转到 Pod，如上面的 Endpoint 所示。 + +{{< note >}} +这些命令使用的是 Pod 端口（9376），而不是 Service 端口（80）。 +{{< /note >}} + +在 Pod 中运行： + +```shell +for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do + wget -qO- $ep +done +``` + + +输出应该类似这样： + +``` +hostnames-632524106-bbpiw +hostnames-632524106-ly40y +hostnames-632524106-tlaok +``` + + +你希望 Endpoint 列表中的每个 Pod 都返回自己的主机名。 +如果情况并非如此（或你自己的 Pod 的正确行为是什么），你应调查发生了什么事情。 + + +## kube-proxy 正常工作吗？ + +如果你到达这里，则说明你的 Service 正在运行，拥有 Endpoints，Pod 真正在提供服务。 +此时，整个 Service 代理机制是可疑的。让我们一步一步地确认它没问题。 + +Service 的默认实现（在大多数集群上应用的）是 kube-proxy。 +这是一个在每个节点上运行的程序，负责配置用于提供 Service 抽象的机制之一。 +如果你的集群不使用 kube-proxy，则以下各节将不适用，你将必须检查你正在使用的 Service 的实现方式。 + + +### kube-proxy 正常运行吗？ + +确认 `kube-proxy` 正在节点上运行。 在节点上直接运行，你将会得到类似以下的输出： + +```shell +ps auxw | grep kube-proxy +``` +```none +root 4194 0.4 0.1 101864 17696 ? Sl Jul04 25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2 +``` + + +下一步，确认它并没有出现明显的失败，比如连接主节点失败。要做到这一点，你必须查看日志。 +访问日志的方式取决于你节点的操作系统。 +在某些操作系统上日志是一个文件，如 /var/log/messages kube-proxy.log， +而其他操作系统使用 `journalctl` 访问日志。你应该看到输出类似于： + +```none +I1027 22:14:53.995134 5063 server.go:200] Running in resource-only container "/kube-proxy" +I1027 22:14:53.998163 5063 server.go:247] Using iptables Proxier. +I1027 22:14:53.999055 5063 server.go:255] Tearing down userspace rules. Errors here are acceptable. +I1027 22:14:54.038140 5063 proxier.go:352] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.244.1.3:53] +I1027 22:14:54.038164 5063 proxier.go:352] Setting endpoints for "kube-system/kube-dns:dns" to [10.244.1.3:53] +I1027 22:14:54.038209 5063 proxier.go:352] Setting endpoints for "default/kubernetes:https" to [10.240.0.2:443] +I1027 22:14:54.038238 5063 proxier.go:429] Not syncing iptables until Services and Endpoints have been received from master +I1027 22:14:54.040048 5063 proxier.go:294] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP +I1027 22:14:54.040154 5063 proxier.go:294] Adding new service "kube-system/kube-dns:dns" at 10.0.0.10:53/UDP +I1027 22:14:54.040223 5063 proxier.go:294] Adding new service "kube-system/kube-dns:dns-tcp" at 10.0.0.10:53/TCP +``` + + +如果你看到有关无法连接主节点的错误消息，则应再次检查节点配置和安装步骤。 + +`kube-proxy` 无法正确运行的可能原因之一是找不到所需的 `conntrack` 二进制文件。 +在一些 Linux 系统上，这也是可能发生的，这取决于你如何安装集群， +例如，你是手动开始一步步安装 Kubernetes。如果是这样的话，你需要手动安装 +`conntrack` 包（例如，在 Ubuntu 上使用 `sudo apt install conntrack`），然后重试。 + + +Kube-proxy 可以以若干模式之一运行。在上述日志中，`Using iptables Proxier` +行表示 kube-proxy 在 "iptables" 模式下运行。 +最常见的另一种模式是 "ipvs"。先前的 "userspace" 模式已经被这些所代替。 + + +#### Iptables 模式 + +在 "iptables" 模式中, 你应该可以在节点上看到如下输出: + +```shell +iptables-save | grep hostnames +``` + +```none +-A KUBE-SEP-57KPRZ3JQVENLNBR -s 10.244.3.6/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000 +-A KUBE-SEP-57KPRZ3JQVENLNBR -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.3.6:9376 +-A KUBE-SEP-WNBA2IHDGP2BOBGZ -s 10.244.1.7/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000 +-A KUBE-SEP-WNBA2IHDGP2BOBGZ -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.1.7:9376 +-A KUBE-SEP-X3P2623AGDH6CDF3 -s 10.244.2.3/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000 +-A KUBE-SEP-X3P2623AGDH6CDF3 -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.2.3:9376 +-A KUBE-SERVICES -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames: cluster IP" -m tcp --dport 80 -j KUBE-SVC-NWV5X2332I4OT4T3 +-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-WNBA2IHDGP2BOBGZ +-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-X3P2623AGDH6CDF3 +-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-57KPRZ3JQVENLNBR +``` + + +对于每个 Service 的每个端口，应有 1 条 `KUBE-SERVICES` 规则、一个 `KUBE-SVC-` 链。 +对于每个 Pod 末端，在那个 `KUBE-SVC-` 链中应该有一些规则与之对应，还应该 +有一个 `KUBE-SEP-` 链与之对应，其中包含为数不多的几条规则。 +实际的规则数量可能会根据你实际的配置（包括 NodePort 和 LoadBalancer 服务）有所不同。 + + +#### IPVS 模式 + +在 "ipvs" 模式中, 你应该在节点下看到如下输出： + +```shell +ipvsadm -ln +``` + +```none +Prot LocalAddress:Port Scheduler Flags + -> RemoteAddress:Port Forward Weight ActiveConn InActConn +... +TCP 10.0.1.175:80 rr + -> 10.244.0.5:9376 Masq 1 0 0 + -> 10.244.0.6:9376 Masq 1 0 0 + -> 10.244.0.7:9376 Masq 1 0 0 +... +``` + + +对于每个 Service 的每个端口，还有 NodePort，External IP 和 LoadBalancer 类型服务 +的 IP，kube-proxy 将创建一个虚拟服务器。 +对于每个 Pod 末端，它将创建相应的真实服务器。 +在此示例中，服务主机名（`10.0.1.175:80`）拥有 3 个末端（`10.244.0.5:9376`、 +`10.244.0.6:9376` 和 `10.244.0.7:9376`）。 + + +#### Userspace 模式 + +在极少数情况下，你可能会用到 "userspace" 模式。在你的节点上运行： + +```shell +iptables-save | grep hostnames +``` + +```none +-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577 +-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577 +``` + + +对于 Service （本例中只有一个）的每个端口，应当有 2 条规则： +一条 "KUBE-PORTALS-CONTAINER" 和一条 "KUBE-PORTALS-HOST" 规则。 + +几乎没有人应该再使用 "userspace" 模式，因此你在这里不会花更多的时间。 + + +### kube-proxy 是否在运行? + +假设你确实遇到上述情况之一，请重试从节点上通过 IP 访问你的 Service ： + +```shell +curl 10.0.1.175:80 +``` + +```none +hostnames-632524106-bbpiw +``` + + +如果失败，并且你正在使用用户空间代理，则可以尝试直接访问代理。 +如果你使用的是 iptables 代理，请跳过本节。 + +回顾上面的 `iptables-save` 输出，并提取 `kube-proxy` 为你的 Service 所使用的端口号。 +在上面的例子中，端口号是 “48577”。现在试着连接它： + +```shell +curl localhost:48577 +``` + +```none +hostnames-632524106-tlaok +``` + + +如果这步操作仍然失败，请查看 `kube-proxy` 日志中的特定行，如： + +```none +Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376] +``` + + +如果你没有看到这些，请尝试将 `-V` 标志设置为 4 并重新启动 `kube-proxy`，然后再查看日志。 + + +### 边缘案例: Pod 无法通过 Service IP 连接到它本身 {#a-pod-fails-to-reach-itself-via-the-service-ip} + +这听起来似乎不太可能，但是确实可能发生，并且应该可行。 + +如果网络没有为“发夹模式（Hairpin）”流量生成正确配置， +通常当 `kube-proxy` 以 `iptables` 模式运行，并且 Pod 与桥接网络连接时，就会发生这种情况。 +`kubelet` 提供了 `hairpin-mode` +[标志](/zh/docs/reference/command-line-tools-reference/kubelet/)。 +如果 Service 的末端尝试访问自己的 Service VIP，则该端点可以把流量负载均衡回来到它们自身。 +`hairpin-mode` 标志必须被设置为 `hairpin-veth` 或者 `promiscuous-bridge`。 + + +诊断此类问题的常见步骤如下： + +* 确认 `hairpin-mode` 被设置为 `hairpin-veth` 或 `promiscuous-bridge`。 + 你应该可以看到下面这样。本例中 `hairpin-mode` 被设置为 `promiscuous-bridge`。 + + ```shell + ps auxw | grep kubelet + ``` + ```none + root 3392 1.1 0.8 186804 65208 ? Sl 00:51 11:11 /usr/local/bin/kubelet --enable-debugging-handlers=true --config=/etc/kubernetes/manifests --allow-privileged=True --v=4 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --hairpin-mode=promiscuous-bridge --runtime-cgroups=/docker-daemon --kubelet-cgroups=/kubelet --babysit-daemons=true --max-pods=110 --serialize-image-pulls=false --outofdisk-transition-frequency=0 + ``` + + +* 确认有效的 `hairpin-mode`。要做到这一点，你必须查看 kubelet 日志。 + 访问日志取决于节点的操作系统。在一些操作系统上，它是一个文件，如 /var/log/kubelet.log， + 而其他操作系统则使用 `journalctl` 访问日志。请注意，由于兼容性， + 有效的 `hairpin-mode` 可能不匹配 `--hairpin-mode` 标志。在 kubelet.log + 中检查是否有带有关键字 `hairpin` 的日志行。应该有日志行指示有效的 + `hairpin-mode`，就像下面这样。 + + ```none + I0629 00:51:43.648698 3252 kubelet.go:380] Hairpin mode set to "promiscuous-bridge" + ``` + + +* 如果有效的发夹模式是 `hairpin-veth`, 要保证 `Kubelet` 有操作节点上 `/sys` 的权限。 + 如果一切正常，你将会看到如下输出: + + ```shell + for intf in /sys/devices/virtual/net/cbr0/brif/*; do cat $intf/hairpin_mode; done + ``` + ```none + 1 + 1 + 1 + 1 + ``` + + +* 如果有效的发卡模式是 `promiscuous-bridge`, 要保证 `Kubelet` 有操作节点上 + Linux 网桥的权限。如果 `cbr0` 桥正在被使用且被正确设置，你将会看到如下输出: + + ```shell + ifconfig cbr0 |grep PROMISC + ``` + ```none + UP BROADCAST RUNNING PROMISC MULTICAST MTU:1460 Metric:1 + ``` + + +* 如果以上步骤都不能解决问题，请寻求帮助。 + + +## 寻求帮助 + +如果你走到这一步，那么就真的是奇怪的事情发生了。你的 Service 正在运行，有 Endpoints 存在， +你的 Pods 也确实在提供服务。你的 DNS 正常，`iptables` 规则已经安装，`kube-proxy` 看起来也正常。 +然而 Service 还是没有正常工作。这种情况下，请告诉我们，以便我们可以帮助调查！ + +通过 +[Slack](/zh/docs/tasks/debug-application-cluster/troubleshooting/#slack) 或者 +[Forum](https://discuss.kubernetes.io) 或者 +[GitHub](https://github.com/kubernetes/kubernetes) +联系我们。 + +## {{% heading "whatsnext" %}} + + +访问[故障排查文档](/zh/docs/tasks/debug-application-cluster/troubleshooting/) 获取更多信息。 From c5b195d41dd5620ba6a14c3342d1ac447ab621d0 Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sat, 30 Apr 2022 23:47:36 +0800 Subject: [PATCH 110/119] [zh] add content/zh/docs/tasks/debug/debug-application/_index.md --- .../tasks/debug/debug-application/_index.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 content/zh/docs/tasks/debug/debug-application/_index.md diff --git a/content/zh/docs/tasks/debug/debug-application/_index.md b/content/zh/docs/tasks/debug/debug-application/_index.md new file mode 100644 index 0000000000..aecdc7983d --- /dev/null +++ b/content/zh/docs/tasks/debug/debug-application/_index.md @@ -0,0 +1,17 @@ +--- +title: "应用故障排除" +description: 调试常见的容器应用问题. +weight: 20 +--- + + + + +此文档包含了一些用来解决容器应用问题的资源。它涵盖了 Kubernetes 资源（如 Pod、Service 或 StatefulSets） +的常见问题，理解容器终止消息的建议以及调试正在运行的容器的方法。 \ No newline at end of file From eaf047e51b13a144efd86b757bf9396f581759d6 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 01:55:28 +0800 Subject: [PATCH 111/119] [zh]Update content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md Update content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md Co-authored-by: Qiming Teng --- .../2022-02-17-updated-dockershim-faq.md | 35 ++++++++++++++++--- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md b/content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md index 8400012b61..61f3860602 100644 --- a/content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md +++ b/content/zh/blog/_posts/2022-02-17-updated-dockershim-faq.md @@ -1,12 +1,14 @@ --- layout: blog title: "更新：弃用 Dockershim 的常见问题" +linkTitle: "弃用 Dockershim 的常见问题" date: 2022-02-17 slug: dockershim-faq --- 另外还有一个需要关注的点，那就是当创建镜像时，系统维护或嵌入容器方面的任务将无法工作。 对于前者，可以用 [`crictl`](https://github.com/kubernetes-sigs/cri-tools) 工具作为临时替代方案 -(参阅[从 docker cli 到 crictl 的映射](/zh/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl))。 +(参阅[从 docker cli 到 crictl 的映射](/zh/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl))。 对于后者，可以用新的容器创建选项，例如 [img](https://github.com/genuinetools/img)、 [buildah](https://github.com/containers/buildah)、 @@ -335,10 +337,10 @@ options are available as you migrate things over. 有关如何在 Kubernetes 中使用 containerd 和 CRI-O 的说明， -请参阅 [Kubernetes 相关文档](/docs/setup/production-environment/container-runtimes/) +请参阅 [Kubernetes 相关文档](/docs/setup/production-environment/container-runtimes/)。 +你可以通过专用 [GitHub 问题](https://github.com/kubernetes/kubernetes/issues/106917) +讨论删除 dockershim 的决定。 + +### 是否有任何工具可以帮助我找到正在使用的 dockershim + + +是的！ [Docker Socket 检测器 (DDS)][dds] 是一个 kubectl 插件， +你可以安装它用于检查你的集群。 DDS 可以检测运行中的 Kubernetes +工作负载是否将 Docker 引擎套接字 (`docker.sock`) 作为卷挂载。 +在 DDS 项目的 [README][dds] 中查找更多详细信息和使用方法。 + +[dds]: https://github.com/aws-containers/kubectl-detector-for-docker-socket + From 388ac4d825349fe784fa16e95e596b88ed6b6bfc Mon Sep 17 00:00:00 2001 From: huangminjie Date: Sun, 1 May 2022 00:07:40 +0800 Subject: [PATCH 112/119] [zh] sync content/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md --- .../resource-metrics-pipeline.md | 417 ------------------ .../resource-metrics-pipeline.md | 2 + 2 files changed, 2 insertions(+), 417 deletions(-) delete mode 100644 content/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md diff --git a/content/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md b/content/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md deleted file mode 100644 index 7318664952..0000000000 --- a/content/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md +++ /dev/null @@ -1,417 +0,0 @@ ---- -title: 资源指标管道 -content_type: concept ---- - - - - - - -对于 Kubernetes，_Metrics API_ 提供了一组基本的指标，以支持自动伸缩和类似的用例。 -该 API 提供有关节点和 Pod 的资源使用情况的信息， -包括 CPU 和内存的指标。如果将 Metrics API 部署到集群中， -那么 Kubernetes API 的客户端就可以查询这些信息，并且可以使用 Kubernetes 的访问控制机制来管理权限。 - -[HorizontalPodAutoscaler](/zh/docs/tasks/run-application/horizontal-pod-autoscale/) (HPA) 和 -[VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme) (VPA) -使用 metrics API 中的数据调整工作负载副本和资源，以满足客户需求。 - -你也可以通过 [`kubectl top`](/zh/docs/reference/generated/kubectl/kubectl-commands#top) 命令来查看资源指标。 - -{{< note >}} - -Metrics API 及其启用的指标管道仅提供最少的 CPU 和内存指标，以启用使用 HPA 和/或 VPA 的自动扩展。 -如果你想提供更完整的指标集，你可以通过部署使用 _Custom Metrics API_ 的第二个 -[指标管道](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline) 来作为简单的 Metrics API 的补充。 -{{< /note >}} - - -图 1 说明了资源指标管道的架构。 - -{{< mermaid >}} -flowchart RL -subgraph cluster[Cluster] -direction RL -S[

] -A[Metrics-
Server] -subgraph B[Nodes] -direction TB -D[cAdvisor] --> C[kubelet] -E[Container
runtime] --> D -E1[Container
runtime] --> D -P[pod data] -.- C -end -L[API
server] -W[HPA] -C ---->|Summary
API| A -->|metrics
API| L --> W -end -L ---> K[kubectl
top] -classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; -class W,B,P,K,cluster,D,E,E1 box -classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 -class S spacewhite -classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; -class A,L,C k8s -{{< /mermaid >}} - - -图 1. 资源指标管道 - -图中从右到左的架构组件包括以下内容： - -* [cAdvisor](https://github.com/google/cadvisor): 用于收集、聚合和公开 Kubelet 中包含的容器指标的守护程序。 -* [kubelet](/zh/docs/concepts/overview/components/#kubelet): 用于管理容器资源的节点代理。 - 可以使用 `/metrics/resource` 和 `/stats` kubelet API 端点访问资源指标。 -* [Summary API](#summary-api-source): kubelet 提供的 API，用于发现和检索可通过 `/stats` 端点获得的每个节点的汇总统计信息。 -* [metrics-server](#metrics-server): 集群插件组件，用于收集和聚合从每个 kubelet 中提取的资源指标。 - API 服务器提供 Metrics API 以供 HPA、VPA 和 `kubectl top` 命令使用。 Metrics Server 是 Metrics API 的参考实现。 -* [Metrics API](#metrics-api): Kubernetes API 支持访问用于工作负载自动缩放的 CPU 和内存。 - 要在你的集群中进行这项工作，你需要一个提供 Metrics API 的 API 扩展服务器。 - - - {{< note >}} - cAdvisor 支持从 cgroups 读取指标，它适用于 Linux 上的典型容器运行时。 - 如果你使用基于其他资源隔离机制的容器运行时，例如虚拟化，那么该容器运行时必须支持 - [CRI 容器指标](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md) - 以便 kubelet 可以使用指标。 - {{< /note >}} - - - - -## Metrics API {#the-metrics-api} - -{{< feature-state for_k8s_version="1.8" state="beta" >}} - -metrics-server 实现了 Metrics API。此 API 允许你访问集群中节点和 Pod 的 CPU 和内存使用情况。 -它的主要作用是将资源使用指标提供给 K8s 自动缩放器组件。 - -下面是一个 `minikube` 节点的 Metrics API 请求示例，通过 `jq` 管道处理以便于阅读： - -```shell -kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.' -``` - - -这是使用 `curl` 来执行的相同 API 调用： - -```shell -curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube -``` - - -响应示例： - -```json -{ - "kind": "NodeMetrics", - "apiVersion": "metrics.k8s.io/v1beta1", - "metadata": { - "name": "minikube", - "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube", - "creationTimestamp": "2022-01-27T18:48:43Z" - }, - "timestamp": "2022-01-27T18:48:33Z", - "window": "30s", - "usage": { - "cpu": "487558164n", - "memory": "732212Ki" - } -} -``` - - - -下面是一个 `kube-system` 命名空间中的 `kube-scheduler-minikube` Pod 的 Metrics API 请求示例， -通过 `jq` 管道处理以便于阅读： - -```shell -kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.' -``` - - -这是使用 `curl` 来完成的相同 API 调用： - -```shell -curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube -``` - - -响应示例： - -```json -{ - "kind": "PodMetrics", - "apiVersion": "metrics.k8s.io/v1beta1", - "metadata": { - "name": "kube-scheduler-minikube", - "namespace": "kube-system", - "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube", - "creationTimestamp": "2022-01-27T19:25:00Z" - }, - "timestamp": "2022-01-27T19:24:31Z", - "window": "30s", - "containers": [ - { - "name": "kube-scheduler", - "usage": { - "cpu": "9559630n", - "memory": "22244Ki" - } - } - ] -} -``` - - - -Metrics API 在 [k8s.io/metrics](https://github.com/kubernetes/metrics) 代码库中定义。 -你必须启用 [API 聚合层](/zh/docs/tasks/extend-kubernetes/configure-aggregation-layer/)并为 -`metrics.k8s.io` API 注册一个 [APIService](/zh/docs/reference/kubernetes-api/cluster-resources/api-service-v1/)。 - -要了解有关 Metrics API 的更多信息， -请参阅资源 [Resource Metrics API Design](https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md)、 -[metrics-server 代码库](https://github.com/kubernetes-sigs/metrics-server) 和 -[Resource Metrics API](https://github.com/kubernetes/metrics#resource-metrics-api)。 - - -{{< note >}} -你必须部署提供 Metrics API 服务的 metrics-server 或其他适配器才能访问它。 -{{< /note >}} - - -## 度量资源用量 {#measuring-resource-usage} - -### CPU - -CPU 报告为以 cpu 为单位测量的平均核心使用率。在 Kubernetes 中， -一个 cpu 相当于云提供商的 1 个 vCPU/Core，以及裸机 Intel 处理器上的 1 个超线程。 - -该值是通过对内核提供的累积 CPU 计数器（在 Linux 和 Windows 内核中）取一个速率得出的。 -用于计算 CPU 的时间窗口显示在 Metrics API 的窗口字段下。 - -要了解更多关于 Kubernetes 如何分配和测量 CPU 资源的信息，请参阅 -[CPU 的含义](/zh/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu)。 - - -### 内存 {#memory} - -内存报告为在收集度量标准的那一刻的工作集大小，以字节为单位。 - -在理想情况下，“工作集”是在内存压力下无法释放的正在使用的内存量。 -然而，工作集的计算因主机操作系统而异，并且通常大量使用启发式算法来产生估计。 - -Kubernetes 模型中，容器工作集是由容器运行时计算的与相关容器关联的匿名内存。 -工作集指标通常还包括一些缓存（文件支持）内存，因为主机操作系统不能总是回收页面。 - -要了解有关 Kubernetes 如何分配和测量内存资源的更多信息， -请参阅[内存的含义](/zh/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory)。 - - -## Metrics 服务器 {#metrics-server} - -metrics-server 从 kubelet 中获取资源指标，并通过 Metrics API 在 Kubernetes API 服务器中公开它们，以供 HPA 和 VPA 使用。 -你还可以使用 `kubectl top` 命令查看这些指标。 - -metrics-server 使用 Kubernetes API 来跟踪集群中的节点和 Pod。metrics-server 服务器通过 HTTP 查询每个节点以获取指标。 -metrics-server 还构建了 Pod 元数据的内部视图，并维护 Pod 健康状况的缓存。 -缓存的 Pod 健康信息可通过 metrics-server 提供的扩展 API 获得。 - -例如，对于 HPA 查询，metrics-server 需要确定哪些 Pod 满足 Deployment 中的标签选择器。 - - -metrics-server 调用 [kubelet](/zh/docs/reference/command-line-tools-reference/kubelet/) API -从每个节点收集指标。根据它使用的度量服务器版本： - -* 版本 v0.6.0+ 中，使用指标资源端点 `/metrics/resource` -* 旧版本中使用 Summary API 端点 `/stats/summary` - - - -了解更多 metrics-server，参阅 [metrics-server 代码库](https://github.com/kubernetes-sigs/metrics-server)。 - -你还可以查看以下内容： - -* [metrics-server 设计](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md) -* [metrics-server FAQ](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md) -* [metrics-server known issues](https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md) -* [metrics-server releases](https://github.com/kubernetes-sigs/metrics-server/releases) -* [Horizontal Pod Autoscaling](/zh/docs/tasks/run-application/horizontal-pod-autoscale/) - - - -### Summary API 来源 - -[Kubelet](/zh/docs/reference/command-line-tools-reference/kubelet/) 在节点、卷、Pod 和容器级别收集统计信息， -并在[Summary API](https://github.com/kubernetes/kubernetes/blob/7d309e0104fedb57280b261e5677d919cb2a0e2d/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go) -中提供它们的统计信息供消费者阅读。 - - - -下面是一个 `minikube` 节点的 Summary API 请求示例： - -```shell -kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary" -``` - - -这是使用 `curl` 来执行的相同 API 调用： - -```shell -curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary -``` - -{{< note >}} - -从 metrics-server 0.6.x 开始，Summary API `/stats/summary` 端点被 `/metrics/resource` 端点替换。 -{{< /note >}} diff --git a/content/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline.md b/content/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline.md index 7318664952..ac5f488edd 100644 --- a/content/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline.md +++ b/content/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline.md @@ -1,6 +1,7 @@ --- title: 资源指标管道 content_type: concept +weight: 15 --- From d640b30faae1a6fee02e707614acc71961fef5c4 Mon Sep 17 00:00:00 2001 From: "wei.wang" Date: Sun, 1 May 2022 18:16:26 +0800 Subject: [PATCH 113/119] [zh]Update content/zh/examples/pods/probe/exec-liveness.yaml --- content/zh/examples/pods/probe/exec-liveness.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/zh/examples/pods/probe/exec-liveness.yaml b/content/zh/examples/pods/probe/exec-liveness.yaml index 07bf75f85c..6a9c9b3213 100644 --- a/content/zh/examples/pods/probe/exec-liveness.yaml +++ b/content/zh/examples/pods/probe/exec-liveness.yaml @@ -11,7 +11,7 @@ spec: args: - /bin/sh - -c - - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 + - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600 livenessProbe: exec: command: From cdea49863969002e3cc9569f6121f6c9561365ac Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Sun, 1 May 2022 18:21:30 +0800 Subject: [PATCH 114/119] [zh] change debug link Signed-off-by: xin.li --- .../zh/docs/tasks/run-application/horizontal-pod-autoscale.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/tasks/run-application/horizontal-pod-autoscale.md b/content/zh/docs/tasks/run-application/horizontal-pod-autoscale.md index 8cc8dc42f4..4ba6b9857a 100644 --- a/content/zh/docs/tasks/run-application/horizontal-pod-autoscale.md +++ b/content/zh/docs/tasks/run-application/horizontal-pod-autoscale.md @@ -148,12 +148,12 @@ The common use for HorizontalPodAutoscaler is to configure it to fetch metrics f (`metrics.k8s.io`, `custom.metrics.k8s.io`, or `external.metrics.k8s.io`). The `metrics.k8s.io` API is usually provided by an add-on named Metrics Server, which needs to be launched separately. For more information about resource metrics, see -[Metrics Server](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server). +[Metrics Server](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server). --> HorizontalPodAutoscaler 的常见用途是将其配置为从{{< glossary_tooltip text="聚合 API" term_id="aggregation-layer" >}} （`metrics.k8s.io`、`custom.metrics.k8s.io` 或 `external.metrics.k8s.io`）获取指标。 `metrics.k8s.io` API 通常由名为 Metrics Server 的插件提供，需要单独启动。有关资源指标的更多信息， -请参阅 [Metrics Server](/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server)。 +请参阅 [Metrics Server](/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server)。 请注意，作为新的内置命令，`kubectl debug` 优先于任何名为 “debug” 的 kubectl 插件。你必须重命名受影响的插件。 - -`kubectl alpha debug` 现在不推荐使用，并将在后续版本中删除。更新你的脚本以使用 `kubectl debug`。 有关更多信息 `kubectl debug`，请参阅[调试正在运行的 Pod]((https://kubernetes.io/zh/docs/tasks/debug-application-cluster/debug-running-pod/)。 + +`kubectl alpha debug` 现在不推荐使用，并将在后续版本中删除。更新你的脚本以使用 `kubectl debug`。 有关更多信息 `kubectl debug`，请参阅[调试正在运行的 Pod]((https://kubernetes.io/zh/docs/tasks/debug/debug-application/debug-running-pod/)。 ### 测试版：API 优先级和公平性 {#beta-api-priority-and-fairness) From 6b326ece291c4f8a558094c3924be43fa62859cc Mon Sep 17 00:00:00 2001 From: "xin.li" Date: Sun, 1 May 2022 19:42:32 +0800 Subject: [PATCH 116/119] [zh] Update statefule-application/cassandra.md Signed-off-by: xin.li --- content/zh/docs/tutorials/stateful-application/cassandra.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/zh/docs/tutorials/stateful-application/cassandra.md b/content/zh/docs/tutorials/stateful-application/cassandra.md index ea55fd4a24..fcf3273d04 100644 --- a/content/zh/docs/tutorials/stateful-application/cassandra.md +++ b/content/zh/docs/tutorials/stateful-application/cassandra.md @@ -156,11 +156,11 @@ cassandra ClusterIP None 9042/TCP 45s 如果没有看到名为 `cassandra` 的服务，则表示创建失败。 -请阅读[调试服务](/zh/docs/tasks/debug-application-cluster/debug-service/)，以解决常见问题。 +请阅读[调试服务](/zh/docs/tasks/debug/debug-application/debug-service/)，以解决常见问题。 -* 了解如何[使用临时调试容器来进行调试](/zh/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container) +* 了解如何[使用临时调试容器来进行调试](/zh/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container) diff --git a/content/zh/docs/tasks/debug-application-cluster/debug-application.md b/content/zh/docs/tasks/debug-application-cluster/debug-application.md index 99678b817a..6fa979bc16 100644 --- a/content/zh/docs/tasks/debug-application-cluster/debug-application.md +++ b/content/zh/docs/tasks/debug-application-cluster/debug-application.md @@ -120,12 +120,12 @@ Again, the information from `kubectl describe ...` should be informative. The m #### My pod is crashing or otherwise unhealthy Once your pod has been scheduled, the methods described in [Debug Running Pods]( -/docs/tasks/debug-application-cluster/debug-running-pod/) are available for debugging. +/docs/tasks/debug/debug-application/debug-running-pod/) are available for debugging. --> #### Pod 处于 Crashing 或别的不健康状态 一旦 Pod 被调度，就可以采用 -[调试运行中的 Pod](/zh/docs/tasks/debug-application-cluster/debug-running-pod/) +[调试运行中的 Pod](/zh/docs/tasks/debug/debug-application/debug-running-pod/) 中的方法来进一步调试。 ### 我的 Pod 一直 Crashing 或者其他不健康状态 一旦 Pod 已经被调度，就可以依据 -[调试运行中的 Pod](/zh/docs/tasks/debug-application-cluster/debug-running-pod/) +[调试运行中的 Pod](/zh/docs/tasks/debug/debug-application/debug-running-pod/) 展开进一步的调试工作。 - *安装节点健康检查*：对于重要的工作负载，你会希望确保节点以及在节点上 运行的 Pod 处于健康状态。通过使用 - [Node Problem Detector](/zh/docs/tasks/debug-application-cluster/monitor-node-health/)， + [Node Problem Detector](/zh/docs/tasks/debug/debug-cluster/monitor-node-health/)， 你可以确保你的节点是健康的。