whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

resync setup section [3]

Browse Source
This commit is contained in:
howieyuen
2021-04-29 16:06:52 +08:00
parent bb0ddd647e
commit fa884fbe50
1 changed files with 164 additions and 199 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/zh/docs/setup/best-practices/cluster-large.md
+164 -199
View File
@@ -1,173 +1,136 @@
---
title: 创建大型集群
title: 大规模集群的注意事项
weight: 20
---
<!--
## Support
<!--
reviewers:
- davidopp
- lavalamp
title: Considerations for large clusters
weight: 20
-->
## 支持
<!--
At {{< param "version" >}}, Kubernetes supports clusters with up to 5000 nodes. More specifically, we support configurations that meet *all* of the following criteria:
A cluster is a set of {{< glossary_tooltip text="nodes" term_id="node" >}} (physical
or virtual machines) running Kubernetes agents, managed by the
{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
Kubernetes {{< param "version" >}} supports clusters with up to 5000 nodes. More specifically,
Kubernetes is designed to accommodate configurations that meet *all* of the following criteria:
-->
在 {{< param "version" >}} 版本中， Kubernetes 支持的最大节点数为 5000。更具体地说，我们支持满足以下*所有*条件的配置：
集群是运行 Kubernetes 代理的、
由{{< glossary_tooltip text="控制平面" term_id="control-plane" >}}管理的一组
{{< glossary_tooltip text="节点" term_id="node" >}}（物理机或虚拟机）。
Kubernetes {{< param "version" >}} 支持的最大节点数为 5000。
更具体地说，Kubernetes旨在适应满足以下*所有*标准的配置：
<!--
* No more than 100 pods per node
* No more than 5000 nodes
* No more than 150000 total pods
* No more than 300000 total containers
* No more than 100 pods per node
-->
* 每个节点的 Pod 数量不超过 100
* 节点数不超过 5000
* Pod 总数不超过 150000
* 容器总数不超过 300000
* 每个节点的 pod 数量不超过 100
<br>
{{< toc >}}
<!--
## Setup
<!--
You can scale your cluster by adding or removing nodes. The way you do this depends
on how your cluster is deployed.
-->
## 设定
你可以通过添加或删除节点来扩展集群。集群扩缩的方式取决于集群的部署方式。
<!--
A cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by a "master" (the cluster-level control plane).
-->
集群是一组运行着 Kubernetes 代理的节点（物理机或者虚拟机），这些节点由主控节点（集群级控制面）控制。
<!--
## Cloud provider resource quotas {#quota-issues}
<!--
Normally the number of nodes in a cluster is controlled by the value `NUM_NODES` in the platform-specific `config-default.sh` file (for example, see [GCE's `config-default.sh`](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/gce/config-default.sh)).
-->
通常，集群中的节点数由特定于云平台的配置文件 `config-default.sh`
（可以参考 [GCE 平台的 `config-default.sh`](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/gce/config-default.sh)）
中的 `NUM_NODES` 参数控制。
<!--
Simply changing that value to something very large, however, may cause the setup script to fail for many cloud providers. A GCE deployment, for example, will run in to quota issues and fail to bring the cluster up.
-->
但是，在许多云供应商的平台上，仅将该值更改为非常大的值，可能会导致安装脚本运行失败。例如，在 GCE，由于配额问题，集群会启动失败。
<!--
When setting up a large Kubernetes cluster, the following issues must be considered.
-->
因此，在创建大型 Kubernetes 集群时，必须考虑以下问题。
<!--
### Quota Issues
-->
### 配额问题
<!--
To avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:
-->
为了避免遇到云供应商配额问题，在创建具有大规模节点的集群时，请考虑：
<!--
* Increase the quota for things like CPU, IPs, etc.
* In [GCE, for example,](https://cloud.google.com/compute/docs/resource-quotas) you'll want to increase the quota for:
To avoid running into cloud provider quota issues, when creating a cluster with many nodes,
consider:
* Request a quota increase for cloud resources such as:
* Computer instances
* CPUs
* VM instances
* Total persistent disk reserved
* Storage volumes
* In-use IP addresses
* Firewall Rules
* Forwarding rules
* Routes
* Target pools
* Gating the setup script so that it brings up new node VMs in smaller batches with waits in between, because some cloud providers rate limit the creation of VMs.
* Packet filtering rule sets
* Number of load balancers
* Network subnets
* Log streams
* Gate the cluster scaling actions to brings up new nodes in batches, with a pause
between batches, because some cloud providers rate limit the creation of new instances.
-->
* 增加诸如 CPU，IP 等资源的配额。
* 例如，在 [GCE](https://cloud.google.com/compute/docs/resource-quotas)，您需要增加以下资源的配额：
## 云供应商资源配额 {#quota-issues}
为避免遇到云供应商配额问题，在创建具有大规模节点的集群时，请考虑以下事项：
* 请求增加云资源的配额，例如：
* 计算实例
* CPUs
* VM 实例
* 永久磁盘总量
* 存储卷
* 使用中的 IP 地址
* 防火墙规则
* 转发规则
* 路由
* 目标池
* 由于某些云供应商会对虚拟机的创建进行流控，因此需要对设置脚本进行更改，使其以较小的批次启动新的节点，并且之间有等待时间。
* 数据包过滤规则集
* 负载均衡数量
* 网络子网
* 日志流
* 由于某些云供应商限制了创建新实例的速度，因此通过分批启动新节点来控制集群扩展操作，并在各批之间有一个暂停。
<!--
### Etcd storage
<!--
## Control plane components
For a large cluster, you need a control plane with sufficient compute and other
resources.
Typically you would run one or two control plane instances per failure zone,
scaling those instances vertically first and then scaling horizontally after reaching
the point of falling returns to (vertical) scale.
-->
### Etcd 存储
## 控制面组件
<!--
To improve performance of large clusters, we store events in a separate dedicated etcd instance.
对于大型集群，你需要一个具有足够计算能力和其他资源的控制平面。
通常，你将在每个故障区域运行一个或两个控制平面实例，
先垂直缩放这些实例，然后在到达下降点（垂直）后再水平缩放。
<!--
You should run at least one instance per failure zone to provide fault-tolerance. Kubernetes
nodes do not automatically steer traffic towards control-plane endpoints that are in the
same failure zone; however, your cloud provider might have its own mechanisms to do this.
For example, using a managed load balancer, you configure the load balancer to send traffic
that originates from the kubelet and Pods in failure zone _A_, and direct that traffic only
to the control plane hosts that are also in zone _A_. If a single control-plane host or
endpoint failure zone _A_ goes offline, that means that all the control-plane traffic for
nodes in zone _A_ is now being sent between zones. Running multiple control plane hosts in
each zone makes that outcome less likely.
-->
为了提高大规模集群的性能，我们将事件存储在专用的 etcd 实例中。
你应该在每个故障区域至少应运行一个实例，以提供容错能力。
Kubernetes 节点不会自动将流量引向相同故障区域中的控制平面端点。
但是，你的云供应商可能有自己的机制来执行此操作。
例如，使用托管的负载均衡器时，你可以配置负载均衡器发送源自故障区域 _A_ 中的 kubelet 和 Pod 的流量，
并将该流量仅定向到也位于区域 _A_ 中的控制平面主机。
如果单个控制平面主机或端点故障区域 _A_ 脱机，则意味着区域 _A_ 中的节点的所有控制平面流量现在都在区域之间发送。
在每个区域中运行多个控制平面主机能降低出现这种结果的可能性。
<!--
When creating a cluster, existing salt scripts:
### etcd storage
-->
### etcd 存储
<!--
To improve performance of large clusters, you can store Event objects in a separate
dedicated etcd instance.
-->
为了提高大规模集群的性能，你可以将事件对象存储在单独的专用 etcd 实例中。
<!--
When creating a cluster, you can (using custom tooling):
* start and configure additional etcd instance
* configure api-server to use it for storing events
* configure the {{< glossary_tooltip term_id="kube-apiserver" text="API server" >}} to use it for storing events
-->
在创建集群时，现有 salt 脚本可以：
在创建集群时，你可以（使用自定义工具）：
* 启动并配置其它 etcd 实例
* 配置 API 服务器以使用 etcd 存储事件
<!--
### Size of master and master components
-->
### 主控节点大小和主控组件
<!--
On GCE/Google Kubernetes Engine, and AWS, `kube-up` automatically configures the proper VM size for your master depending on the number of nodes
in your cluster. On other providers, you will need to configure it manually. For reference, the sizes we use on GCE are
-->
在 GCE/Google Kubernetes Engine 和 AWS 上，`kube-up` 会根据节点数量自动为您集群中的 master 节点配置适当的虚拟机大小。在其它云供应商的平台上，您将需要手动配置它。作为参考，我们在 GCE 上使用的规格为：
<!--
* 1-5 nodes: n1-standard-1
* 6-10 nodes: n1-standard-2
* 11-100 nodes: n1-standard-4
* 101-250 nodes: n1-standard-8
* 251-500 nodes: n1-standard-16
* more than 500 nodes: n1-standard-32
-->
* 1-5 个节点：n1-standard-1
* 6-10 个节点：n1-standard-2
* 11-100 个节点：n1-standard-4
* 101-250 个节点：n1-standard-8
* 251-500 个节点：n1-standard-16
* 超过 500 节点：n1-standard-32
<!--
And the sizes we use on AWS are
* 1-5 nodes: m3.medium
* 6-10 nodes: m3.large
* 11-100 nodes: m3.xlarge
* 101-250 nodes: m3.2xlarge
* 251-500 nodes: c4.4xlarge
* more than 500 nodes: c4.8xlarge
-->
在 AWS 上使用的规格为
* 1-5 个节点：m3.medium
* 6-10 个节点：m3.large
* 11-100 个节点：m3.xlarge
* 101-250 个节点：m3.2xlarge
* 251-500 个节点：c4.4xlarge
* 超过 500 节点：c4.8xlarge
{{< note >}}
<!--
On Google Kubernetes Engine, the size of the master node adjusts automatically based on the size of your cluster. For more information, see [this blog post](https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster-Management-Fees-on-Google-Kubernetes-Engine.html).
-->
在 Google Kubernetes Engine 上，主控节点的大小会根据集群的大小自动调整。更多有关信息，请参阅 [此博客文章](https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster-Management-Fees-on-Google-Kubernetes-Engine.html)。
<!--
On AWS, master node sizes are currently set at cluster startup time and do not change, even if you later scale your cluster up or down by manually removing or adding nodes or using a cluster autoscaler.
-->
在 AWS 上，主控节点的规格是在集群启动时设置的，并且，即使以后通过手动删除或添加节点的方式使集群缩容或扩容，主控节点的大小也不会更改。
{{< /note >}}
* 启动并配置额外的 etcd 实例
* 配置 {{< glossary_tooltip term_id="kube-apiserver" text="API 服务器" >}}，将它用于存储事件
<!--
### Addon Resources
@@ -175,90 +138,92 @@ On AWS, master node sizes are currently set at cluster startup time and do not c
### 插件资源 {#addon-resources}
<!--
To prevent memory leaks or other resource issues in [cluster addons](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons) from consuming all the resources available on a node, Kubernetes sets resource limits on addon containers to limit the CPU and Memory resources they can consume (See PR [#10653](http://pr.k8s.io/10653/files) and [#10778](http://pr.k8s.io/10778/files)).
-->
为了防止内存泄漏或 [集群插件](https://releases.k8s.io/{{<param "githubbranch" >}}/cluster/addons)
中的其它资源问题导致节点上所有可用资源被消耗，Kubernetes 限制了插件容器可以消耗的 CPU 和内存资源
（请参阅 PR [#10653](http://pr.k8s.io/10653/files) 和 [#10778](http://pr.k8s.io/10778/files)）。
Kubernetes [resource limits](/docs/concepts/configuration/manage-resources-containers/)
help to minimize the impact of memory leaks and other ways that pods and containers can
impact on other components. These resource limits apply to
{{< glossary_tooltip text="addon" term_id="addons" >}} resources just as they apply to application workloads.
例如：
For example, you can set CPU and memory limits for a logging component:
-->
Kubernetes [资源限制](/zh/docs/concepts/configuration/manage-resources-containers/)
有助于最大程度地减少内存泄漏的影响以及 Pod 和容器可能对其他组件的其他方式的影响。
这些资源限制适用于{{< glossary_tooltip text="插件" term_id="addons" >}}资源，
就像它们适用于应用程序工作负载一样。
例如，你可以对日志组件设置 CPU 和内存限制
```yaml
...
containers:
- name: fluentd-cloud-logging
image: k8s.gcr.io/fluentd-gcp:1.16
image: fluent/fluentd-kubernetes-daemonset:v1
resources:
limits:
cpu: 100m
memory: 200Mi
```
<!--
Except for Heapster, these limits are static and are based on data we collected from addons running on 4-node clusters (see [#10335](http://issue.k8s.io/10335#issuecomment-117861225)). The addons consume a lot more resources when running on large deployment clusters (see [#5880](http://issue.k8s.io/5880#issuecomment-113984085)). So, if a large cluster is deployed without adjusting these values, the addons may continuously get killed because they keep hitting the limits.
<!--
Addons' default limits are typically based on data collected from experience running
each addon on small or medium Kubernetes clusters. When running on large
clusters, addons often consume more of some resources than their default limits.
If a large cluster is deployed without adjusting these values, the addon(s)
may continuously get killed because they keep hitting the memory limit.
Alternatively, the addon may run but with poor performance due to CPU time
slice restrictions.
-->
除了 Heapster 之外，这些限制都是静态的，并且限制是基于 4 节点集群上运行的插件数据得出的（请参阅 [#10335](http://issue.k8s.io/10335#issuecomment-117861225)）。在大规模集群上运行时，插件会消耗大量资源（请参阅 [#5880](http://issue.k8s.io/5880#issuecomment-113984085)）。因此，如果在不调整这些值的情况下部署了大规模集群，插件容器可能会由于达到限制而不断被杀死。
插件的默认限制通常基于从中小规模 Kubernetes 集群上运行每个插件的经验收集的数据。
插件在大规模集群上运行时，某些资源消耗常常比其默认限制更多。
如果在不调整这些值的情况下部署了大规模集群，则插件可能会不断被杀死，因为它们不断达到内存限制。
或者，插件可能会运行，但由于 CPU 时间片的限制而导致性能不佳。
<!--
To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:
<!--
To avoid running into cluster addon resource issues, when creating a cluster with
many nodes, consider the following:
* Some addons scale vertically - there is one replica of the addon for the cluster
or serving a whole failure zone. For these addons, increase requests and limits
as you scale out your cluster.
* Many addons scale horizontally - you add capacity by running more pods - but with
a very large cluster you may also need to raise CPU or memory limits slightly.
The VerticalPodAutoscaler can run in _recommender_ mode to provide suggested
figures for requests and limits.
* Some addons run as one copy per node, controlled by a {{< glossary_tooltip text="DaemonSet"
term_id="daemonset" >}}: for example, a node-level log aggregator. Similar to
the case with horizontally-scaled addons, you may also need to raise CPU or memory
limits slightly.
-->
为避免遇到集群插件资源问题，在创建大规模集群时，请考虑以下事项：
<!--
* Scale memory and CPU limits for each of the following addons, if used, as you scale up the size of cluster (there is one replica of each handling the entire cluster so memory and CPU usage tends to grow proportionally with size/load on cluster):
* [InfluxDB and Grafana](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml)
* [kubedns, dnsmasq, and sidecar](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/kube-dns/kube-dns.yaml.in)
* [Kibana](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/kibana-deployment.yaml)
* Scale number of replicas for the following addons, if used, along with the size of cluster (there are multiple replicas of each so increasing replicas should help handle increased load, but, since load per replica also increases slightly, also consider increasing CPU/memory limits):
* [elasticsearch](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/es-statefulset.yaml)
* Increase memory and CPU limits slightly for each of the following addons, if used, along with the size of cluster (there is one replica per node but CPU/memory usage increases slightly along with cluster load/size as well):
* [FluentD with ElasticSearch Plugin](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml)
* [FluentD with GCP Plugin](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml)
-->
* 根据集群的规模，如果使用了以下插件，提高其内存和 CPU 上限（每个插件都有一个副本处理整个群集，因此内存和 CPU 使用率往往与集群的规模/负载成比例增长） ：
* [InfluxDB 和 Grafana](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml)
* [kubedns、dnsmasq 和 sidecar](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/kube-dns/kube-dns.yaml.in)
* [Kibana](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/kibana-deployment.yaml)
* 根据集群的规模，如果使用了以下插件，调整其副本数量（每个插件都有多个副本，增加副本数量有助于处理增加的负载，但是，由于每个副本的负载也略有增加，因此也请考虑增加 CPU/内存限制）：
* [elasticsearch](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/es-statefulset.yaml)
* 根据集群的规模，如果使用了以下插件，限制其内存和 CPU 上限（这些插件在每个节点上都有一个副本，但是 CPU/内存使用量也会随集群负载/规模而略有增加）：
* [FluentD 和 ElasticSearch 插件](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml)
* [FluentD 和 GCP 插件](https://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml)
* 部分垂直扩展插件 —— 总有一个插件副本服务于整个集群或服务于整个故障区域。
对于这些附加组件，请在扩大集群时加大资源请求和资源限制。
* 许多水平扩展插件 —— 你可以通过运行更多的 Pod 来增加容量——但是在大规模集群下，
可能还需要稍微提高 CPU 或内存限制。
VerticalPodAutoscaler 可以在 _recommender_ 模式下运行，
以提供有关请求和限制的建议数字。
* 一些插件在每个节点上运行一个副本，并由 DaemonSet 控制：
例如，节点级日志聚合器。与水平扩展插件的情况类似，
你可能还需要稍微提高 CPU 或内存限制。
<!--
Heapster's resource limits are set dynamically based on the initial size of your cluster (see [#16185](http://issue.k8s.io/16185)
and [#22940](http://issue.k8s.io/22940)). If you find that Heapster is running
out of resources, you should adjust the formulas that compute heapster memory request (see those PRs for details).
-->
Heapster 的资源限制与您集群的初始大小有关（请参阅 [#16185](https://issue.k8s.io/16185)
和 [#22940](http://issue.k8s.io/22940)）。如果您发现 Heapster 资源不足，您应该调整堆内存请求的计算公式（有关详细信息，请参阅相关 PR）。
<!--
## {{% heading "whatsnext" %}}
<!--
For directions on how to detect if addon containers are hitting resource limits, see the [Troubleshooting section of Compute Resources](/docs/concepts/configuration/manage-compute-resources-container/#troubleshooting).
-->
关于如何检测插件容器是否达到资源限制，参见
[计算资源的故障排除](/zh/docs/concepts/configuration/manage-resources-containers/#troubleshooting) 部分。
`VerticalPodAutoscaler` is a custom resource that you can deploy into your cluster
to help you manage resource requests and limits for pods.
Visit [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme)
to learn more about `VerticalPodAutoscaler` and how you can use it to scale cluster
components, including cluster-critical addons.
<!--
In the [future](http://issue.k8s.io/13048), we anticipate to set all cluster addon resource limits based on cluster size, and to dynamically adjust them if you grow or shrink your cluster.
We welcome PRs that implement those features.
The [cluster autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#readme)
integrates with a number of cloud providers to help you run the right number of
nodes for the level of resource demand in your cluster.
-->
[未来](https://issue.k8s.io/13048)，我们期望根据集群规模大小来设置所有群集附加资源限制，并在集群扩缩容时动态调整它们。
我们欢迎您来实现这些功能。
## {{% heading "whatsnext" %}}
<!--
### Allowing minor node failure at startup
-->
### 允许启动时次要节点失败
`VerticalPodAutoscaler` 是一种自定义资源，你可以将其部署到集群中，帮助你管理资源请求和 Pod 的限制。
访问 [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme)
以了解有关 `VerticalPodAutoscaler` 的更多信息，
以及如何使用它来扩展集群组件（包括对集群至关重要的插件）的信息。
<!--
For various reasons (see [#18969](https://github.com/kubernetes/kubernetes/issues/18969) for more details) running
`kube-up.sh` with a very large `NUM_NODES` may fail due to a very small number of nodes not coming up properly.
Currently you have two choices: restart the cluster (`kube-down.sh` and then `kube-up.sh` again), or before
running `kube-up.sh` set the environment variable `ALLOWED_NOTREADY_NODES` to whatever value you feel comfortable
with. This will allow `kube-up.sh` to succeed with fewer than `NUM_NODES` coming up. Depending on the
reason for the failure, those additional nodes may join later or the cluster may remain at a size of
`NUM_NODES - ALLOWED_NOTREADY_NODES`.
-->
出于各种原因（更多详细信息，请参见 [#18969](https://github.com/kubernetes/kubernetes/issues/18969)），
在 `kube-up.sh` 中设置很大的 `NUM_NODES` 时，可能会由于少数节点无法正常启动而失败。
此时，您有两个选择：重新启动集群（运行 `kube-down.sh`，然后再运行 `kube-up.sh`），或者在运行 `kube-up.sh` 之前将环境变量 `ALLOWED_NOTREADY_NODES` 设置为您认为合适的任何值。采取后者时，即使运行成功的节点数量少于 `NUM_NODES`，`kube-up.sh` 仍可以运行成功。根据失败的原因，这些节点可能会稍后加入集群，又或者群集的大小保持在 `NUM_NODES-ALLOWED_NOTREADY_NODES`。
[集群自动扩缩器](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#readme)
与许多云供应商集成在一起，帮助你在你的集群中，按照资源需求级别运行正确数量的节点。