Fix links to etcd official documentation

This commit is contained in:
Qiming Teng
2021-02-16 11:29:50 +08:00
parent 092cbba2c3
commit 9cac41b0a8
@@ -27,14 +27,14 @@ content_type: task
* Ensure that no resource starvation occurs. * Ensure that no resource starvation occurs.
Performance and stability of the cluster is sensitive to network and disk Performance and stability of the cluster is sensitive to network and disk
IO. Any resource starvation can lead to heartbeat timeout, causing instability I/O. Any resource starvation can lead to heartbeat timeout, causing instability
of the cluster. An unstable etcd indicates that no leader is elected. Under of the cluster. An unstable etcd indicates that no leader is elected. Under
such circumstances, a cluster cannot make any changes to its current state, such circumstances, a cluster cannot make any changes to its current state,
which implies no new pods can be scheduled. which implies no new pods can be scheduled.
* Keeping stable etcd clusters is critical to the stability of Kubernetes * Keeping etcd clusters stable is critical to the stability of Kubernetes
clusters. Therefore, run etcd clusters on dedicated machines or isolated clusters. Therefore, run etcd clusters on dedicated machines or isolated
environments for [guaranteed resource requirements](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations). environments for [guaranteed resource requirements](https://etcd.io/docs/current/op-guide/hardware/).
* The minimum recommended version of etcd to run in production is `3.2.10+`. * The minimum recommended version of etcd to run in production is `3.2.10+`.
@@ -43,7 +43,7 @@ content_type: task
Operating etcd with limited resources is suitable only for testing purposes. Operating etcd with limited resources is suitable only for testing purposes.
For deploying in production, advanced hardware configuration is required. For deploying in production, advanced hardware configuration is required.
Before deploying etcd in production, see Before deploying etcd in production, see
[resource requirement reference documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#example-hardware-configurations). [resource requirement reference](https://etcd.io/docs/current/op-guide/hardware/#example-hardware-configurations).
## Starting etcd clusters ## Starting etcd clusters
@@ -60,7 +60,7 @@ Use a single-node etcd cluster only for testing purpose.
--advertise-client-urls=http://$PRIVATE_IP:2379 --advertise-client-urls=http://$PRIVATE_IP:2379
``` ```
2. Start Kubernetes API server with the flag 2. Start the Kubernetes API server with the flag
`--etcd-servers=$PRIVATE_IP:2379`. `--etcd-servers=$PRIVATE_IP:2379`.
Make sure `PRIVATE_IP` is set to your etcd client IP. Make sure `PRIVATE_IP` is set to your etcd client IP.
@@ -69,12 +69,12 @@ Use a single-node etcd cluster only for testing purpose.
For durability and high availability, run etcd as a multi-node cluster in For durability and high availability, run etcd as a multi-node cluster in
production and back it up periodically. A five-member cluster is recommended production and back it up periodically. A five-member cluster is recommended
in production. For more information, see [FAQ in production. For more information, see
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-is-failure-tolerance). [FAQ documentation](https://etcd.io/docs/current/faq/#what-is-failure-tolerance).
Configure an etcd cluster either by static member information or by dynamic Configure an etcd cluster either by static member information or by dynamic
discovery. For more information on clustering, see [etcd Clustering discovery. For more information on clustering, see
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md). [etcd clustering documentation](https://etcd.io/docs/current/op-guide/clustering/).
For an example, consider a five-member etcd cluster running with the following For an example, consider a five-member etcd cluster running with the following
client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`, client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`,
@@ -86,10 +86,10 @@ client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`,
etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379
``` ```
2. Start Kubernetes API servers with the flag 2. Start the Kubernetes API servers with the flag
`--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379`. `--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379`.
Replace `IP<n>` with your client IP addresses. Make sure the `IP<n>` variables are set to your client IP addresses.
### Multi-node etcd cluster with load balancer ### Multi-node etcd cluster with load balancer
@@ -121,16 +121,16 @@ authentication.
To configure etcd with secure peer communication, specify flags To configure etcd with secure peer communication, specify flags
`--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as
URL schema. the URL schema.
Similarly, to configure etcd with secure client communication, specify flags Similarly, to configure etcd with secure client communication, specify flags
`--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as
URL schema. the URL schema.
### Limiting access of etcd clusters ### Limiting access of etcd clusters
After configuring secure communication, restrict the access of etcd cluster to After configuring secure communication, restrict the access of etcd cluster to
only the Kubernetes API server. Use TLS authentication to do so. only the Kubernetes API servers. Use TLS authentication to do so.
For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are
trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth`
@@ -140,7 +140,7 @@ or the CA passed in by `--trusted-ca-file` flag. Specifying flags
access to clients with the certificate `k8sclient.cert`. access to clients with the certificate `k8sclient.cert`.
Once etcd is configured correctly, only clients with valid certificates can Once etcd is configured correctly, only clients with valid certificates can
access it. To give Kubernetes API server the access, configure it with the access it. To give Kubernetes API servers the access, configure them with the
flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and
`--etcd-cafile=ca.cert`. `--etcd-cafile=ca.cert`.
@@ -160,11 +160,11 @@ member.
Though etcd keeps unique member IDs internally, it is recommended to use a Though etcd keeps unique member IDs internally, it is recommended to use a
unique name for each member to avoid human errors. For example, consider a unique name for each member to avoid human errors. For example, consider a
three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1, three-member etcd cluster. Let the URLs be, `member1=http://10.0.0.1`,
member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails, `member2=http://10.0.0.2`, and `member3=http://10.0.0.3`. When `member1` fails,
replace it with member4=http://10.0.0.4. replace it with `member4=http://10.0.0.4`.
1. Get the member ID of the failed member1: 1. Get the member ID of the failed `member1`:
```shell ```shell
etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list
@@ -213,21 +213,22 @@ replace it with member4=http://10.0.0.4.
5. Do either of the following: 5. Do either of the following:
1. Update its `--etcd-servers` flag to make Kubernetes aware of the 1. Update the `--etcd-servers` flag for the Kubernetes API servers to make
configuration changes, then restart the Kubernetes API server. Kubernetes aware of the configuration changes, then restart the
Kubernetes API servers.
2. Update the load balancer configuration if a load balancer is used in the 2. Update the load balancer configuration if a load balancer is used in the
deployment. deployment.
For more information on cluster reconfiguration, see [etcd Reconfiguration For more information on cluster reconfiguration, see
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member). [etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member).
## Backing up an etcd cluster ## Backing up an etcd cluster
All Kubernetes objects are stored on etcd. Periodically backing up the etcd All Kubernetes objects are stored on etcd. Periodically backing up the etcd
cluster data is important to recover Kubernetes clusters under disaster cluster data is important to recover Kubernetes clusters under disaster
scenarios, such as losing all master nodes. The snapshot file contains all the scenarios, such as losing all control plane nodes. The snapshot file contains
Kubernetes states and critical information. In order to keep the sensitive all the Kubernetes states and critical information. In order to keep the
Kubernetes data safe, encrypt the snapshot files. sensitive Kubernetes data safe, encrypt the snapshot files.
Backing up an etcd cluster can be accomplished in two ways: etcd built-in Backing up an etcd cluster can be accomplished in two ways: etcd built-in
snapshot and volume snapshot. snapshot and volume snapshot.
@@ -236,10 +237,10 @@ snapshot and volume snapshot.
etcd supports built-in snapshot. A snapshot may either be taken from a live etcd supports built-in snapshot. A snapshot may either be taken from a live
member with the `etcdctl snapshot save` command or by copying the member with the `etcdctl snapshot save` command or by copying the
`member/snap/db` file from an etcd [data `member/snap/db` file from an etcd
directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir) [data directory](https://etcd.io/docs/current/op-guide/configuration/#--data-dir)
that is not currently used by an etcd process. Taking the snapshot will that is not currently used by an etcd process. Taking the snapshot will
normally not affect the performance of the member. not affect the performance of the member.
Below is an example for taking a snapshot of the keyspace served by Below is an example for taking a snapshot of the keyspace served by
`$ENDPOINT` to the file `snapshotdb`: `$ENDPOINT` to the file `snapshotdb`:
@@ -278,8 +279,8 @@ five-member etcd cluster for production Kubernetes clusters at any officially
supported scale. supported scale.
A reasonable scaling is to upgrade a three-member cluster to a five-member A reasonable scaling is to upgrade a three-member cluster to a five-member
one, when more reliability is desired. See [etcd Reconfiguration one, when more reliability is desired. See
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member) [etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member)
for information on how to add members into an existing cluster. for information on how to add members into an existing cluster.
## Restoring an etcd cluster ## Restoring an etcd cluster
@@ -290,16 +291,14 @@ different patch version of etcd also is supported. A restore operation is
employed to recover the data of a failed cluster. employed to recover the data of a failed cluster.
Before starting the restore operation, a snapshot file must be present. It can Before starting the restore operation, a snapshot file must be present. It can
either be a snapshot file from a previous backup operation, or from a either be a snapshot file from a previous backup operation, or from a remaining
remaining [data [data directory]( https://etcd.io/docs/current/op-guide/configuration/#--data-dir).
directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir). For more information and examples on restoring a cluster from a snapshot file, see
For more information and examples on restoring a cluster from a snapshot file, [etcd disaster recovery documentation](https://etcd.io/docs/current/op-guide/recovery/#restoring-a-cluster).
see [etcd disaster recovery
documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster).
If the access URLs of the restored cluster is changed from the previous If the access URLs of the restored cluster is changed from the previous
cluster, the Kubernetes API server must be reconfigured accordingly. In this cluster, the Kubernetes API server must be reconfigured accordingly. In this
case, restart Kubernetes API server with the flag case, restart Kubernetes API servers with the flag
`--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag `--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag
`--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and `--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and
`$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is `$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is
@@ -310,20 +309,19 @@ If the majority of etcd members have permanently failed, the etcd cluster is
considered failed. In this scenario, Kubernetes cannot make any changes to its considered failed. In this scenario, Kubernetes cannot make any changes to its
current state. Although the scheduled pods might continue to run, no new pods current state. Although the scheduled pods might continue to run, no new pods
can be scheduled. In such cases, recover the etcd cluster and potentially can be scheduled. In such cases, recover the etcd cluster and potentially
reconfigure Kubernetes API server to fix the issue. reconfigure Kubernetes API servers to fix the issue.
{{< note >}} {{< note >}}
If any API servers are running in your cluster, you should not attempt to If any API servers are running in your cluster, you should not attempt to
restore instances of etcd. restore instances of etcd. Instead, follow these steps to restore etcd:
Instead, follow these steps to restore etcd:
- stop *all* kube-apiserver instances - stop *all* API server instances
- restore state in all etcd instances - restore state in all etcd instances
- restart all kube-apiserver instances - restart all API server instances
We also recommend restarting any components (e.g. kube-scheduler, We also recommend restarting any components (e.g. `kube-scheduler`,
kube-controller-manager, kubelet) to ensure that they don't rely on some stale `kube-controller-manager`, `kubelet`) to ensure that they don't rely on some
data. Note that in practice, the restore takes a bit of time. During the stale data. Note that in practice, the restore takes a bit of time. During the
restoration, critical components will lose leader lock and restart themselves. restoration, critical components will lose leader lock and restart themselves.
{{< /note >}} {{< /note >}}
@@ -364,9 +362,9 @@ properly work against secure endpoints. As a result, etcd servers may fail or
disconnect briefly from the kube-apiserver. This affects kube-apiserver HA disconnect briefly from the kube-apiserver. This affects kube-apiserver HA
deployments. deployments.
The fix was made in [etcd v3.4](https://github.com/etcd-io/etcd/pull/10911) The fix was made in etcd v3.4 (and backported to v3.3.14 or later): the new
(and backported to v3.3.14 or later): the new client now creates its own client now creates its own credential bundle to correctly set authority target
credential bundle to correctly set authority target in dial function. in dial function.
Because the fix requires gRPC dependency upgrade (to v1.23.0), downstream Because the fix requires gRPC dependency upgrade (to v1.23.0), downstream
Kubernetes [did not backport etcd Kubernetes [did not backport etcd