Fix links to etcd official documentation
This commit is contained in:
@@ -27,14 +27,14 @@ content_type: task
|
|||||||
* Ensure that no resource starvation occurs.
|
* Ensure that no resource starvation occurs.
|
||||||
|
|
||||||
Performance and stability of the cluster is sensitive to network and disk
|
Performance and stability of the cluster is sensitive to network and disk
|
||||||
IO. Any resource starvation can lead to heartbeat timeout, causing instability
|
I/O. Any resource starvation can lead to heartbeat timeout, causing instability
|
||||||
of the cluster. An unstable etcd indicates that no leader is elected. Under
|
of the cluster. An unstable etcd indicates that no leader is elected. Under
|
||||||
such circumstances, a cluster cannot make any changes to its current state,
|
such circumstances, a cluster cannot make any changes to its current state,
|
||||||
which implies no new pods can be scheduled.
|
which implies no new pods can be scheduled.
|
||||||
|
|
||||||
* Keeping stable etcd clusters is critical to the stability of Kubernetes
|
* Keeping etcd clusters stable is critical to the stability of Kubernetes
|
||||||
clusters. Therefore, run etcd clusters on dedicated machines or isolated
|
clusters. Therefore, run etcd clusters on dedicated machines or isolated
|
||||||
environments for [guaranteed resource requirements](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendations).
|
environments for [guaranteed resource requirements](https://etcd.io/docs/current/op-guide/hardware/).
|
||||||
|
|
||||||
* The minimum recommended version of etcd to run in production is `3.2.10+`.
|
* The minimum recommended version of etcd to run in production is `3.2.10+`.
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ content_type: task
|
|||||||
Operating etcd with limited resources is suitable only for testing purposes.
|
Operating etcd with limited resources is suitable only for testing purposes.
|
||||||
For deploying in production, advanced hardware configuration is required.
|
For deploying in production, advanced hardware configuration is required.
|
||||||
Before deploying etcd in production, see
|
Before deploying etcd in production, see
|
||||||
[resource requirement reference documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#example-hardware-configurations).
|
[resource requirement reference](https://etcd.io/docs/current/op-guide/hardware/#example-hardware-configurations).
|
||||||
|
|
||||||
## Starting etcd clusters
|
## Starting etcd clusters
|
||||||
|
|
||||||
@@ -60,7 +60,7 @@ Use a single-node etcd cluster only for testing purpose.
|
|||||||
--advertise-client-urls=http://$PRIVATE_IP:2379
|
--advertise-client-urls=http://$PRIVATE_IP:2379
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Start Kubernetes API server with the flag
|
2. Start the Kubernetes API server with the flag
|
||||||
`--etcd-servers=$PRIVATE_IP:2379`.
|
`--etcd-servers=$PRIVATE_IP:2379`.
|
||||||
|
|
||||||
Make sure `PRIVATE_IP` is set to your etcd client IP.
|
Make sure `PRIVATE_IP` is set to your etcd client IP.
|
||||||
@@ -69,12 +69,12 @@ Use a single-node etcd cluster only for testing purpose.
|
|||||||
|
|
||||||
For durability and high availability, run etcd as a multi-node cluster in
|
For durability and high availability, run etcd as a multi-node cluster in
|
||||||
production and back it up periodically. A five-member cluster is recommended
|
production and back it up periodically. A five-member cluster is recommended
|
||||||
in production. For more information, see [FAQ
|
in production. For more information, see
|
||||||
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/faq.md#what-is-failure-tolerance).
|
[FAQ documentation](https://etcd.io/docs/current/faq/#what-is-failure-tolerance).
|
||||||
|
|
||||||
Configure an etcd cluster either by static member information or by dynamic
|
Configure an etcd cluster either by static member information or by dynamic
|
||||||
discovery. For more information on clustering, see [etcd Clustering
|
discovery. For more information on clustering, see
|
||||||
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md).
|
[etcd clustering documentation](https://etcd.io/docs/current/op-guide/clustering/).
|
||||||
|
|
||||||
For an example, consider a five-member etcd cluster running with the following
|
For an example, consider a five-member etcd cluster running with the following
|
||||||
client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`,
|
client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`,
|
||||||
@@ -86,10 +86,10 @@ client URLs: `http://$IP1:2379`, `http://$IP2:2379`, `http://$IP3:2379`,
|
|||||||
etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379
|
etcd --listen-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379 --advertise-client-urls=http://$IP1:2379,http://$IP2:2379,http://$IP3:2379,http://$IP4:2379,http://$IP5:2379
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Start Kubernetes API servers with the flag
|
2. Start the Kubernetes API servers with the flag
|
||||||
`--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379`.
|
`--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379`.
|
||||||
|
|
||||||
Replace `IP<n>` with your client IP addresses.
|
Make sure the `IP<n>` variables are set to your client IP addresses.
|
||||||
|
|
||||||
### Multi-node etcd cluster with load balancer
|
### Multi-node etcd cluster with load balancer
|
||||||
|
|
||||||
@@ -121,16 +121,16 @@ authentication.
|
|||||||
|
|
||||||
To configure etcd with secure peer communication, specify flags
|
To configure etcd with secure peer communication, specify flags
|
||||||
`--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as
|
`--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as
|
||||||
URL schema.
|
the URL schema.
|
||||||
|
|
||||||
Similarly, to configure etcd with secure client communication, specify flags
|
Similarly, to configure etcd with secure client communication, specify flags
|
||||||
`--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as
|
`--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as
|
||||||
URL schema.
|
the URL schema.
|
||||||
|
|
||||||
### Limiting access of etcd clusters
|
### Limiting access of etcd clusters
|
||||||
|
|
||||||
After configuring secure communication, restrict the access of etcd cluster to
|
After configuring secure communication, restrict the access of etcd cluster to
|
||||||
only the Kubernetes API server. Use TLS authentication to do so.
|
only the Kubernetes API servers. Use TLS authentication to do so.
|
||||||
|
|
||||||
For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are
|
For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are
|
||||||
trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth`
|
trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth`
|
||||||
@@ -140,7 +140,7 @@ or the CA passed in by `--trusted-ca-file` flag. Specifying flags
|
|||||||
access to clients with the certificate `k8sclient.cert`.
|
access to clients with the certificate `k8sclient.cert`.
|
||||||
|
|
||||||
Once etcd is configured correctly, only clients with valid certificates can
|
Once etcd is configured correctly, only clients with valid certificates can
|
||||||
access it. To give Kubernetes API server the access, configure it with the
|
access it. To give Kubernetes API servers the access, configure them with the
|
||||||
flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and
|
flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and
|
||||||
`--etcd-cafile=ca.cert`.
|
`--etcd-cafile=ca.cert`.
|
||||||
|
|
||||||
@@ -160,11 +160,11 @@ member.
|
|||||||
|
|
||||||
Though etcd keeps unique member IDs internally, it is recommended to use a
|
Though etcd keeps unique member IDs internally, it is recommended to use a
|
||||||
unique name for each member to avoid human errors. For example, consider a
|
unique name for each member to avoid human errors. For example, consider a
|
||||||
three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1,
|
three-member etcd cluster. Let the URLs be, `member1=http://10.0.0.1`,
|
||||||
member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails,
|
`member2=http://10.0.0.2`, and `member3=http://10.0.0.3`. When `member1` fails,
|
||||||
replace it with member4=http://10.0.0.4.
|
replace it with `member4=http://10.0.0.4`.
|
||||||
|
|
||||||
1. Get the member ID of the failed member1:
|
1. Get the member ID of the failed `member1`:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list
|
etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list
|
||||||
@@ -213,21 +213,22 @@ replace it with member4=http://10.0.0.4.
|
|||||||
|
|
||||||
5. Do either of the following:
|
5. Do either of the following:
|
||||||
|
|
||||||
1. Update its `--etcd-servers` flag to make Kubernetes aware of the
|
1. Update the `--etcd-servers` flag for the Kubernetes API servers to make
|
||||||
configuration changes, then restart the Kubernetes API server.
|
Kubernetes aware of the configuration changes, then restart the
|
||||||
|
Kubernetes API servers.
|
||||||
2. Update the load balancer configuration if a load balancer is used in the
|
2. Update the load balancer configuration if a load balancer is used in the
|
||||||
deployment.
|
deployment.
|
||||||
|
|
||||||
For more information on cluster reconfiguration, see [etcd Reconfiguration
|
For more information on cluster reconfiguration, see
|
||||||
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member).
|
[etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member).
|
||||||
|
|
||||||
## Backing up an etcd cluster
|
## Backing up an etcd cluster
|
||||||
|
|
||||||
All Kubernetes objects are stored on etcd. Periodically backing up the etcd
|
All Kubernetes objects are stored on etcd. Periodically backing up the etcd
|
||||||
cluster data is important to recover Kubernetes clusters under disaster
|
cluster data is important to recover Kubernetes clusters under disaster
|
||||||
scenarios, such as losing all master nodes. The snapshot file contains all the
|
scenarios, such as losing all control plane nodes. The snapshot file contains
|
||||||
Kubernetes states and critical information. In order to keep the sensitive
|
all the Kubernetes states and critical information. In order to keep the
|
||||||
Kubernetes data safe, encrypt the snapshot files.
|
sensitive Kubernetes data safe, encrypt the snapshot files.
|
||||||
|
|
||||||
Backing up an etcd cluster can be accomplished in two ways: etcd built-in
|
Backing up an etcd cluster can be accomplished in two ways: etcd built-in
|
||||||
snapshot and volume snapshot.
|
snapshot and volume snapshot.
|
||||||
@@ -236,10 +237,10 @@ snapshot and volume snapshot.
|
|||||||
|
|
||||||
etcd supports built-in snapshot. A snapshot may either be taken from a live
|
etcd supports built-in snapshot. A snapshot may either be taken from a live
|
||||||
member with the `etcdctl snapshot save` command or by copying the
|
member with the `etcdctl snapshot save` command or by copying the
|
||||||
`member/snap/db` file from an etcd [data
|
`member/snap/db` file from an etcd
|
||||||
directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir)
|
[data directory](https://etcd.io/docs/current/op-guide/configuration/#--data-dir)
|
||||||
that is not currently used by an etcd process. Taking the snapshot will
|
that is not currently used by an etcd process. Taking the snapshot will
|
||||||
normally not affect the performance of the member.
|
not affect the performance of the member.
|
||||||
|
|
||||||
Below is an example for taking a snapshot of the keyspace served by
|
Below is an example for taking a snapshot of the keyspace served by
|
||||||
`$ENDPOINT` to the file `snapshotdb`:
|
`$ENDPOINT` to the file `snapshotdb`:
|
||||||
@@ -278,8 +279,8 @@ five-member etcd cluster for production Kubernetes clusters at any officially
|
|||||||
supported scale.
|
supported scale.
|
||||||
|
|
||||||
A reasonable scaling is to upgrade a three-member cluster to a five-member
|
A reasonable scaling is to upgrade a three-member cluster to a five-member
|
||||||
one, when more reliability is desired. See [etcd Reconfiguration
|
one, when more reliability is desired. See
|
||||||
Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member)
|
[etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member)
|
||||||
for information on how to add members into an existing cluster.
|
for information on how to add members into an existing cluster.
|
||||||
|
|
||||||
## Restoring an etcd cluster
|
## Restoring an etcd cluster
|
||||||
@@ -290,16 +291,14 @@ different patch version of etcd also is supported. A restore operation is
|
|||||||
employed to recover the data of a failed cluster.
|
employed to recover the data of a failed cluster.
|
||||||
|
|
||||||
Before starting the restore operation, a snapshot file must be present. It can
|
Before starting the restore operation, a snapshot file must be present. It can
|
||||||
either be a snapshot file from a previous backup operation, or from a
|
either be a snapshot file from a previous backup operation, or from a remaining
|
||||||
remaining [data
|
[data directory]( https://etcd.io/docs/current/op-guide/configuration/#--data-dir).
|
||||||
directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir).
|
For more information and examples on restoring a cluster from a snapshot file, see
|
||||||
For more information and examples on restoring a cluster from a snapshot file,
|
[etcd disaster recovery documentation](https://etcd.io/docs/current/op-guide/recovery/#restoring-a-cluster).
|
||||||
see [etcd disaster recovery
|
|
||||||
documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/recovery.md#restoring-a-cluster).
|
|
||||||
|
|
||||||
If the access URLs of the restored cluster is changed from the previous
|
If the access URLs of the restored cluster is changed from the previous
|
||||||
cluster, the Kubernetes API server must be reconfigured accordingly. In this
|
cluster, the Kubernetes API server must be reconfigured accordingly. In this
|
||||||
case, restart Kubernetes API server with the flag
|
case, restart Kubernetes API servers with the flag
|
||||||
`--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag
|
`--etcd-servers=$NEW_ETCD_CLUSTER` instead of the flag
|
||||||
`--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and
|
`--etcd-servers=$OLD_ETCD_CLUSTER`. Replace `$NEW_ETCD_CLUSTER` and
|
||||||
`$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is
|
`$OLD_ETCD_CLUSTER` with the respective IP addresses. If a load balancer is
|
||||||
@@ -310,20 +309,19 @@ If the majority of etcd members have permanently failed, the etcd cluster is
|
|||||||
considered failed. In this scenario, Kubernetes cannot make any changes to its
|
considered failed. In this scenario, Kubernetes cannot make any changes to its
|
||||||
current state. Although the scheduled pods might continue to run, no new pods
|
current state. Although the scheduled pods might continue to run, no new pods
|
||||||
can be scheduled. In such cases, recover the etcd cluster and potentially
|
can be scheduled. In such cases, recover the etcd cluster and potentially
|
||||||
reconfigure Kubernetes API server to fix the issue.
|
reconfigure Kubernetes API servers to fix the issue.
|
||||||
|
|
||||||
{{< note >}}
|
{{< note >}}
|
||||||
If any API servers are running in your cluster, you should not attempt to
|
If any API servers are running in your cluster, you should not attempt to
|
||||||
restore instances of etcd.
|
restore instances of etcd. Instead, follow these steps to restore etcd:
|
||||||
Instead, follow these steps to restore etcd:
|
|
||||||
|
|
||||||
- stop *all* kube-apiserver instances
|
- stop *all* API server instances
|
||||||
- restore state in all etcd instances
|
- restore state in all etcd instances
|
||||||
- restart all kube-apiserver instances
|
- restart all API server instances
|
||||||
|
|
||||||
We also recommend restarting any components (e.g. kube-scheduler,
|
We also recommend restarting any components (e.g. `kube-scheduler`,
|
||||||
kube-controller-manager, kubelet) to ensure that they don't rely on some stale
|
`kube-controller-manager`, `kubelet`) to ensure that they don't rely on some
|
||||||
data. Note that in practice, the restore takes a bit of time. During the
|
stale data. Note that in practice, the restore takes a bit of time. During the
|
||||||
restoration, critical components will lose leader lock and restart themselves.
|
restoration, critical components will lose leader lock and restart themselves.
|
||||||
{{< /note >}}
|
{{< /note >}}
|
||||||
|
|
||||||
@@ -364,9 +362,9 @@ properly work against secure endpoints. As a result, etcd servers may fail or
|
|||||||
disconnect briefly from the kube-apiserver. This affects kube-apiserver HA
|
disconnect briefly from the kube-apiserver. This affects kube-apiserver HA
|
||||||
deployments.
|
deployments.
|
||||||
|
|
||||||
The fix was made in [etcd v3.4](https://github.com/etcd-io/etcd/pull/10911)
|
The fix was made in etcd v3.4 (and backported to v3.3.14 or later): the new
|
||||||
(and backported to v3.3.14 or later): the new client now creates its own
|
client now creates its own credential bundle to correctly set authority target
|
||||||
credential bundle to correctly set authority target in dial function.
|
in dial function.
|
||||||
|
|
||||||
Because the fix requires gRPC dependency upgrade (to v1.23.0), downstream
|
Because the fix requires gRPC dependency upgrade (to v1.23.0), downstream
|
||||||
Kubernetes [did not backport etcd
|
Kubernetes [did not backport etcd
|
||||||
|
|||||||
Reference in New Issue
Block a user