Revise manual CA rotation task page (#33874)
* Tweak namespace fetch command * Update CA certificate rotation task * Document when to reconfigure cloud-controller-manager for CA rotation * Assume that Kubernetes is at least v1.13 At the time of writing, the oldest supported version was v1.21 * Wrap page
This commit is contained in:
Tim Bannister
@@ -1,6 +1,5 @@
|
||||
---
|
||||
title: Manual Rotation of CA Certificates
|
||||
min-kubernetes-server-version: v1.13
|
||||
content_type: task
|
||||
---
|
||||
|
||||
@@ -10,11 +9,13 @@ This page shows how to manually rotate the certificate authority (CA) certificat
|
||||
|
||||
## {{% heading "prerequisites" %}}
|
||||
|
||||
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
|
||||
{{< include "task-tutorial-prereqs.md" >}}
|
||||
|
||||
|
||||
- For more information about authentication in Kubernetes, see [Authenticating](/docs/reference/access-authn-authz/authentication).
|
||||
- For more information about best practices for CA certificates, see [Single root CA](/docs/setup/best-practices/certificates/#single-root-ca).
|
||||
- For more information about authentication in Kubernetes, see
|
||||
[Authenticating](/docs/reference/access-authn-authz/authentication).
|
||||
- For more information about best practices for CA certificates, see
|
||||
[Single root CA](/docs/setup/best-practices/certificates/#single-root-ca).
|
||||
|
||||
<!-- steps -->
|
||||
|
||||
@@ -24,36 +25,41 @@ This page shows how to manually rotate the certificate authority (CA) certificat
|
||||
Make sure to back up your certificate directory along with configuration files and any other necessary files.
|
||||
|
||||
This approach assumes operation of the Kubernetes control plane in a HA configuration with multiple API servers.
|
||||
Graceful termination of the API server is also assumed so clients can cleanly disconnect from one API server and reconnect to another.
|
||||
Graceful termination of the API server is also assumed so clients can cleanly disconnect from one API server and
|
||||
reconnect to another.
|
||||
|
||||
Configurations with a single API server will experience unavailability while the API server is being restarted.
|
||||
{{< /caution >}}
|
||||
|
||||
1. Distribute the new CA certificates and private keys
|
||||
(ex: `ca.crt`, `ca.key`, `front-proxy-ca.crt`, and `front-proxy-ca.key`)
|
||||
to all your control plane nodes in the Kubernetes certificates directory.
|
||||
1. Distribute the new CA certificates and private keys (for example: `ca.crt`, `ca.key`, `front-proxy-ca.crt`,
|
||||
and `front-proxy-ca.key`) to all your control plane nodes in the Kubernetes certificates directory.
|
||||
|
||||
1. Update {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}'s `--root-ca-file` to
|
||||
include both old and new CA. Then restart the component.
|
||||
1. Update the `--root-ca-file` flag for the {{< glossary_tooltip term_id="kube-controller-manager" >}} to include
|
||||
both old and new CA, then restart the kube-controller-manager.
|
||||
|
||||
Any service account created after this point will get secrets that include both old and new CAs.
|
||||
Any {{< glossary_tooltip text="ServiceAccount" term_id="service-account" >}} created after this point will get
|
||||
Secrets that include both old and new CAs.
|
||||
|
||||
{{< note >}}
|
||||
The files specified by the kube-controller-manager flags `--client-ca-file` and `--cluster-signing-cert-file`
|
||||
cannot be CA bundles. If these flags and `--root-ca-file` point to the same `ca.crt` file which is now a
|
||||
bundle (includes both old and new CA) you will face an error. To workaround this problem you can copy the new CA to a separate
|
||||
file and make the flags `--client-ca-file` and `--cluster-signing-cert-file` point to the copy. Once `ca.crt` is no longer
|
||||
a bundle you can restore the problem flags to point to `ca.crt` and delete the copy.
|
||||
bundle (includes both old and new CA) you will face an error. To workaround this problem you can copy the new CA
|
||||
to a separate file and make the flags `--client-ca-file` and `--cluster-signing-cert-file` point to the copy.
|
||||
Once `ca.crt` is no longer a bundle you can restore the problem flags to point to `ca.crt` and delete the copy.
|
||||
|
||||
[Issue 1350](https://github.com/kubernetes/kubeadm/issues/1350) for kubeadm tracks an bug with the
|
||||
kube-controller-manager being unable to accept a CA bundle.
|
||||
{{< /note >}}
|
||||
|
||||
1. Update all service account tokens to include both old and new CA certificates.
|
||||
1. Update all Secrets that hold service account tokens to include both old and new CA certificates.
|
||||
|
||||
If any pods are started before new CA is used by API servers, they will get this update and trust both old and new CAs.
|
||||
If any Pods are started before new CA is used by API servers, the new Pods get this update and will trust both
|
||||
old and new CAs.
|
||||
|
||||
```shell
|
||||
base64_encoded_ca="$(base64 -w0 <path to file containing both old and new CAs>)"
|
||||
|
||||
for namespace in $(kubectl get ns --no-headers | awk '{print $1}'); do
|
||||
for namespace in $(kubectl get namespace --no-headers -o name | cut -d / -f 2 ); do
|
||||
for token in $(kubectl get secrets --namespace "$namespace" --field-selector type=kubernetes.io/service-account-token -o name); do
|
||||
kubectl get $token --namespace "$namespace" -o yaml | \
|
||||
/bin/sed "s/\(ca.crt:\).*/\1 ${base64_encoded_ca}/" | \
|
||||
@@ -62,15 +68,18 @@ Configurations with a single API server will experience unavailability while the
|
||||
done
|
||||
```
|
||||
|
||||
1. Restart all pods using in-cluster configs (ex: kube-proxy, coredns, etc) so they can use the updated certificate authority data from *ServiceAccount* secrets.
|
||||
1. Restart all pods using in-cluster configurations (for example: kube-proxy, CoreDNS, etc) so they can use the
|
||||
updated certificate authority data from Secrets that link to ServiceAccounts.
|
||||
|
||||
* Make sure coredns, kube-proxy and other pods using in-cluster configs are working as expected.
|
||||
* Make sure CoreDNS, kube-proxy and other Pods using in-cluster configurations are working as expected.
|
||||
|
||||
1. Append the both old and new CA to the file against `--client-ca-file` and `--kubelet-certificate-authority` flag in the `kube-apiserver` configuration.
|
||||
1. Append the both old and new CA to the file against `--client-ca-file` and `--kubelet-certificate-authority`
|
||||
flag in the `kube-apiserver` configuration.
|
||||
|
||||
1. Append the both old and new CA to the file against `--client-ca-file` flag in the `kube-scheduler` configuration.
|
||||
|
||||
1. Update certificates for user accounts by replacing the content of `client-certificate-data` and `client-key-data` respectively.
|
||||
1. Update certificates for user accounts by replacing the content of `client-certificate-data` and `client-key-data`
|
||||
respectively.
|
||||
|
||||
For information about creating certificates for individual user accounts, see
|
||||
[Configure certificates for user accounts](/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts).
|
||||
@@ -78,40 +87,46 @@ Configurations with a single API server will experience unavailability while the
|
||||
Additionally, update the `certificate-authority-data` section in the kubeconfig files,
|
||||
respectively with Base64-encoded old and new certificate authority data
|
||||
|
||||
1. Follow below steps in a rolling fashion.
|
||||
1. Update the `--root-ca-file` flag for the {{< glossary_tooltip term_id="cloud-controller-manager" >}} to include
|
||||
both old and new CA, then restart the cloud-controller-manager.
|
||||
|
||||
1. Restart any other *[aggregated api servers](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)*
|
||||
or *webhook handlers* to trust the new CA certificates.
|
||||
{{< note >}}
|
||||
If your cluster does not have a cloud-controller-manager, you can skip this step.
|
||||
{{< /note >}}
|
||||
|
||||
1. Follow the steps below in a rolling fashion.
|
||||
|
||||
1. Restart any other
|
||||
[aggregated API servers](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) or
|
||||
webhook handlers to trust the new CA certificates.
|
||||
|
||||
1. Restart the kubelet by update the file against `clientCAFile` in kubelet configuration and
|
||||
`certificate-authority-data` in kubelet.conf to use both the old and new CA on all nodes.
|
||||
`certificate-authority-data` in `kubelet.conf` to use both the old and new CA on all nodes.
|
||||
|
||||
If your kubelet is not using client certificate rotation update `client-certificate-data` and
|
||||
`client-key-data` in kubelet.conf on all nodes along with the kubelet client certificate file
|
||||
If your kubelet is not using client certificate rotation, update `client-certificate-data` and
|
||||
`client-key-data` in `kubelet.conf` on all nodes along with the kubelet client certificate file
|
||||
usually found in `/var/lib/kubelet/pki`.
|
||||
|
||||
|
||||
1. Restart API servers with the certificates (`apiserver.crt`, `apiserver-kubelet-client.crt` and
|
||||
`front-proxy-client.crt`) signed by new CA.
|
||||
You can use the existing private keys or new private keys.
|
||||
If you changed the private keys then update these in the Kubernetes certificates directory as well.
|
||||
|
||||
Since the pod trusts both old and new CAs, there will be a momentarily disconnection
|
||||
after which the pod's kube client will reconnect to the new API server
|
||||
that uses the certificate signed by the new CA.
|
||||
|
||||
* Restart Scheduler to use the new CAs.
|
||||
Since the Pods in your cluster trust both old and new CAs, there will be a momentarily disconnection
|
||||
after which pods' Kubernetes clients reconnect to the new API server.
|
||||
The new API server uses a certificate signed by the new CA.
|
||||
|
||||
* Restart the {{< glossary_tooltip term_id="kube-scheduler" text="kube-scheduler" >}} to use and
|
||||
trust the new CAs.
|
||||
* Make sure control plane components logs no TLS errors.
|
||||
|
||||
{{< note >}}
|
||||
To generate certificates and private keys for your cluster using the `openssl` command line tool, see [Certificates (`openssl`)](/docs/tasks/administer-cluster/certificates/#openssl).
|
||||
To generate certificates and private keys for your cluster using the `openssl` command line tool,
|
||||
see [Certificates (`openssl`)](/docs/tasks/administer-cluster/certificates/#openssl).
|
||||
You can also use [`cfssl`](/docs/tasks/administer-cluster/certificates/#cfssl).
|
||||
{{< /note >}}
|
||||
|
||||
1. Annotate any Daemonsets and Deployments to trigger pod replacement in a safer rolling fashion.
|
||||
|
||||
Example:
|
||||
1. Annotate any DaemonSets and Deployments to trigger pod replacement in a safer rolling fashion.
|
||||
|
||||
```shell
|
||||
for namespace in $(kubectl get namespace -o jsonpath='{.items[*].metadata.name}'); do
|
||||
@@ -129,7 +144,10 @@ Configurations with a single API server will experience unavailability while the
|
||||
see [configure pod disruption budget](/docs/tasks/run-application/configure-pdb/).
|
||||
{{< /note >}}
|
||||
|
||||
1. If your cluster is using bootstrap tokens to join nodes, update the ConfigMap `cluster-info` in the `kube-public` namespace with new CA.
|
||||
Depending on how you use StatefulSets you may also need to perform similar rolling replacement.
|
||||
|
||||
1. If your cluster is using bootstrap tokens to join nodes, update the ConfigMap `cluster-info` in the `kube-public`
|
||||
namespace with new CA.
|
||||
|
||||
```shell
|
||||
base64_encoded_ca="$(base64 -w0 /etc/kubernetes/pki/ca.crt)"
|
||||
@@ -141,19 +159,24 @@ Configurations with a single API server will experience unavailability while the
|
||||
|
||||
1. Verify the cluster functionality.
|
||||
|
||||
1. Validate the logs from control plane components, along with the kubelet and the
|
||||
kube-proxy are not throwing any tls errors, see
|
||||
[looking at the logs](/docs/tasks/debug/debug-cluster/#looking-at-logs).
|
||||
1. Check the logs from control plane components, along with the kubelet and the kube-proxy.
|
||||
Ensure those components are not reporting any TLS errors; see
|
||||
[looking at the logs](/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs) for more details.
|
||||
|
||||
1. Validate logs from any aggregated api servers and pods using in-cluster config.
|
||||
1. Validate logs from any aggregated api servers and pods using in-cluster config.
|
||||
|
||||
1. Once the cluster functionality is successfully verified:
|
||||
|
||||
1. Update all service account tokens to include new CA certificate only.
|
||||
|
||||
* All pods using an in-cluster kubeconfig will eventually need to be restarted to pick up the new SA secret for the old CA to be completely untrusted.
|
||||
* All pods using an in-cluster kubeconfig will eventually need to be restarted to pick up the new Secret,
|
||||
so that no Pods are relying on the old cluster CA.
|
||||
|
||||
1. Restart the control plane components by removing the old CA from the kubeconfig files and the files against `--client-ca-file`, `--root-ca-file` flags resp.
|
||||
1. Restart the control plane components by removing the old CA from the kubeconfig files and the files against
|
||||
`--client-ca-file`, `--root-ca-file` flags resp.
|
||||
|
||||
1. Restart kubelet by removing the old CA from file against the `clientCAFile` flag and kubelet kubeconfig file.
|
||||
1. On each node, restart the kubelet by removing the old CA from file against the `clientCAFile` flag
|
||||
and from the kubelet kubeconfig file. You should carry this out as a rolling update.
|
||||
|
||||
If your cluster lets you make this change, you can also roll it out by replacing nodes rather than
|
||||
reconfiguring them.
|
||||
|
||||
Reference in New Issue
Block a user