Ashley Davis
292 lines
14 KiB
Markdown
292 lines
14 KiB
Markdown
---
|
|
reviewers:
|
|
- sig-cluster-lifecycle
|
|
title: Certificate Management with kubeadm
|
|
content_type: task
|
|
weight: 10
|
|
---
|
|
|
|
<!-- overview -->
|
|
|
|
{{< feature-state for_k8s_version="v1.15" state="stable" >}}
|
|
|
|
Client certificates generated by [kubeadm](/docs/reference/setup-tools/kubeadm/) expire after 1 year. This page explains how to manage certificate renewals with kubeadm.
|
|
|
|
## {{% heading "prerequisites" %}}
|
|
|
|
|
|
You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/best-practices/certificates/).
|
|
|
|
<!-- steps -->
|
|
|
|
## Using custom certificates {#custom-certificates}
|
|
|
|
By default, kubeadm generates all the certificates needed for a cluster to run.
|
|
You can override this behavior by providing your own certificates.
|
|
|
|
To do so, you must place them in whatever directory is specified by the
|
|
`--cert-dir` flag or the `certificatesDir` field of kubeadm's `ClusterConfiguration`.
|
|
By default this is `/etc/kubernetes/pki`.
|
|
|
|
If a given certificate and private key pair exists before running `kubeadm init`,
|
|
kubeadm does not overwrite them. This means you can, for example, copy an existing
|
|
CA into `/etc/kubernetes/pki/ca.crt` and `/etc/kubernetes/pki/ca.key`,
|
|
and kubeadm will use this CA for signing the rest of the certificates.
|
|
|
|
## External CA mode {#external-ca-mode}
|
|
|
|
It is also possible to provide only the `ca.crt` file and not the
|
|
`ca.key` file (this is only available for the root CA file, not other cert pairs).
|
|
If all other certificates and kubeconfig files are in place, kubeadm recognizes
|
|
this condition and activates the "External CA" mode. kubeadm will proceed without the
|
|
CA key on disk.
|
|
|
|
Instead, run the controller-manager standalone with `--controllers=csrsigner` and
|
|
point to the CA certificate and key.
|
|
|
|
[PKI certificates and requirements](/docs/setup/best-practices/certificates/) includes guidance on
|
|
setting up a cluster to use an external CA.
|
|
|
|
## Check certificate expiration
|
|
|
|
You can use the `check-expiration` subcommand to check when certificates expire:
|
|
|
|
```
|
|
kubeadm certs check-expiration
|
|
```
|
|
|
|
The output is similar to this:
|
|
|
|
```
|
|
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
|
|
admin.conf Dec 30, 2020 23:36 UTC 364d no
|
|
apiserver Dec 30, 2020 23:36 UTC 364d ca no
|
|
apiserver-etcd-client Dec 30, 2020 23:36 UTC 364d etcd-ca no
|
|
apiserver-kubelet-client Dec 30, 2020 23:36 UTC 364d ca no
|
|
controller-manager.conf Dec 30, 2020 23:36 UTC 364d no
|
|
etcd-healthcheck-client Dec 30, 2020 23:36 UTC 364d etcd-ca no
|
|
etcd-peer Dec 30, 2020 23:36 UTC 364d etcd-ca no
|
|
etcd-server Dec 30, 2020 23:36 UTC 364d etcd-ca no
|
|
front-proxy-client Dec 30, 2020 23:36 UTC 364d front-proxy-ca no
|
|
scheduler.conf Dec 30, 2020 23:36 UTC 364d no
|
|
|
|
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
|
|
ca Dec 28, 2029 23:36 UTC 9y no
|
|
etcd-ca Dec 28, 2029 23:36 UTC 9y no
|
|
front-proxy-ca Dec 28, 2029 23:36 UTC 9y no
|
|
```
|
|
|
|
The command shows expiration/residual time for the client certificates in the `/etc/kubernetes/pki` folder and for the client certificate embedded in the KUBECONFIG files used by kubeadm (`admin.conf`, `controller-manager.conf` and `scheduler.conf`).
|
|
|
|
Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools.
|
|
|
|
{{< warning >}}
|
|
`kubeadm` cannot manage certificates signed by an external CA.
|
|
{{< /warning >}}
|
|
|
|
{{< note >}}
|
|
`kubelet.conf` is not included in the list above because kubeadm configures kubelet
|
|
for [automatic certificate renewal](/docs/tasks/tls/certificate-rotation/)
|
|
with rotatable certificates under `/var/lib/kubelet/pki`.
|
|
To repair an expired kubelet client certificate see
|
|
[Kubelet client certificate rotation fails](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert).
|
|
{{< /note >}}
|
|
|
|
{{< warning >}}
|
|
On nodes created with `kubeadm init`, prior to kubeadm version 1.17, there is a
|
|
[bug](https://github.com/kubernetes/kubeadm/issues/1753) where you manually have to modify the contents of `kubelet.conf`. After `kubeadm init` finishes, you should update `kubelet.conf` to point to the
|
|
rotated kubelet client certificates, by replacing `client-certificate-data` and `client-key-data` with:
|
|
|
|
```yaml
|
|
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
|
|
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
|
|
```
|
|
{{< /warning >}}
|
|
|
|
## Automatic certificate renewal
|
|
|
|
kubeadm renews all the certificates during control plane [upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
|
|
|
|
This feature is designed for addressing the simplest use cases;
|
|
if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.
|
|
|
|
{{< note >}}
|
|
It is a best practice to upgrade your cluster frequently in order to stay secure.
|
|
{{< /note >}}
|
|
|
|
If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing `--certificate-renewal=false` to `kubeadm upgrade apply` or to `kubeadm upgrade node`.
|
|
|
|
{{< warning >}}
|
|
Prior to kubeadm version 1.17 there is a [bug](https://github.com/kubernetes/kubeadm/issues/1818)
|
|
where the default value for `--certificate-renewal` is `false` for the `kubeadm upgrade node`
|
|
command. In that case, you should explicitly set `--certificate-renewal=true`.
|
|
{{< /warning >}}
|
|
|
|
## Manual certificate renewal
|
|
|
|
You can renew your certificates manually at any time with the `kubeadm certs renew` command.
|
|
|
|
This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in `/etc/kubernetes/pki`.
|
|
|
|
After running the command you should restart the control plane Pods. This is required since
|
|
dynamic certificate reload is currently not supported for all components and certificates.
|
|
[Static Pods](/docs/tasks/configure-pod-container/static-pod/) are managed by the local kubelet
|
|
and not by the API Server, thus kubectl cannot be used to delete and restart them.
|
|
To restart a static Pod you can temporarily remove its manifest file from `/etc/kubernetes/manifests/`
|
|
and wait for 20 seconds (see the `fileCheckFrequency` value in [KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/).
|
|
The kubelet will terminate the Pod if it's no longer in the manifest directory.
|
|
You can then move the file back and after another `fileCheckFrequency` period, the kubelet will recreate
|
|
the Pod and the certificate renewal for the component can complete.
|
|
|
|
{{< warning >}}
|
|
If you are running an HA cluster, this command needs to be executed on all the control-plane nodes.
|
|
{{< /warning >}}
|
|
|
|
{{< note >}}
|
|
`certs renew` uses the existing certificates as the authoritative source for attributes (Common Name, Organization, SAN, etc.) instead of the kubeadm-config ConfigMap. It is strongly recommended to keep them both in sync.
|
|
{{< /note >}}
|
|
|
|
`kubeadm certs renew` provides the following options:
|
|
|
|
The Kubernetes certificates normally reach their expiration date after one year.
|
|
|
|
- `--csr-only` can be used to renew certificates with an external CA by generating certificate signing requests (without actually renewing certificates in place); see next paragraph for more information.
|
|
|
|
- It's also possible to renew a single certificate instead of all.
|
|
|
|
## Renew certificates with the Kubernetes certificates API
|
|
|
|
This section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API.
|
|
|
|
{{< caution >}}
|
|
These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead.
|
|
{{< /caution >}}
|
|
|
|
### Set up a signer
|
|
|
|
The Kubernetes Certificate Authority does not work out of the box.
|
|
You can configure an external signer such as [cert-manager](https://cert-manager.io/docs/configuration/ca/), or you can use the built-in signer.
|
|
|
|
The built-in signer is part of [`kube-controller-manager`](/docs/reference/command-line-tools-reference/kube-controller-manager/).
|
|
|
|
To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags.
|
|
|
|
If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3):
|
|
|
|
```yaml
|
|
apiVersion: kubeadm.k8s.io/v1beta3
|
|
kind: ClusterConfiguration
|
|
controllerManager:
|
|
extraArgs:
|
|
cluster-signing-cert-file: /etc/kubernetes/pki/ca.crt
|
|
cluster-signing-key-file: /etc/kubernetes/pki/ca.key
|
|
```
|
|
|
|
### Create certificate signing requests (CSR)
|
|
|
|
See [Create CertificateSigningRequest](/docs/reference/access-authn-authz/certificate-signing-requests/#create-certificatesigningrequest) for creating CSRs with the Kubernetes API.
|
|
|
|
## Renew certificates with external CA
|
|
|
|
This section provide more details about how to execute manual certificate renewal using an external CA.
|
|
|
|
To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs).
|
|
A CSR represents a request to a CA for a signed certificate for a client.
|
|
In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR.
|
|
|
|
### Create certificate signing requests (CSR)
|
|
|
|
You can create certificate signing requests with `kubeadm certs renew --csr-only`.
|
|
|
|
Both the CSR and the accompanying private key are given in the output.
|
|
You can pass in a directory with `--csr-dir` to output the CSRs to the specified location.
|
|
If `--csr-dir` is not specified, the default certificate directory (`/etc/kubernetes/pki`) is used.
|
|
|
|
Certificates can be renewed with `kubeadm certs renew --csr-only`.
|
|
As with `kubeadm init`, an output directory can be specified with the `--csr-dir` flag.
|
|
|
|
A CSR contains a certificate's name, domains, and IPs, but it does not specify usages.
|
|
It is the responsibility of the CA to specify [the correct cert usages](/docs/setup/best-practices/certificates/#all-certificates)
|
|
when issuing a certificate.
|
|
|
|
* In `openssl` this is done with the
|
|
[`openssl ca` command](https://superuser.com/questions/738612/openssl-ca-keyusage-extension).
|
|
* In `cfssl` you specify
|
|
[usages in the config file](https://github.com/cloudflare/cfssl/blob/master/doc/cmd/cfssl.txt#L170).
|
|
|
|
After a certificate is signed using your preferred method, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`).
|
|
|
|
## Certificate authority (CA) rotation {#certificate-authority-rotation}
|
|
|
|
Kubeadm does not support rotation or replacement of CA certificates out of the box.
|
|
|
|
For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/).
|
|
|
|
## Enabling signed kubelet serving certificates {#kubelet-serving-certs}
|
|
|
|
By default the kubelet serving certificate deployed by kubeadm is self-signed.
|
|
This means a connection from external services like the
|
|
[metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a
|
|
kubelet cannot be secured with TLS.
|
|
|
|
To configure the kubelets in a new kubeadm cluster to obtain properly signed serving
|
|
certificates you must pass the following minimal configuration to `kubeadm init`:
|
|
|
|
```yaml
|
|
apiVersion: kubeadm.k8s.io/v1beta3
|
|
kind: ClusterConfiguration
|
|
---
|
|
apiVersion: kubelet.config.k8s.io/v1beta1
|
|
kind: KubeletConfiguration
|
|
serverTLSBootstrap: true
|
|
```
|
|
|
|
If you have already created the cluster you must adapt it by doing the following:
|
|
- Find and edit the `kubelet-config-{{< skew latestVersion >}}` ConfigMap in the `kube-system` namespace.
|
|
In that ConfigMap, the `kubelet` key has a
|
|
[KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
|
|
document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`.
|
|
- On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml`
|
|
and restart the kubelet with `systemctl restart kubelet`
|
|
|
|
The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving
|
|
certificates by requesting them from the `certificates.k8s.io` API. One known limitation
|
|
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
|
|
approved by the default signer in the kube-controller-manager -
|
|
[`kubernetes.io/kubelet-serving`](/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
|
|
This will require action from the user or a third party controller.
|
|
|
|
These CSRs can be viewed using:
|
|
|
|
```shell
|
|
kubectl get csr
|
|
NAME AGE SIGNERNAME REQUESTOR CONDITION
|
|
csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending
|
|
csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending
|
|
```
|
|
|
|
To approve them you can do the following:
|
|
```shell
|
|
kubectl certificate approve <CSR-name>
|
|
```
|
|
|
|
By default, these serving certificate will expire after one year. Kubeadm sets the
|
|
`KubeletConfiguration` field `rotateCertificates` to `true`, which means that close
|
|
to expiration a new set of CSRs for the serving certificates will be created and must
|
|
be approved to complete the rotation. To understand more see
|
|
[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation).
|
|
|
|
If you are looking for a solution for automatic approval of these CSRs it is recommended
|
|
that you contact your cloud provider and ask if they have a CSR signer that verifies
|
|
the node identity with an out of band mechanism.
|
|
|
|
{{% thirdparty-content %}}
|
|
|
|
Third party custom controllers can be used:
|
|
- [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp)
|
|
|
|
Such a controller is not a secure mechanism unless it not only verifies the CommonName
|
|
in the CSR but also verifies the requested IPs and domain names. This would prevent
|
|
a malicious actor that has access to a kubelet client certificate to create
|
|
CSRs requesting serving certificates for any IP or domain name.
|