whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

TLS Bootstrap and Server Cert Rotation feature documentation (#10232)

Browse Source 
* TokenRequest feature documentation

* line wrapping to make review not insane

* update content for GA without major refactor

* Update kubelet-tls-bootstrapping.md
This commit is contained in:
Mike Danese
2018-09-20 09:41:43 -07:00
committed by k8s-ci-robot
parent 1cc51309a1
commit 7dbf37f2e1
1 changed files with 136 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md
+136 -91
View File
@@ -1,17 +1,20 @@
--- ---
reviewers: reviewers:
- ericchiang
- mikedanese - mikedanese
- jcbsmpsn - liggitt
- smarterclayton
- awly
title: TLS bootstrapping title: TLS bootstrapping
content_template: templates/concept content_template: templates/concept
--- ---
{{% capture overview %}} {{% capture overview %}}
This document describes how to set up TLS client certificate bootstrapping for kubelets. This document describes how to set up TLS client certificate bootstrapping for
Kubernetes 1.4 introduced an API for requesting certificates from a cluster-level Certificate Authority (CA). The original intent of this API is to enable provisioning of TLS client certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439) kubelets. Kubernetes 1.4 introduced an API for requesting certificates from a
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43). cluster-level Certificate Authority (CA). The original intent of this API is to
enable provisioning of TLS client certificates for kubelets. The proposal can be
found [here](https://github.com/kubernetes/kubernetes/pull/20439).
{{% /capture %}} {{% /capture %}}
@@ -19,53 +22,76 @@ and progress on the feature is being tracked as [feature #43](https://github.com
## kube-apiserver configuration ## kube-apiserver configuration
The API server should be configured with an [authenticator](/docs/reference/access-authn-authz/authentication/) that can authenticate tokens as a user in the `system:bootstrappers` group. The API server should be configured with an
[authenticator](/docs/reference/access-authn-authz/authentication/) that can
authenticate tokens as a user in the `system:bootstrappers` group.
This group will later be used in the controller-manager configuration to scope approvals in the default approval This group will later be used in the controller-manager configuration to scope
controller. As this feature matures, you should ensure tokens are bound to a Role-Based Access Control (RBAC) policy which limits requests approvals in the default approval controller. As this feature matures, you
(using the bootstrap token) strictly to client requests related to certificate provisioning. With RBAC in place, scoping the tokens to a group allows for great flexibility (e.g. you could disable a particular bootstrap group's access when you are done provisioning the nodes). should ensure tokens are bound to a Role Based Access Control (RBAC) policy
which limits requests (using the [bootstrap
token](/docs/reference/access-authn-authz/bootstrap-tokens/)) strictly to client
requests related to certificate provisioning. With RBAC in place, scoping the
tokens to a group allows for great flexibility (e.g. you could disable a
particular bootstrap group's access when you are done provisioning the nodes).
While any authentication strategy can be used for the kubelet's initial bootstrap credentials, the following two authenticators are recommended for ease of provisioning. While any authentication strategy can be used for the kubelet's initial
bootstrap credentials, the following two authenticators are recommended for ease
of provisioning.
1. [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) - __beta__ 1. [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) - __beta__
2. [Token authentication file](#token-authentication-file) 2. [Token authentication file](#token-authentication-file)
Using bootstrap tokens is currently __beta__ and will simplify the management of bootstrap token management especially in a HA scenario. Using bootstrap tokens is currently __beta__ and will simplify the management of
bootstrap token management especially in a HA scenario.
### Token authentication file ### Token authentication file
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example: Tokens are arbitrary but should represent at least 128 bits of entropy derived
from a secure random number generator (such as /dev/urandom on most modern Linux
systems). There are multiple ways you can generate a token. For example:
``` ```
head -c 16 /dev/urandom | od -An -t x | tr -d ' ' head -c 16 /dev/urandom | od -An -t x | tr -d ' '
``` ```
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8` will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`.
The token file should look like the following example, where the first three values can be anything and the quoted group The token file should look like the following example, where the first three
name should be as depicted: values can be anything and the quoted group name should be as depicted:
``` ```
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:bootstrappers" 02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:bootstrappers"
``` ```
Add the `--token-auth-file=FILENAME` flag to the kube-apiserver command (in your systemd unit file perhaps) to enable the token file. Add the `--token-auth-file=FILENAME` flag to the kube-apiserver command (in your
See docs [here](/docs/reference/access-authn-authz/authentication/#static-token-file) for further details. systemd unit file perhaps) to enable the token file. See docs
[here](/docs/reference/access-authn-authz/authentication/#static-token-file) for
further details.
### Client certificate CA bundle ### Client certificate CA bundle
Add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable client certificate authentication, Add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable
referencing a certificate authority bundle containing the signing certificate (e.g. `--client-ca-file=/var/lib/kubernetes/ca.pem`). client certificate authentication, referencing a certificate authority bundle
containing the signing certificate (e.g.
`--client-ca-file=/var/lib/kubernetes/ca.pem`).
## kube-controller-manager configuration ## kube-controller-manager configuration
The API for requesting certificates adds a certificate-issuing control loop to the Kubernetes Controller Manager. This takes the form of a
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk. Currently, all certificates issued have one year validity and a default set of key usages. The API for requesting certificates adds a certificate-issuing control loop to
the Kubernetes Controller Manager. This takes the form of a
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using
assets on disk. Currently, all certificates issued have one year validity and a
default set of key usages.
### Signing assets ### Signing assets
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
This CA should be trusted by kube-apiserver for authentication with the `--client-ca-file=FILENAME` flag. The management You must provide a Certificate Authority in order to provide the cryptographic
of the CA is beyond the scope of this document but it is recommended that you generate a dedicated CA for Kubernetes. materials necessary to issue certificates. This CA should be trusted by
Both certificate and key are assumed to be PEM-encoded. kube-apiserver for authentication with the `--client-ca-file=FILENAME` flag. The
management of the CA is beyond the scope of this document but it is recommended
that you generate a dedicated CA for Kubernetes. Both certificate and key are
assumed to be PEM-encoded.
The kube-controller-manager flags are: The kube-controller-manager flags are:
@@ -73,31 +99,31 @@ The kube-controller-manager flags are:
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key" --cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
``` ```
### Approval controller The validity duration of signed certificates can be configured with flag:
In 1.7 the experimental "group auto approver" controller is dropped in favor of the new `csrapproving` controller ```
that ships as part of [kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled by default. --experimental-cluster-signing-duration
The controller uses the [`SubjectAccessReview` API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to determine ```
if a given user is authorized to request a CSR, then approves based on the authorization outcome. To prevent
conflicts with other approvers, the builtin approver doesn't explicitly deny CSRs, only ignoring unauthorized requests. ### SubjectAccessReview Approval Controller
The `csrapproving` controller that ships as part of
[kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled
by default. The controller uses the [`SubjectAccessReview`
API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to
determine if a given user is authorized to request a CSR, then approves based on
the authorization outcome. To prevent conflicts with other approvers, the
builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized
requests. The controller also prunes expired certificates as part of garbage
collection.
The controller categorizes CSRs into three subresources: The controller categorizes CSRs into three subresources:
1. `nodeclient` - a request by a user for a client certificate with `O=system:nodes` and `CN=system:node:(node name)`. 1. `nodeclient` - a request by a user for a client certificate with `O=system:nodes` and `CN=system:node:(node name)`.
2. `selfnodeclient` - a node renewing a client certificate with the same `O` and `CN`. 2. `selfnodeclient` - a node renewing a client certificate with the same `O` and `CN`. A node can use its existing client certificate to authenticate this request.
3. `selfnodeserver` - a node renewing a serving certificate. (ALPHA, requires feature gate)
The checks to determine if a CSR is a `selfnodeserver` request is currently tied to the kubelet's credential rotation The following RBAC `ClusterRoles` represent the `nodeclient` and
implementation, an __alpha__ feature. As such, the definition of `selfnodeserver` will likely change in a future and `selfnodeclient`, capabilities.
requires the `RotateKubeletServerCertificate` feature gate on the controller manager. The feature progress can be
tracked at [kubernetes/features#267](https://github.com/kubernetes/features/issues/267).
```
--feature-gates=RotateKubeletServerCertificate=true
```
The following RBAC `ClusterRoles` represent the `nodeclient`, `selfnodeclient`, and `selfnodeserver` capabilities. Similar roles
may be automatically created in future releases.
```yml ```yml
# A ClusterRole which instructs the CSR approver to approve a user requesting # A ClusterRole which instructs the CSR approver to approve a user requesting
@@ -121,36 +147,21 @@ rules:
- apiGroups: ["certificates.k8s.io"] - apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeclient"] resources: ["certificatesigningrequests/selfnodeclient"]
verbs: ["create"] verbs: ["create"]
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
``` ```
As of 1.8, equivalent roles to the ones listed above are automatically created as part of the default RBAC roles. As of 1.8, equivalent roles to the ones listed above are automatically created
For 1.8 clusters admins are recommended to bind tokens to the following roles instead of creating their own: as part of the default RBAC roles. For 1.8 clusters admins are recommended to
bind node bootstrap identities to the following roles instead of creating their
own:
* `system:certificates.k8s.io:certificatesigningrequests:nodeclient` * `system:certificates.k8s.io:certificatesigningrequests:nodeclient`
- Automatically approve CSRs for client certs bound to this role. - Automatically approve CSRs for client certs bound to this role.
* `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` * `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient`
- Automatically approve CSRs when a client bound to its role renews its own certificate. - Automatically approve CSRs when a client bound to its role renews its own certificate.
These powers can be granted to credentials, such as bootstrapping tokens. For example, to replicate the behavior For example, to grant these permissions to identities attached to bootstrap
provided by the removed auto-approval flag, of approving all CSRs by a single group: tokens, an admin would create a `ClusterRoleBinding` targeting the
`system:bootstrappers` group:
```
# REMOVED: This flag no longer works as of 1.7.
--insecure-experimental-approve-all-kubelet-csrs-for-group="system:bootstrappers"
```
An admin would create a `ClusterRoleBinding` targeting that group.
```yml ```yml
# Approve all CSRs for the group "system:bootstrappers" # Approve all CSRs for the group "system:bootstrappers"
@@ -168,8 +179,8 @@ roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
``` ```
To let a node renew its own credentials, an admin can construct a `ClusterRoleBinding` targeting To let all nodes renew their own credentials, an admin can create a
that node's credentials: `ClusterRoleBinding` targeting node identities:
```yml ```yml
kind: ClusterRoleBinding kind: ClusterRoleBinding
@@ -177,8 +188,8 @@ apiVersion: rbac.authorization.k8s.io/v1
metadata: metadata:
name: node1-client-cert-renewal name: node1-client-cert-renewal
subjects: subjects:
- kind: User - kind: Group
name: system:node:node-1 # Let "node-1" renew its client certificate. name: system:nodes
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
roleRef: roleRef:
kind: ClusterRole kind: ClusterRole
@@ -186,18 +197,28 @@ roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
``` ```
Deleting the binding will prevent the node from renewing its client credentials, effectively
removing it from the cluster once its certificate expires.
## kubelet configuration ## kubelet configuration
To request a client certificate from kube-apiserver, the kubelet first needs a path to a kubeconfig file that contains the
bootstrap authentication token. You can use `kubectl config set-cluster`, `set-credentials`, and `set-context` to build this kubeconfig. Provide the name `kubelet-bootstrap` to `kubectl config set-credentials` and include `--token=<token-value>` as follows: To request a client certificate from kube-apiserver, the kubelet first needs a
path to a kubeconfig file that contains the credentials for the identity that it
will use to bootstrap its individual node identity.
If you are using a bootstrap token, you can use `kubectl config set-cluster`,
`set-credentials`, and `set-context` to build this kubeconfig. Provide the name
`kubelet-bootstrap` to `kubectl config set-credentials` and include
`--token=<token-value>` as follows:
``` ```
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig
``` ```
When starting the kubelet, if the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On approval of the certificate request and receipt back by the kubelet, a kubeconfig file referencing the generated key and obtained certificate is written to the path specified by `--kubeconfig`. The certificate and key file will be placed in the directory specified by `--cert-dir`. When starting the kubelet, if the file specified via `--kubeconfig` does not
exist, the bootstrap kubeconfig specified via `--bootstrap-kubeconfig` is used
to request a client certificate from the API server. On approval of the
certificate request and receipt back by the kubelet, a kubeconfig file
referencing the generated key and obtained certificate is written to the path
specified by `--kubeconfig`. The certificate and key file will be placed in the
directory specified by `--cert-dir`.
{{< note >}} {{< note >}}
**Note:** The following flags are required to enable this bootstrapping when starting the kubelet: **Note:** The following flags are required to enable this bootstrapping when starting the kubelet:
@@ -207,25 +228,49 @@ When starting the kubelet, if the file specified by `--kubeconfig` does not exis
``` ```
{{< /note >}} {{< /note >}}
Additionally, in 1.7 the kubelet implements __alpha__ features for enabling rotation of both its client and/or serving certs. Additionally, in 1.7 the kubelet implements __beta__ features for enabling
These can be enabled through the respective `RotateKubeletClientCertificate` and `RotateKubeletServerCertificate` feature rotation of both its client and/or serving certs. These can be enabled through
flags on the kubelet, but may change in backward incompatible ways in future releases. the respective `RotateKubeletClientCertificate` and
`RotateKubeletServerCertificate` feature flags on the kubelet and are enabled by
default.
`RotateKubeletClientCertificate` causes the kubelet to rotate its client
certificates by creating new CSRs as its existing credentials expire. To enable
this feature pass the following flag to the kubelet:
``` ```
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --rotate-certificates
``` ```
`RotateKubeletClientCertificate` causes the kubelet to rotate its client certificates by creating new CSRs as its existing `RotateKubeletServerCertificate` causes the kubelet to both request a serving
credentials expire. `RotateKubeletServerCertificate` causes the kubelet to both request a serving certificate after certificate after bootstrapping its client credentials and to rotate that
bootstrapping its client credentials and rotate the certificate. The serving cert currently does not request DNS or IP certificate. To enable this feature pass the following flag to the kubelet:
SANs.
```
--rotate-server-certificates
```
{{< note >}}
**Note:** The CSR approving controllers implemented in core Kubernetes do not
approve node serving certificates for [security
reasons](https://github.com/kubernetes/community/pull/1982). To use
`RotateKubeletServerCertificate` operators need to run a custom approving
controller, or manually approve the serving certificate requests.
{{< /note >}}
## kubectl approval ## kubectl approval
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
"Approved" status by an appropriately-privileged user. This is intended to eventually be an automated process handled by an external CSRs can be approved outside of the approval flows builtin to the controller
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl. manager.
An administrator can list CSRs with `kubectl get csr` and describe one in detail with `kubectl describe csr <name>`. Before the 1.6 release there were
[no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver had to update The signing controller does not immediately sign all certificate requests.
the Status field directly ([rough how-to](https://github.com/gtank/csrctl)). Later versions of Kubernetes offer `kubectl certificate approve <name>` and `kubectl certificate deny <name>` commands. Instead, it waits until they have been flagged with an "Approved" status by an
appropriately-privileged user. This flow is intended to allow for automated
approval handled by an external approval controller or the approval controller
implemented in the core controller-manager. However cluster administrators can
also manually approve certificate requests using kubectl. An administrator can
list CSRs with `kubectl get csr` and describe one in detail with `kubectl
describe csr <name>`. An administrator can approve or deny a CSR with `kubectl
certificate approve <name>` and `kubectl certificate deny <name>`.
{{% /capture %}} {{% /capture %}}