whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revamp TLS bootstrapping doc (#11181)

Browse Source 
* Revamp TLS bootstrapping doc

* Add capitalization, remove Latin abbreviations, clarify intro

* minor intro tweak
This commit is contained in:
Avi Deitcher
2018-11-23 14:53:45 +02:00
committed by k8s-ci-robot
parent 6537726ebe
commit 4d30a79573
1 changed files with 343 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md
+343 -86
View File
@@ -10,45 +10,160 @@ content_template: templates/concept
{{% capture overview %}} {{% capture overview %}}
This document describes how to set up TLS client certificate bootstrapping for In a Kubernetes cluster, the components on the worker nodes - kubelet and kube-proxy - need to communicate with Kubernetes master components, specifically kube-apiserver.
kubelets. Kubernetes 1.4 introduced an API for requesting certificates from a In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, we strongly
cluster-level Certificate Authority (CA). The original intent of this API is to recommend using client TLS certificates on nodes.
enable provisioning of TLS client certificates for kubelets. The proposal can be
The normal process of bootstrapping these components, especially worker nodes that need certificates so they can communicate safely with kube-apiserver,
can be a challenging process as it is often outside of the scope of Kubernetes and requires significant additional work.
This in turn, can make it challenging to initialize or scale a cluster.
In order to simplify the process, beginning in version 1.4, Kubernetes introduced a certificate request and signing API to simplify the process. The proposal can be
found [here](https://github.com/kubernetes/kubernetes/pull/20439). found [here](https://github.com/kubernetes/kubernetes/pull/20439).
This document describes the process of node initialization, how to set up TLS client certificate bootstrapping for
kubelets, and how it works.
{{% /capture %}} {{% /capture %}}
{{% capture body %}} {{% capture body %}}
## Initialization Process
When a worker node starts up, the kubelet does the following:
1. Look for its `kubeconfig` file
2. Retrieve the URL of the API server and credentials, normally a TLS key and signed certificate from the `kubeconfig` file
3. Attempt to communicate with the API server using the credentials.
Assuming that the kube-apiserver successfully validates the kubelet's credentials, it will treat the kubelet as a valid node, and begin to assign pods to it.
Note that the above process depends upon:
* Existence of a key and certificate on the local host in the `kubeconfig`
* The certificate having been signed by a Certificate Authority (CA) trusted by the kube-apiserver
All of the following are responsibilities of whoever sets up and manages the cluster:
1. Creating the CA key and certificate
2. Distributing the CA certificate to the master nodes, where kube-apiserver is running
3. Creating a key and certificate for each kubelet; strongly recommended to have a unique one, with a unique CN, for each kubelet
4. Signing the kubelet certificate using the CA key
5. Distributing the kubelet key and signed certificate to the specific node on which the kubelet is running
The TLS Bootstrapping described in this document is intended to simplify, and partially or even completely automate, steps 3 onwards, as these are the most common when initializing or scaling
a cluster.
### Bootstrap Initialization
In the bootstrap initialization process, the following occurs:
1. kubelet begins
2. kubelet sees that it does _not_ have a `kubeconfig` file
3. kubelet searches for and finds a `bootstrap-kubeconfig` file
4. kubelet reads its bootstrap file, retrieving the URL of the API server and a limited usage "token"
5. kubelet connects to the API server, authenticates using the token
6. kubelet now has limited credentials to create and retrieve a certificate signing request (CSR)
7. kubelet creates a CSR for itself
8. CSR is approved in one of two ways:
* If configured, kube-controller-manager automatically approves the CSR
* If configured, an outside process, possibly a person, approves the CSR using the Kubernetes API or via `kubectl`
9. Certificate is created for the kubelet
10. Certificate is issued to the kubelet
11. kubelet retrieves the certificate
12. kubelet creates a proper `kubeconfig` with the key and signed certificate
13. kubelet begins normal operation
14. Optional: if configured, kubelet automatically requests renewal of the certificate when it is close to expiry
15. The renewed certificate is approved and issued, either automatically or manually, depending on configuration.
The rest of this document describes the necessary steps to configure TLS Bootstrapping, and its limitations.
## Configuration
To configure for TLS bootstrapping and optional automatic approval, you must configure options on the following components:
* kube-apiserver
* kube-controller-manager
* kubelet
* in-cluster resources: `ClusterRoleBinding` and potentially `ClusterRole`
In addition, you need your Kubernetes Certificate Authority (CA).
## Certificate Authority
As without bootstrapping, you will need a Certificate Authority (CA) key and certificate. As without bootstrapping, these will be used
to sign the kubelet certificate. As before, it is your responsibility to distribute them to master nodes.
For the purposes of this document, we will assume these have been distributed to master nodes at `/var/lib/kubernetes/ca.pem` (certificate) and `/var/lib/kubernetes/ca-key.pem` (key).
We will refer to these as "Kubernetes CA certificate and key".
All Kubernetes components that use these certificates - kubelet, kube-apiserver, kube-controller-manager - assume the key and certificate to be PEM-encoded.
## kube-apiserver configuration ## kube-apiserver configuration
The kube-apiserver has several requirements to enable TLS bootstrapping:
The API server should be configured with an * Recognizing CA that signs the client certificate
[authenticator](/docs/reference/access-authn-authz/authentication/) that can * Authenticating the bootstrapping kubelet to the `system:bootstrappers` group
authenticate tokens as a user in the `system:bootstrappers` group. * Authorize the bootstrapping kubelet to create a certificate signing request (CSR)
This group will later be used in the controller-manager configuration to scope ### Recognizing client certificates
approvals in the default approval controller. As this feature matures, you This is normal for all client certificate authentication.
should ensure tokens are bound to a Role Based Access Control (RBAC) policy If not already set, add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable
which limits requests (using the [bootstrap client certificate authentication, referencing a certificate authority bundle
token](/docs/reference/access-authn-authz/bootstrap-tokens/)) strictly to client containing the signing certificate, for example
requests related to certificate provisioning. With RBAC in place, scoping the `--client-ca-file=/var/lib/kubernetes/ca.pem`.
tokens to a group allows for great flexibility (e.g. you could disable a
particular bootstrap group's access when you are done provisioning the nodes). ### Initial bootstrap authentication
In order for the bootstrapping kubelet to connect to kube-apiserver and request a certificate, it must first authenticate to the server.
You can use any [authenticator](https://kubernetes.io/docs/reference/access-authn-authz/authentication/) that can authenticate the kubelet.
While any authentication strategy can be used for the kubelet's initial While any authentication strategy can be used for the kubelet's initial
bootstrap credentials, the following two authenticators are recommended for ease bootstrap credentials, the following two authenticators are recommended for ease
of provisioning. of provisioning.
1. [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) - __beta__ 1. [Bootstrap Tokens](#bootstrap-tokens) - __beta__
2. [Token authentication file](#token-authentication-file) 2. [Token authentication file](#token-authentication-file)
Using bootstrap tokens is currently __beta__ and will simplify the management of Bootstrap tokens are a simpler and more easily managed method to authenticate kubelets, and do not require any additional flags when starting kube-apiserver.
bootstrap token management especially in a HA scenario. Using bootstrap tokens is currently __beta__ as of Kubernetes version 1.12.
### Token authentication file Whichever method you choose, the requirement is that the kubelet be able to authenticate as a user in the `system:bootstrappers` group. This group
serves two purposes:
Tokens are arbitrary but should represent at least 128 bits of entropy derived 1. Will be granted rights to create and retrieve CSRs.
from a secure random number generator (such as /dev/urandom on most modern Linux 2. Used in the controller-manager configuration to scope
approvals in the default approval controller, if automatic approval is enabled.
As this feature matures, you
should ensure tokens are bound to a Role Based Access Control (RBAC) policy
which limits requests (using the [bootstrap
token](/docs/reference/access-authn-authz/bootstrap-tokens/)) strictly to client
requests related to certificate provisioning. With RBAC in place, scoping the
tokens to a group allows for great flexibility. For example, you could disable a
particular bootstrap group's access when you are done provisioning the nodes.
#### Bootstrap tokens
Bootstrap tokens are described in detail [here](/docs/reference/access-authn-authz/bootstrap-tokens/). These are tokens that are stored as secrets in the Kubernetes cluster,
and then issued to the individual kubelet. You can use a single token for an entire cluster, or issue one per worker node.
The process is two-fold:
1. Create a Kubernetes secret with the token ID, secret and scope(s).
2. Issue the token to the kubelet
From the kubelet's perspective, one token is like another and has no special meaning.
From the kube-apiserver's perspective, however, the bootstrap token is special. Due to its `Type`, `namespace` and `name`, kube-apiserver recognizes it as a special token,
and grants anyone authenticating with that token special bootstrap rights, notably treating them as a member of the `system:bootstrappers` group. This fulfills a basic requirement
for TLS bootstrapping.
The details for creating the secret are available [here](/docs/reference/access-authn-authz/bootstrap-tokens/).
If you want to use bootstrap tokens, you must enable it on kube-apiserver with the flag:
```
--enable-bootstrap-token-auth=true
```
#### Token authentication file
kube-apiserver has an ability to accept tokens as authentication.
These tokens are arbitrary but should represent at least 128 bits of entropy derived
from a secure random number generator (such as `/dev/urandom` on most modern Linux
systems). There are multiple ways you can generate a token. For example: systems). There are multiple ways you can generate a token. For example:
``` ```
@@ -69,61 +184,130 @@ systemd unit file perhaps) to enable the token file. See docs
[here](/docs/reference/access-authn-authz/authentication/#static-token-file) for [here](/docs/reference/access-authn-authz/authentication/#static-token-file) for
further details. further details.
### Client certificate CA bundle ### Authorize kubelet to create CSR
Now that the bootstrapping node is _authenticated_ as part of the `system:bootstrappers` group, it needs to be _authorized_ to create a certificate signing request (CSR) as well as retrieve it when done. Fortunately, Kubernetes ships with a `ClusterRole` with precisely these (and just these) permissions, `system:node-bootstrappers`.
Add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable To do this, you just need to create a `ClusterRoleBinding` that binds the `system:bootstrappers` group to the cluster role `system:node-bootstrappers`.
client certificate authentication, referencing a certificate authority bundle
containing the signing certificate (e.g. ```
`--client-ca-file=/var/lib/kubernetes/ca.pem`). # enable bootstrapping nodes to create CSR
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: create-csrs-for-bootstrapping
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:node-bootstrappers
apiGroup: rbac.authorization.k8s.io
```
## kube-controller-manager configuration ## kube-controller-manager configuration
While the apiserver receives the requests for certificates from the kubelet and authenticates those requests,
the controller-manager is responsible for issuing actual signed certificates.
The API for requesting certificates adds a certificate-issuing control loop to The controller-manager performs this function via a certificate-issuing control loop.
the Kubernetes Controller Manager. This takes the form of a This takes the form of a
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using [cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using
assets on disk. Currently, all certificates issued have one year validity and a assets on disk. Currently, all certificates issued have one year validity and a
default set of key usages. default set of key usages.
### Signing assets In order for the controller-manager to sign certificates, it needs the following:
You must provide a Certificate Authority in order to provide the cryptographic * access to the "kuberetes CA key and certificate" that you created and distributed
materials necessary to issue certificates. This CA should be trusted by * enabling CSR signing
kube-apiserver for authentication with the `--client-ca-file=FILENAME` flag. The
management of the CA is beyond the scope of this document but it is recommended
that you generate a dedicated CA for Kubernetes. Both certificate and key are
assumed to be PEM-encoded.
The kube-controller-manager flags are: #### Access to key and certificate
As described earlier, you need to create a Kubernetes CA key and certificate, and distribute it to the master nodes.
These will be used by the controller-manager to sign the kubelet certificates.
Since these signed certificates will, in turn, be used by the kubelet to authenticate as a regular kubelet to kube-apiserver, it is important that the CA
provided to the controller-manager at this stage also be trusted by kube-apiserver for authentication. This is provided to kube-apiserver
with the flag `--client-ca-file=FILENAME` (for example, `--client-ca-file=/var/lib/kubernetes/ca.pem`), as described in the kube-apiserver configuration section.
To provide the Kubernetes CA key and certificate to kube-controller-manager, use the following flags:
``` ```
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key" --cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
``` ```
for example:
```
--cluster-signing-cert-file="/var/lib/kubernetes/ca.pem" --cluster-signing-key-file="/var/lib/kubernetes/ca-key.pem"
```
The validity duration of signed certificates can be configured with flag: The validity duration of signed certificates can be configured with flag:
``` ```
--experimental-cluster-signing-duration --experimental-cluster-signing-duration
``` ```
### SubjectAccessReview Approval Controller #### Approval
In order to approve CSRs, you need to tell the controller-manager that it is acceptable to approve them. This is done by granting
RBAC permissions to the correct group.
The `csrapproving` controller that ships as part of There are two distinct sets of permissions:
[kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled
by default. The controller uses the [`SubjectAccessReview`
API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to
determine if a given user is authorized to request a CSR, then approves based on
the authorization outcome. To prevent conflicts with other approvers, the
builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized
requests. The controller also prunes expired certificates as part of garbage
collection.
The controller categorizes CSRs into three subresources: * `nodeclient`: If a node is creating a new certificate for a node, then it does not have a certificate yet. It is authenticating using one of the tokens listed above, and thus is part of the group `system:bootstrappers`.
* `selfnodeclient`: If a node is renewing its certificate, then it already has a certificate (by definition), which it uses continuously to authenticate as part of the group `system:nodes`.
1. `nodeclient` - a request by a user for a client certificate with `O=system:nodes` and `CN=system:node:(node name)`. How you enable these permissions depends on which version of Kubernetes, you are running: v1.8+ or below v1.8.
2. `selfnodeclient` - a node renewing a client certificate with the same `O` and `CN`. A node can use its existing client certificate to authenticate this request.
The following RBAC `ClusterRoles` represent the `nodeclient` and ##### Approval 1.8
`selfnodeclient`, capabilities. If you are running Kubernetes v1.8 or higher, Kubernetes ships with a `ClusterRole` that has the right to sign certificates.
To enable the kubelet to request and receive a new certificate, create a `ClusterRoleBinding` that binds the group in which the bootstrapping node is a member `system:bootstrappers` to the `ClusterRole` that
grants it permission, `system:certificates.k8s.io:certificatesigningrequests:nodeclient`:
```yml
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
```
To enable the kubelet to renew certificate, create a `ClusterRoleBinding` that binds the group in which the fully functioning node is a member `system:nodes` to the `ClusterRole` that
grants it permission, `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient`:
```yml
# Approve renewal CSRs for the group "system:nodes"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
```
##### Approval Below 1.8
If you are running Kubernetes before v1.8, you will need to create _both_ the `ClusterRole`s _and_ the `ClusterRoleBinding`.
The differences from 1.8 and above are:
* You must create a `ClusterRole` with the correct permissions
* You cannot create a role with the names used in 1.8 and above `system:certificates.k8s.io:certificatesigningrequests:nodeclient` and `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` as those are reserved
* Your `ClusterRoleBinding` will have a different name in its `roleRef` because of the different `ClusterRole` name.
The `ClusterRole`s you require are:
```yml ```yml
# A ClusterRole which instructs the CSR approver to approve a user requesting # A ClusterRole which instructs the CSR approver to approve a user requesting
@@ -149,19 +333,7 @@ rules:
verbs: ["create"] verbs: ["create"]
``` ```
As of 1.8, equivalent roles to the ones listed above are automatically created To enable the kubelet to request and receive a new certificate, create a `ClusterRoleBinding` that binds the group in which the bootstrapping node is a member `system:bootstrappers` to the `ClusterRole` you created that grants it permission, `approve-node-client-csr`:
as part of the default RBAC roles. For 1.8 clusters admins are recommended to
bind node bootstrap identities to the following roles instead of creating their
own:
* `system:certificates.k8s.io:certificatesigningrequests:nodeclient`
- Automatically approve CSRs for client certs bound to this role.
* `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient`
- Automatically approve CSRs when a client bound to its role renews its own certificate.
For example, to grant these permissions to identities attached to bootstrap
tokens, an admin would create a `ClusterRoleBinding` targeting the
`system:bootstrappers` group:
```yml ```yml
# Approve all CSRs for the group "system:bootstrappers" # Approve all CSRs for the group "system:bootstrappers"
@@ -179,14 +351,15 @@ roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
``` ```
To let all nodes renew their own credentials, an admin can create a To enable the kubelet to renew certificate, create a `ClusterRoleBinding` that binds the group in which the fully functioning node is a member `system:nodes` to the `ClusterRole` you created that
`ClusterRoleBinding` targeting node identities: grants it permission, `approve-node-client-renewal-csr`:
```yml ```yml
# Approve renewal CSRs for the group "system:nodes"
kind: ClusterRoleBinding kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
metadata: metadata:
name: node1-client-cert-renewal name: auto-approve-renewals-for-nodes
subjects: subjects:
- kind: Group - kind: Group
name: system:nodes name: system:nodes
@@ -197,19 +370,74 @@ roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
``` ```
The `csrapproving` controller that ships as part of
[kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled
by default. The controller uses the [`SubjectAccessReview`
API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to
determine if a given user is authorized to request a CSR, then approves based on
the authorization outcome. To prevent conflicts with other approvers, the
builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized
requests. The controller also prunes expired certificates as part of garbage
collection.
## kubelet configuration ## kubelet configuration
Finally, with the master nodes properly set up and all of the necessary authentication and authorization in place, we can configure the kubelet.
To request a client certificate from kube-apiserver, the kubelet first needs a The kubelet requires the following configuration to bootstrap:
path to a kubeconfig file that contains the credentials for the identity that it
will use to bootstrap its individual node identity.
If you are using a bootstrap token, you can use `kubectl config set-cluster`, * A path to store the key and certificate it generates (optional, can use default)
`set-credentials`, and `set-context` to build this kubeconfig. Provide the name * A path to a `kubeconfig` file that does not yet exist; it will place the bootstrapped config file here
`kubelet-bootstrap` to `kubectl config set-credentials` and include * A path to a bootstrap `kubeconfig` file to provide the bootstrap token and URL for the server
`--token=<token-value>` as follows: * Optional: instructions to rotate certificates
The bootstrap `kubeconfig` should be in a path available to the kubelet, for example `/var/lib/kubelet/bootstrap-kubeconfig`.
Its format is identical to a normal `kubeconfig` file. A sample file might look as follows:
```yml
apiVersion: v1
clusters:
- cluster:
certificate-authority: /var/lib/kubernetes/ca.pem
server: https://my.server.example.com:6443
name: bootstrap
contexts:
- context:
cluster: bootstrap
user: kubelet-bootstrap
name: bootstrap
current-context: bootstrap
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: 07401b.f395accd246ae52d
```
The important elements to note are:
* `certificate-authority`: path to a CA file, used to validate the server certificate presented by kube-apiserver
* `server`: URL to kube-apiserver
* `token`: the token to use
The format of the token does not matter, as long as it matches what kube-apiserver expects. In the above example, we used a bootstrap token.
Because the bootstrap `kubeconfig` _is_ a standard `kubeconfig`, you can use `kubectl` to generate it. To create the above example file:
``` ```
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-cluster bootstrap --server='https://my.server.example.com:6443' --certificate-authority=/var/lib/kubernetes/ca.pem
kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-credentials kubelet-bootstrap --token=07401b.f395accd246ae52d
kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-context bootstrap --user=kubelet-bootstrap --cluster=bootstrap
kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig use-context bootstrap
```
To indicate to the kubelet to use the bootstrap `kubeconfig`, use the following kubelet flag:
```
--bootstrap-kubeconfig="/var/lib/kubelet/bootstrap-kubeconfig" --kubeconfig="/var/lib/kubelet/kubeconfig"
``` ```
When starting the kubelet, if the file specified via `--kubeconfig` does not When starting the kubelet, if the file specified via `--kubeconfig` does not
@@ -220,16 +448,24 @@ referencing the generated key and obtained certificate is written to the path
specified by `--kubeconfig`. The certificate and key file will be placed in the specified by `--kubeconfig`. The certificate and key file will be placed in the
directory specified by `--cert-dir`. directory specified by `--cert-dir`.
{{< note >}} ### Client and Serving Certificates
The following flags are required to enable this bootstrapping when starting the kubelet: All of the above relate to kubelet _client_ certificates, specifically, the certificates a kubelet
uses to authenticate to kube-apiserver.
``` A kubelet also can use _serving_ certificates. The kubelet itself exposes an https endpoint for certain features.
--bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig" To secure these, the kubelet can do one of:
```
{{< /note >}}
Additionally, in 1.7 the kubelet implements __beta__ features for enabling * use provided key and certificate, via the `--tls-private-key-file` and `--tls-cert-file` flags
rotation of both its client and/or serving certs. These can be enabled through * create self-signed key and certificate, if a key and certificate are not provided
The client certificate provided by TLS bootstrapping is signed, by default, for `client auth` only, and thus cannot
be used as serving certificates, or `server auth`.
However, you _can_ enable its server certificate, at least partially, via certificate rotation.
### Certificate Rotation
Kubernetes v1.7 and higher kubelet implements __beta__ features for enabling
rotation of its client and/or serving certficates. These can be enabled through
the respective `RotateKubeletClientCertificate` and the respective `RotateKubeletClientCertificate` and
`RotateKubeletServerCertificate` feature flags on the kubelet and are enabled by `RotateKubeletServerCertificate` feature flags on the kubelet and are enabled by
default. default.
@@ -242,8 +478,8 @@ this feature pass the following flag to the kubelet:
--rotate-certificates --rotate-certificates
``` ```
`RotateKubeletServerCertificate` causes the kubelet to both request a serving `RotateKubeletServerCertificate` causes the kubelet **both** to request a serving
certificate after bootstrapping its client credentials and to rotate that certificate after bootstrapping its client credentials **and** to rotate that
certificate. To enable this feature pass the following flag to the kubelet: certificate. To enable this feature pass the following flag to the kubelet:
``` ```
@@ -252,12 +488,23 @@ certificate. To enable this feature pass the following flag to the kubelet:
{{< note >}} {{< note >}}
The CSR approving controllers implemented in core Kubernetes do not The CSR approving controllers implemented in core Kubernetes do not
approve node serving certificates for [security approve node _serving_ certificates for [security
reasons](https://github.com/kubernetes/community/pull/1982). To use reasons](https://github.com/kubernetes/community/pull/1982). To use
`RotateKubeletServerCertificate` operators need to run a custom approving `RotateKubeletServerCertificate` operators need to run a custom approving
controller, or manually approve the serving certificate requests. controller, or manually approve the serving certificate requests.
{{< /note >}} {{< /note >}}
## kube-proxy
All of TLS bootstrapping described in this document relates to the kubelet. However,
kube-proxy needs to run on every node, and requires the ability to authenticate to kube-apiserver.
You have several options for generating kube-proxy certificates:
* The old way: Create annd distribute certificates the same way you did for kubelet before TLS bootstrapping
* Share: Ideally, we do not share certificates. However, since both the kubelet and kube-proxy run on the same node and function together, you may consider sharing the bootstrapped kubelet certificates with kube-proxy. Be aware, however, that when the certificates expire and rotate, kubelet alone may be aware of and reload the updated certificates. kube-proxy may have the old certificates loaded in memory and may have failures in authentication.
* DaemonSet: Since the kubelet itself is loaded on each node, and is sufficient to start base services, you can run kube-proxy not as a standalone process, but rather as a daemonset in the `kube-system` namespace. Since it will be in-cluster, you can give it a proper service account with appropriate permissions to perform its activities. This may be the simplest way to configure kube-proxy.
## kubectl approval ## kubectl approval
CSRs can be approved outside of the approval flows builtin to the controller CSRs can be approved outside of the approval flows builtin to the controller
@@ -273,4 +520,14 @@ list CSRs with `kubectl get csr` and describe one in detail with `kubectl
describe csr <name>`. An administrator can approve or deny a CSR with `kubectl describe csr <name>`. An administrator can approve or deny a CSR with `kubectl
certificate approve <name>` and `kubectl certificate deny <name>`. certificate approve <name>` and `kubectl certificate deny <name>`.
## Limits
Although Kubernetes supports running control plane master components like kube-apiserver and kube-controller-manager in containers, and even as `Pod`s in a kubelet, as of this writing, you cannot both TLS Bootstrap a kubelet and run master plane components on it.
The reason for this limitation is that the kubelet attempts to bootstrap communication with kube-apiserver _before_ starting any pods, even static ones define on disk and referenced via the kubelet option `--pod-manifest-path=<PATH>`. Trying to do both TLS Bootstrapping and master components in kubelet leads to a race condition: kubelet needs to communicate to kube-apiserver to request certificates, yet requires those certificates to be available to start kube-apiserver.
A fix for this issue is in progress [here](https://github.com/kubernetes/kubernetes/pull/71174).
{{% /capture %}} {{% /capture %}}