From 4d30a795739d7dc6fd2d92c466f01066d8a8bfd0 Mon Sep 17 00:00:00 2001 From: Avi Deitcher Date: Fri, 23 Nov 2018 14:53:45 +0200 Subject: [PATCH] Revamp TLS bootstrapping doc (#11181) * Revamp TLS bootstrapping doc * Add capitalization, remove Latin abbreviations, clarify intro * minor intro tweak --- .../kubelet-tls-bootstrapping.md | 429 ++++++++++++++---- 1 file changed, 343 insertions(+), 86 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md b/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md index 551726cf10..7d861e08eb 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md @@ -10,45 +10,160 @@ content_template: templates/concept {{% capture overview %}} -This document describes how to set up TLS client certificate bootstrapping for -kubelets. Kubernetes 1.4 introduced an API for requesting certificates from a -cluster-level Certificate Authority (CA). The original intent of this API is to -enable provisioning of TLS client certificates for kubelets. The proposal can be +In a Kubernetes cluster, the components on the worker nodes - kubelet and kube-proxy - need to communicate with Kubernetes master components, specifically kube-apiserver. +In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, we strongly +recommend using client TLS certificates on nodes. + +The normal process of bootstrapping these components, especially worker nodes that need certificates so they can communicate safely with kube-apiserver, +can be a challenging process as it is often outside of the scope of Kubernetes and requires significant additional work. +This in turn, can make it challenging to initialize or scale a cluster. + +In order to simplify the process, beginning in version 1.4, Kubernetes introduced a certificate request and signing API to simplify the process. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439). +This document describes the process of node initialization, how to set up TLS client certificate bootstrapping for +kubelets, and how it works. + {{% /capture %}} {{% capture body %}} +## Initialization Process +When a worker node starts up, the kubelet does the following: + +1. Look for its `kubeconfig` file +2. Retrieve the URL of the API server and credentials, normally a TLS key and signed certificate from the `kubeconfig` file +3. Attempt to communicate with the API server using the credentials. + +Assuming that the kube-apiserver successfully validates the kubelet's credentials, it will treat the kubelet as a valid node, and begin to assign pods to it. + +Note that the above process depends upon: + +* Existence of a key and certificate on the local host in the `kubeconfig` +* The certificate having been signed by a Certificate Authority (CA) trusted by the kube-apiserver + +All of the following are responsibilities of whoever sets up and manages the cluster: + +1. Creating the CA key and certificate +2. Distributing the CA certificate to the master nodes, where kube-apiserver is running +3. Creating a key and certificate for each kubelet; strongly recommended to have a unique one, with a unique CN, for each kubelet +4. Signing the kubelet certificate using the CA key +5. Distributing the kubelet key and signed certificate to the specific node on which the kubelet is running + +The TLS Bootstrapping described in this document is intended to simplify, and partially or even completely automate, steps 3 onwards, as these are the most common when initializing or scaling +a cluster. + +### Bootstrap Initialization +In the bootstrap initialization process, the following occurs: + +1. kubelet begins +2. kubelet sees that it does _not_ have a `kubeconfig` file +3. kubelet searches for and finds a `bootstrap-kubeconfig` file +4. kubelet reads its bootstrap file, retrieving the URL of the API server and a limited usage "token" +5. kubelet connects to the API server, authenticates using the token +6. kubelet now has limited credentials to create and retrieve a certificate signing request (CSR) +7. kubelet creates a CSR for itself +8. CSR is approved in one of two ways: + * If configured, kube-controller-manager automatically approves the CSR + * If configured, an outside process, possibly a person, approves the CSR using the Kubernetes API or via `kubectl` +9. Certificate is created for the kubelet +10. Certificate is issued to the kubelet +11. kubelet retrieves the certificate +12. kubelet creates a proper `kubeconfig` with the key and signed certificate +13. kubelet begins normal operation +14. Optional: if configured, kubelet automatically requests renewal of the certificate when it is close to expiry +15. The renewed certificate is approved and issued, either automatically or manually, depending on configuration. + +The rest of this document describes the necessary steps to configure TLS Bootstrapping, and its limitations. + +## Configuration +To configure for TLS bootstrapping and optional automatic approval, you must configure options on the following components: + +* kube-apiserver +* kube-controller-manager +* kubelet +* in-cluster resources: `ClusterRoleBinding` and potentially `ClusterRole` + +In addition, you need your Kubernetes Certificate Authority (CA). + +## Certificate Authority +As without bootstrapping, you will need a Certificate Authority (CA) key and certificate. As without bootstrapping, these will be used +to sign the kubelet certificate. As before, it is your responsibility to distribute them to master nodes. + +For the purposes of this document, we will assume these have been distributed to master nodes at `/var/lib/kubernetes/ca.pem` (certificate) and `/var/lib/kubernetes/ca-key.pem` (key). +We will refer to these as "Kubernetes CA certificate and key". + +All Kubernetes components that use these certificates - kubelet, kube-apiserver, kube-controller-manager - assume the key and certificate to be PEM-encoded. + ## kube-apiserver configuration +The kube-apiserver has several requirements to enable TLS bootstrapping: -The API server should be configured with an -[authenticator](/docs/reference/access-authn-authz/authentication/) that can -authenticate tokens as a user in the `system:bootstrappers` group. +* Recognizing CA that signs the client certificate +* Authenticating the bootstrapping kubelet to the `system:bootstrappers` group +* Authorize the bootstrapping kubelet to create a certificate signing request (CSR) -This group will later be used in the controller-manager configuration to scope -approvals in the default approval controller. As this feature matures, you -should ensure tokens are bound to a Role Based Access Control (RBAC) policy -which limits requests (using the [bootstrap -token](/docs/reference/access-authn-authz/bootstrap-tokens/)) strictly to client -requests related to certificate provisioning. With RBAC in place, scoping the -tokens to a group allows for great flexibility (e.g. you could disable a -particular bootstrap group's access when you are done provisioning the nodes). +### Recognizing client certificates +This is normal for all client certificate authentication. +If not already set, add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable +client certificate authentication, referencing a certificate authority bundle +containing the signing certificate, for example +`--client-ca-file=/var/lib/kubernetes/ca.pem`. + +### Initial bootstrap authentication +In order for the bootstrapping kubelet to connect to kube-apiserver and request a certificate, it must first authenticate to the server. +You can use any [authenticator](https://kubernetes.io/docs/reference/access-authn-authz/authentication/) that can authenticate the kubelet. While any authentication strategy can be used for the kubelet's initial bootstrap credentials, the following two authenticators are recommended for ease of provisioning. -1. [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) - __beta__ +1. [Bootstrap Tokens](#bootstrap-tokens) - __beta__ 2. [Token authentication file](#token-authentication-file) -Using bootstrap tokens is currently __beta__ and will simplify the management of -bootstrap token management especially in a HA scenario. +Bootstrap tokens are a simpler and more easily managed method to authenticate kubelets, and do not require any additional flags when starting kube-apiserver. +Using bootstrap tokens is currently __beta__ as of Kubernetes version 1.12. -### Token authentication file +Whichever method you choose, the requirement is that the kubelet be able to authenticate as a user in the `system:bootstrappers` group. This group +serves two purposes: -Tokens are arbitrary but should represent at least 128 bits of entropy derived -from a secure random number generator (such as /dev/urandom on most modern Linux +1. Will be granted rights to create and retrieve CSRs. +2. Used in the controller-manager configuration to scope +approvals in the default approval controller, if automatic approval is enabled. + +As this feature matures, you +should ensure tokens are bound to a Role Based Access Control (RBAC) policy +which limits requests (using the [bootstrap +token](/docs/reference/access-authn-authz/bootstrap-tokens/)) strictly to client +requests related to certificate provisioning. With RBAC in place, scoping the +tokens to a group allows for great flexibility. For example, you could disable a +particular bootstrap group's access when you are done provisioning the nodes. + +#### Bootstrap tokens +Bootstrap tokens are described in detail [here](/docs/reference/access-authn-authz/bootstrap-tokens/). These are tokens that are stored as secrets in the Kubernetes cluster, +and then issued to the individual kubelet. You can use a single token for an entire cluster, or issue one per worker node. + +The process is two-fold: + +1. Create a Kubernetes secret with the token ID, secret and scope(s). +2. Issue the token to the kubelet + +From the kubelet's perspective, one token is like another and has no special meaning. +From the kube-apiserver's perspective, however, the bootstrap token is special. Due to its `Type`, `namespace` and `name`, kube-apiserver recognizes it as a special token, +and grants anyone authenticating with that token special bootstrap rights, notably treating them as a member of the `system:bootstrappers` group. This fulfills a basic requirement +for TLS bootstrapping. + +The details for creating the secret are available [here](/docs/reference/access-authn-authz/bootstrap-tokens/). + +If you want to use bootstrap tokens, you must enable it on kube-apiserver with the flag: + +``` +--enable-bootstrap-token-auth=true +``` + +#### Token authentication file +kube-apiserver has an ability to accept tokens as authentication. +These tokens are arbitrary but should represent at least 128 bits of entropy derived +from a secure random number generator (such as `/dev/urandom` on most modern Linux systems). There are multiple ways you can generate a token. For example: ``` @@ -69,61 +184,130 @@ systemd unit file perhaps) to enable the token file. See docs [here](/docs/reference/access-authn-authz/authentication/#static-token-file) for further details. -### Client certificate CA bundle +### Authorize kubelet to create CSR +Now that the bootstrapping node is _authenticated_ as part of the `system:bootstrappers` group, it needs to be _authorized_ to create a certificate signing request (CSR) as well as retrieve it when done. Fortunately, Kubernetes ships with a `ClusterRole` with precisely these (and just these) permissions, `system:node-bootstrappers`. -Add the `--client-ca-file=FILENAME` flag to the kube-apiserver command to enable -client certificate authentication, referencing a certificate authority bundle -containing the signing certificate (e.g. -`--client-ca-file=/var/lib/kubernetes/ca.pem`). +To do this, you just need to create a `ClusterRoleBinding` that binds the `system:bootstrappers` group to the cluster role `system:node-bootstrappers`. + +``` +# enable bootstrapping nodes to create CSR +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: create-csrs-for-bootstrapping +subjects: +- kind: Group + name: system:bootstrappers + apiGroup: rbac.authorization.k8s.io +roleRef: + kind: ClusterRole + name: system:node-bootstrappers + apiGroup: rbac.authorization.k8s.io +``` ## kube-controller-manager configuration +While the apiserver receives the requests for certificates from the kubelet and authenticates those requests, +the controller-manager is responsible for issuing actual signed certificates. -The API for requesting certificates adds a certificate-issuing control loop to -the Kubernetes Controller Manager. This takes the form of a +The controller-manager performs this function via a certificate-issuing control loop. +This takes the form of a [cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk. Currently, all certificates issued have one year validity and a default set of key usages. -### Signing assets +In order for the controller-manager to sign certificates, it needs the following: -You must provide a Certificate Authority in order to provide the cryptographic -materials necessary to issue certificates. This CA should be trusted by -kube-apiserver for authentication with the `--client-ca-file=FILENAME` flag. The -management of the CA is beyond the scope of this document but it is recommended -that you generate a dedicated CA for Kubernetes. Both certificate and key are -assumed to be PEM-encoded. +* access to the "kuberetes CA key and certificate" that you created and distributed +* enabling CSR signing -The kube-controller-manager flags are: +#### Access to key and certificate +As described earlier, you need to create a Kubernetes CA key and certificate, and distribute it to the master nodes. +These will be used by the controller-manager to sign the kubelet certificates. + +Since these signed certificates will, in turn, be used by the kubelet to authenticate as a regular kubelet to kube-apiserver, it is important that the CA +provided to the controller-manager at this stage also be trusted by kube-apiserver for authentication. This is provided to kube-apiserver +with the flag `--client-ca-file=FILENAME` (for example, `--client-ca-file=/var/lib/kubernetes/ca.pem`), as described in the kube-apiserver configuration section. + +To provide the Kubernetes CA key and certificate to kube-controller-manager, use the following flags: ``` --cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key" ``` +for example: + +``` +--cluster-signing-cert-file="/var/lib/kubernetes/ca.pem" --cluster-signing-key-file="/var/lib/kubernetes/ca-key.pem" +``` + The validity duration of signed certificates can be configured with flag: ``` --experimental-cluster-signing-duration ``` -### SubjectAccessReview Approval Controller +#### Approval +In order to approve CSRs, you need to tell the controller-manager that it is acceptable to approve them. This is done by granting +RBAC permissions to the correct group. -The `csrapproving` controller that ships as part of -[kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled -by default. The controller uses the [`SubjectAccessReview` -API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to -determine if a given user is authorized to request a CSR, then approves based on -the authorization outcome. To prevent conflicts with other approvers, the -builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized -requests. The controller also prunes expired certificates as part of garbage -collection. +There are two distinct sets of permissions: -The controller categorizes CSRs into three subresources: +* `nodeclient`: If a node is creating a new certificate for a node, then it does not have a certificate yet. It is authenticating using one of the tokens listed above, and thus is part of the group `system:bootstrappers`. +* `selfnodeclient`: If a node is renewing its certificate, then it already has a certificate (by definition), which it uses continuously to authenticate as part of the group `system:nodes`. -1. `nodeclient` - a request by a user for a client certificate with `O=system:nodes` and `CN=system:node:(node name)`. -2. `selfnodeclient` - a node renewing a client certificate with the same `O` and `CN`. A node can use its existing client certificate to authenticate this request. +How you enable these permissions depends on which version of Kubernetes, you are running: v1.8+ or below v1.8. -The following RBAC `ClusterRoles` represent the `nodeclient` and -`selfnodeclient`, capabilities. +##### Approval 1.8 +If you are running Kubernetes v1.8 or higher, Kubernetes ships with a `ClusterRole` that has the right to sign certificates. + +To enable the kubelet to request and receive a new certificate, create a `ClusterRoleBinding` that binds the group in which the bootstrapping node is a member `system:bootstrappers` to the `ClusterRole` that +grants it permission, `system:certificates.k8s.io:certificatesigningrequests:nodeclient`: + +```yml +# Approve all CSRs for the group "system:bootstrappers" +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: auto-approve-csrs-for-group +subjects: +- kind: Group + name: system:bootstrappers + apiGroup: rbac.authorization.k8s.io +roleRef: + kind: ClusterRole + name: system:certificates.k8s.io:certificatesigningrequests:nodeclient + apiGroup: rbac.authorization.k8s.io +``` + +To enable the kubelet to renew certificate, create a `ClusterRoleBinding` that binds the group in which the fully functioning node is a member `system:nodes` to the `ClusterRole` that +grants it permission, `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient`: + +```yml +# Approve renewal CSRs for the group "system:nodes" +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: auto-approve-renewals-for-nodes +subjects: +- kind: Group + name: system:nodes + apiGroup: rbac.authorization.k8s.io +roleRef: + kind: ClusterRole + name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient + apiGroup: rbac.authorization.k8s.io +``` + +##### Approval Below 1.8 +If you are running Kubernetes before v1.8, you will need to create _both_ the `ClusterRole`s _and_ the `ClusterRoleBinding`. + +The differences from 1.8 and above are: + +* You must create a `ClusterRole` with the correct permissions +* You cannot create a role with the names used in 1.8 and above `system:certificates.k8s.io:certificatesigningrequests:nodeclient` and `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` as those are reserved +* Your `ClusterRoleBinding` will have a different name in its `roleRef` because of the different `ClusterRole` name. + +The `ClusterRole`s you require are: ```yml # A ClusterRole which instructs the CSR approver to approve a user requesting @@ -149,19 +333,7 @@ rules: verbs: ["create"] ``` -As of 1.8, equivalent roles to the ones listed above are automatically created -as part of the default RBAC roles. For 1.8 clusters admins are recommended to -bind node bootstrap identities to the following roles instead of creating their -own: - -* `system:certificates.k8s.io:certificatesigningrequests:nodeclient` - - Automatically approve CSRs for client certs bound to this role. -* `system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` - - Automatically approve CSRs when a client bound to its role renews its own certificate. - -For example, to grant these permissions to identities attached to bootstrap -tokens, an admin would create a `ClusterRoleBinding` targeting the -`system:bootstrappers` group: +To enable the kubelet to request and receive a new certificate, create a `ClusterRoleBinding` that binds the group in which the bootstrapping node is a member `system:bootstrappers` to the `ClusterRole` you created that grants it permission, `approve-node-client-csr`: ```yml # Approve all CSRs for the group "system:bootstrappers" @@ -179,14 +351,15 @@ roleRef: apiGroup: rbac.authorization.k8s.io ``` -To let all nodes renew their own credentials, an admin can create a -`ClusterRoleBinding` targeting node identities: +To enable the kubelet to renew certificate, create a `ClusterRoleBinding` that binds the group in which the fully functioning node is a member `system:nodes` to the `ClusterRole` you created that +grants it permission, `approve-node-client-renewal-csr`: ```yml +# Approve renewal CSRs for the group "system:nodes" kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: - name: node1-client-cert-renewal + name: auto-approve-renewals-for-nodes subjects: - kind: Group name: system:nodes @@ -197,19 +370,74 @@ roleRef: apiGroup: rbac.authorization.k8s.io ``` + +The `csrapproving` controller that ships as part of +[kube-controller-manager](/docs/admin/kube-controller-manager/) and is enabled +by default. The controller uses the [`SubjectAccessReview` +API](/docs/reference/access-authn-authz/authorization/#checking-api-access) to +determine if a given user is authorized to request a CSR, then approves based on +the authorization outcome. To prevent conflicts with other approvers, the +builtin approver doesn't explicitly deny CSRs. It only ignores unauthorized +requests. The controller also prunes expired certificates as part of garbage +collection. + + ## kubelet configuration +Finally, with the master nodes properly set up and all of the necessary authentication and authorization in place, we can configure the kubelet. -To request a client certificate from kube-apiserver, the kubelet first needs a -path to a kubeconfig file that contains the credentials for the identity that it -will use to bootstrap its individual node identity. +The kubelet requires the following configuration to bootstrap: -If you are using a bootstrap token, you can use `kubectl config set-cluster`, -`set-credentials`, and `set-context` to build this kubeconfig. Provide the name -`kubelet-bootstrap` to `kubectl config set-credentials` and include -`--token=` as follows: +* A path to store the key and certificate it generates (optional, can use default) +* A path to a `kubeconfig` file that does not yet exist; it will place the bootstrapped config file here +* A path to a bootstrap `kubeconfig` file to provide the bootstrap token and URL for the server +* Optional: instructions to rotate certificates + +The bootstrap `kubeconfig` should be in a path available to the kubelet, for example `/var/lib/kubelet/bootstrap-kubeconfig`. + +Its format is identical to a normal `kubeconfig` file. A sample file might look as follows: + +```yml +apiVersion: v1 +clusters: +- cluster: + certificate-authority: /var/lib/kubernetes/ca.pem + server: https://my.server.example.com:6443 + name: bootstrap +contexts: +- context: + cluster: bootstrap + user: kubelet-bootstrap + name: bootstrap +current-context: bootstrap +kind: Config +preferences: {} +users: +- name: kubelet-bootstrap + user: + token: 07401b.f395accd246ae52d +``` + +The important elements to note are: + +* `certificate-authority`: path to a CA file, used to validate the server certificate presented by kube-apiserver +* `server`: URL to kube-apiserver +* `token`: the token to use + +The format of the token does not matter, as long as it matches what kube-apiserver expects. In the above example, we used a bootstrap token. + +Because the bootstrap `kubeconfig` _is_ a standard `kubeconfig`, you can use `kubectl` to generate it. To create the above example file: ``` -kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=bootstrap.kubeconfig +kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-cluster bootstrap --server='https://my.server.example.com:6443' --certificate-authority=/var/lib/kubernetes/ca.pem +kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-credentials kubelet-bootstrap --token=07401b.f395accd246ae52d +kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-context bootstrap --user=kubelet-bootstrap --cluster=bootstrap +kubectl config -kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig use-context bootstrap +``` + +To indicate to the kubelet to use the bootstrap `kubeconfig`, use the following kubelet flag: + +``` +--bootstrap-kubeconfig="/var/lib/kubelet/bootstrap-kubeconfig" --kubeconfig="/var/lib/kubelet/kubeconfig" ``` When starting the kubelet, if the file specified via `--kubeconfig` does not @@ -220,16 +448,24 @@ referencing the generated key and obtained certificate is written to the path specified by `--kubeconfig`. The certificate and key file will be placed in the directory specified by `--cert-dir`. -{{< note >}} -The following flags are required to enable this bootstrapping when starting the kubelet: +### Client and Serving Certificates +All of the above relate to kubelet _client_ certificates, specifically, the certificates a kubelet +uses to authenticate to kube-apiserver. -``` ---bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig" -``` -{{< /note >}} +A kubelet also can use _serving_ certificates. The kubelet itself exposes an https endpoint for certain features. +To secure these, the kubelet can do one of: -Additionally, in 1.7 the kubelet implements __beta__ features for enabling -rotation of both its client and/or serving certs. These can be enabled through +* use provided key and certificate, via the `--tls-private-key-file` and `--tls-cert-file` flags +* create self-signed key and certificate, if a key and certificate are not provided + +The client certificate provided by TLS bootstrapping is signed, by default, for `client auth` only, and thus cannot +be used as serving certificates, or `server auth`. + +However, you _can_ enable its server certificate, at least partially, via certificate rotation. + +### Certificate Rotation +Kubernetes v1.7 and higher kubelet implements __beta__ features for enabling +rotation of its client and/or serving certficates. These can be enabled through the respective `RotateKubeletClientCertificate` and `RotateKubeletServerCertificate` feature flags on the kubelet and are enabled by default. @@ -242,8 +478,8 @@ this feature pass the following flag to the kubelet: --rotate-certificates ``` -`RotateKubeletServerCertificate` causes the kubelet to both request a serving -certificate after bootstrapping its client credentials and to rotate that +`RotateKubeletServerCertificate` causes the kubelet **both** to request a serving +certificate after bootstrapping its client credentials **and** to rotate that certificate. To enable this feature pass the following flag to the kubelet: ``` @@ -252,12 +488,23 @@ certificate. To enable this feature pass the following flag to the kubelet: {{< note >}} The CSR approving controllers implemented in core Kubernetes do not -approve node serving certificates for [security +approve node _serving_ certificates for [security reasons](https://github.com/kubernetes/community/pull/1982). To use `RotateKubeletServerCertificate` operators need to run a custom approving controller, or manually approve the serving certificate requests. {{< /note >}} +## kube-proxy +All of TLS bootstrapping described in this document relates to the kubelet. However, +kube-proxy needs to run on every node, and requires the ability to authenticate to kube-apiserver. + +You have several options for generating kube-proxy certificates: + +* The old way: Create annd distribute certificates the same way you did for kubelet before TLS bootstrapping +* Share: Ideally, we do not share certificates. However, since both the kubelet and kube-proxy run on the same node and function together, you may consider sharing the bootstrapped kubelet certificates with kube-proxy. Be aware, however, that when the certificates expire and rotate, kubelet alone may be aware of and reload the updated certificates. kube-proxy may have the old certificates loaded in memory and may have failures in authentication. +* DaemonSet: Since the kubelet itself is loaded on each node, and is sufficient to start base services, you can run kube-proxy not as a standalone process, but rather as a daemonset in the `kube-system` namespace. Since it will be in-cluster, you can give it a proper service account with appropriate permissions to perform its activities. This may be the simplest way to configure kube-proxy. + + ## kubectl approval CSRs can be approved outside of the approval flows builtin to the controller @@ -273,4 +520,14 @@ list CSRs with `kubectl get csr` and describe one in detail with `kubectl describe csr `. An administrator can approve or deny a CSR with `kubectl certificate approve ` and `kubectl certificate deny `. + +## Limits +Although Kubernetes supports running control plane master components like kube-apiserver and kube-controller-manager in containers, and even as `Pod`s in a kubelet, as of this writing, you cannot both TLS Bootstrap a kubelet and run master plane components on it. + +The reason for this limitation is that the kubelet attempts to bootstrap communication with kube-apiserver _before_ starting any pods, even static ones define on disk and referenced via the kubelet option `--pod-manifest-path=`. Trying to do both TLS Bootstrapping and master components in kubelet leads to a race condition: kubelet needs to communicate to kube-apiserver to request certificates, yet requires those certificates to be available to start kube-apiserver. + +A fix for this issue is in progress [here](https://github.com/kubernetes/kubernetes/pull/71174). + + + {{% /capture %}}