Update docs/admin/kubeadm.md for 1.8 (#5440)
- Made a couple of minor wording changes (not strictly 1.8 related). - Did some reformatting (not strictly 1.8 related). - Updated references to the default token TTL (was infinite, now 24 hours). - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`. - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples. - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes. - Documented the new `--groups` flag for `kubeadm token create`. - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section. - Updated the component version table to drop 1.6 and add 1.8. - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`.
This commit is contained in:
Matt Moyer
+1
-3
@@ -65,10 +65,8 @@ toc:
|
||||
|
||||
- title: Setup Tools
|
||||
section:
|
||||
- title: Kubeadm
|
||||
path: /docs/admin/kubeadm/
|
||||
- docs/admin/kubeadm.md
|
||||
- title: Kubefed
|
||||
path: /docs/admin/kubefed/
|
||||
section:
|
||||
- docs/admin/kubefed.md
|
||||
- docs/admin/kubefed_options.md
|
||||
|
||||
+191
-46
@@ -4,11 +4,16 @@ approvers:
|
||||
- luxas
|
||||
- errordeveloper
|
||||
- jbeda
|
||||
title: kubeadm Setup Tool Reference Guide
|
||||
title: Kubeadm
|
||||
notitle: true
|
||||
---
|
||||
# kubeadm Setup Tool Reference Guide
|
||||
|
||||
This document provides information on how to use kubeadm's advanced options.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
Running `kubeadm init` bootstraps a Kubernetes cluster. This consists of the
|
||||
following steps:
|
||||
|
||||
@@ -27,8 +32,9 @@ following steps:
|
||||
dropping it in the cert directory (configured via `--cert-dir`, by default
|
||||
`/etc/kubernetes/pki`), this step is skipped.
|
||||
|
||||
1. Outputting a kubeconfig file for the kubelet to use to connect to the API
|
||||
server, as well as an additional kubeconfig file for administration.
|
||||
1. kubeadm outputs a kubeconfig file for the master's kubelet to use to connect
|
||||
to the API server, as well as an additional kubeconfig file for cluster
|
||||
administrators.
|
||||
|
||||
1. kubeadm generates Kubernetes static Pod manifests for the API server,
|
||||
controller manager and scheduler. It places them in
|
||||
@@ -46,8 +52,9 @@ following steps:
|
||||
Running `kubeadm join` on each node in the cluster consists of the following
|
||||
steps:
|
||||
|
||||
1. kubeadm downloads root CA information from the API server. It uses the token
|
||||
to verify the authenticity of that data.
|
||||
1. kubeadm downloads root CA information from the API server. By default, it
|
||||
uses the bootstrap token and the CA key hash to verify the authenticity of
|
||||
that data. The root CA can also be discovered directly via a file or URL.
|
||||
|
||||
1. kubeadm creates a local key pair. It prepares a certificate signing request
|
||||
(CSR) and sends that off to the API server for signing. The bootstrap token
|
||||
@@ -72,6 +79,8 @@ It is usually sufficient to run `kubeadm init` without any flags, but in some
|
||||
cases you might like to override the default behaviour. Here we specify all the
|
||||
flags that can be used to customise the Kubernetes installation.
|
||||
|
||||
**Options for `kubeadm init`:**
|
||||
|
||||
- `--apiserver-advertise-address`
|
||||
|
||||
This is the address the API Server will advertise to other members of the
|
||||
@@ -185,8 +194,8 @@ managing tokens](#manage-tokens) below.
|
||||
|
||||
This sets an expiration time for the token. This is specified as a duration
|
||||
from the current time. After this time the token will no longer be valid and
|
||||
will be removed. A value of 0 specifies that the token never expires. 0 is the
|
||||
default. See the [section on managing tokens](#manage-tokens) below.
|
||||
will be removed. A value of 0 specifies that the token never expires. The
|
||||
default is 24 hours. See the [section on managing tokens](#manage-tokens) below.
|
||||
|
||||
### `kubeadm join`
|
||||
|
||||
@@ -194,15 +203,24 @@ When joining a kubeadm initialized cluster, we need to establish bidirectional
|
||||
trust. This is split into discovery (having the Node trust the Kubernetes
|
||||
master) and TLS bootstrap (having the Kubernetes master trust the Node).
|
||||
|
||||
There are 2 main schemes for discovery. The first is to use a shared token along
|
||||
with the IP address of the API server. The second is to provide a file (a subset
|
||||
of the standard kubeconfig file). This file can be a local file or downloaded
|
||||
via an HTTPS URL. The forms are `kubeadm join --discovery-token
|
||||
abcdef.1234567890abcdef 1.2.3.4:6443`, `kubeadm join --discovery-file
|
||||
path/to/file.conf` or `kubeadm join --discovery-file https://url/file.conf`.
|
||||
There are two main schemes for discovery:
|
||||
|
||||
- **Using a shared token** along with the IP address of the API server and a
|
||||
hash of the root CA key:
|
||||
|
||||
`kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443`
|
||||
|
||||
- **Using a file** (a subset of the standard kubeconfig file). This file can
|
||||
be a local file or downloaded via an HTTPS URL:
|
||||
|
||||
`kubeadm join --discovery-file path/to/file.conf`
|
||||
|
||||
`kubeadm join --discovery-file https://url/file.conf`
|
||||
|
||||
Only one form can be used. If the discovery information is loaded from a URL,
|
||||
HTTPS must be used and the host installed CA bundle is used to verify the
|
||||
connection.
|
||||
connection. For details on the security tradeoffs of these mechanisms, see the
|
||||
[security model](#security-model) section below.
|
||||
|
||||
The TLS bootstrap mechanism is also driven via a shared token. This is used to
|
||||
temporarily authenticate with the Kubernetes master to submit a certificate
|
||||
@@ -216,13 +234,13 @@ can be used instead of specifying the each token individually.
|
||||
|
||||
Here's an example on how to use it:
|
||||
|
||||
`kubeadm join --token=abcdef.1234567890abcdef 192.168.1.1:6443`
|
||||
`kubeadm join --token=abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 192.168.1.1:6443`
|
||||
|
||||
Specific options:
|
||||
**Options for `kubeadm join`:**
|
||||
|
||||
- `--config`
|
||||
|
||||
Extended options a specified in the [kubeadm specific config file](#config-file).
|
||||
Extended options are specified in the [kubeadm specific config file](#config-file).
|
||||
|
||||
- `--skip-preflight-checks`
|
||||
|
||||
@@ -260,6 +278,29 @@ unnamed argument) to download and verify information about the cluster. The
|
||||
most critical part of the cluster information is the root CA bundle used to
|
||||
verify the identity of the server during subsequent TLS connections.
|
||||
|
||||
- `--discovery-token-ca-cert-hash`
|
||||
|
||||
The CA key hash is used to verify the full root CA certificate discovered during
|
||||
token-based bootstrapping. It has the format `sha256:<hex_encoded_hash>`. By
|
||||
default, the hash value is returned in the `kubeadm join` command printed at the
|
||||
end of `kubeadm init`. It is in a standard format (see
|
||||
[RFC7469](https://tools.ietf.org/html/rfc7469#section-2.4)) and can also be
|
||||
calculated by 3rd party tools or provisioning systems. For example, using the
|
||||
OpenSSL CLI:
|
||||
`openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>&/dev/null | openssl dgst -sha256 -hex`
|
||||
|
||||
_Skipping this flag is allowed in Kubernetes 1.8, but makes certain spoofing
|
||||
attacks possible._ See the [security model](#security-model) for details.
|
||||
Pass `--discovery-token-unsafe-skip-ca-verification` to silence warnings (which
|
||||
will become errors in Kubernetes 1.9).
|
||||
|
||||
- `--discovery-token-unsafe-skip-ca-verification`
|
||||
|
||||
Disable the warning/error that occurs when `--discovery-token-ca-cert-hash` is
|
||||
not provided. Passing this flag is an acknowledgement of the
|
||||
[security tradeoffs](#security-model) involved in skipping this verification
|
||||
(which may or may not be appropriate in your environment).
|
||||
|
||||
- `--tls-bootstrap-token`
|
||||
|
||||
The token used to authenticate to the API server for the purposes of TLS
|
||||
@@ -414,34 +455,53 @@ created cluster (`/etc/kubernetes/admin.conf`). You can specify an alternate
|
||||
kubeconfig file for credentials with the `--kubeconfig` to the following
|
||||
commands.
|
||||
|
||||
* `kubeadm token list` Lists the tokens along with when they expire and what the
|
||||
approved usages are.
|
||||
* `kubeadm token create` Creates a new token.
|
||||
* `--description` Set the description on the new token.
|
||||
* `--ttl duration` Set expiration time of the token as a delta from "now".
|
||||
Default is 0 for no expiration. The unit of the duration is seconds.
|
||||
* `--usages` Set the ways that the token can be used. The default is
|
||||
`signing,authentication`. These are the usages as described above.
|
||||
* `kubeadm token delete <token id>|<token id>.<token secret>` Delete a token.
|
||||
The token can either be identified with just an ID or with the entire token
|
||||
value. Only the ID is used; the token is still deleted if the secret does not
|
||||
match.
|
||||
* `kubeadm token list` - List tokens (along with their expirations, usages, and groups).
|
||||
|
||||
In addition, you can use the `kubeadm token generate` command to locally creates
|
||||
a new token. This token is of the correct form for specifying with the
|
||||
`--token` argument to `kubeadm init`.
|
||||
* `kubeadm token create` - Create a new token.
|
||||
|
||||
* `--description <description>`
|
||||
|
||||
Set the human-readable description for the new token.
|
||||
|
||||
* `--ttl <duration>`
|
||||
|
||||
Set the expiration time-to-live of the token relative to the current time.
|
||||
Default is 24 hours. A value of 0 means "never expire". The default unit
|
||||
of the duration is seconds but other units can be specified (e.g., `15m`, `1h`).
|
||||
|
||||
* `--usages <usage>[,<usage>...]`
|
||||
|
||||
Set the ways that the token can be used. The default is
|
||||
`signing,authentication`. These are the usages as described above.
|
||||
|
||||
* `--groups <group>[,<group>...]`
|
||||
|
||||
Add extra bootstrap groups that the new token will authenticate as. Can be
|
||||
specified multiple times. Each extra group must start with
|
||||
`system:bootstrappers:`. This is an advanced feature meant for custom
|
||||
bootstrap scenarios where you want to change how CSR approval works for
|
||||
different groups of nodes.
|
||||
|
||||
* `kubeadm token delete <token id>|<token id>.<token secret>` - Delete a token.
|
||||
|
||||
The token can either be identified with just an ID or with the
|
||||
entire token value. Only the ID is used; the token is still deleted if the
|
||||
secret does not match.
|
||||
|
||||
* `kubeadm token generate` - Generate a token locally.
|
||||
|
||||
Locally generate a token but do not create it on the server. This token is of
|
||||
the correct form for specifying with the `--token` argument to `kubeadm init`.
|
||||
|
||||
For the gory details on how the tokens are implemented (including managing them
|
||||
outside of kubeadm) see the [Bootstrap Token
|
||||
docs](/docs/admin/bootstrap-tokens/).
|
||||
outside of kubeadm) see the [Bootstrap Token docs](/docs/admin/bootstrap-tokens/).
|
||||
|
||||
## Automating kubeadm
|
||||
|
||||
Rather than copying the token you obtained from `kubeadm init` to each node, as
|
||||
in the [basic kubeadm tutorial](/docs/admin/kubeadm/), you can
|
||||
parallelize the token distribution for easier automation. To implement this
|
||||
automation, you must know the IP address that the master will have after it is
|
||||
started.
|
||||
in the [basic kubeadm tutorial](/docs/admin/kubeadm/), you can parallelize the
|
||||
token distribution for easier automation. To implement this automation, you must
|
||||
know the IP address that the master will have after it is started.
|
||||
|
||||
1. Generate a token. This token must have the form `<6 character string>.<16
|
||||
character string>`. More formally, it must match the regex:
|
||||
@@ -460,6 +520,91 @@ started.
|
||||
Once the cluster is up, you can grab the admin credentials from the master node
|
||||
at `/etc/kubernetes/admin.conf` and use that to talk to the cluster.
|
||||
|
||||
Note that this style of bootstrap has some relaxed security guarantees because
|
||||
it does not allow the root CA hash to be validated with
|
||||
`--discovery-token-ca-cert-hash` (since it's not generated when the nodes are
|
||||
provisioned). For details, see the [security model](#security-model).
|
||||
|
||||
## Security model
|
||||
The kubeadm discovery system has several options, each with security tradeoffs.
|
||||
The right method for your environment depends on how you provision nodes and the
|
||||
security expectations you have about your network and node lifecycles.
|
||||
|
||||
### Token-based discovery with CA pinning
|
||||
_This is the default mode in Kubernetes 1.8._ In this mode, kubeadm downloads
|
||||
the cluster configuration (including root CA) and validates it using the token
|
||||
as well as validating that the root CA public key matches the provided hash and
|
||||
that the API server certificate is valid under the root CA.
|
||||
|
||||
**Example `kubeadm join` command:**
|
||||
|
||||
- `kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443`
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Allows bootstrapping nodes to securely discover a root of trust for the
|
||||
master even if other worker nodes or the network are compromised.
|
||||
|
||||
- Convenient to execute manually since all of the information required fits
|
||||
into a single `kubeadm join` command that is easy to copy and paste.
|
||||
|
||||
**Disadvantages:**
|
||||
|
||||
- The CA hash is not normally known until the master has been provisioned,
|
||||
which can make it more difficult to build automated provisioning tools that
|
||||
use kubeadm.
|
||||
|
||||
### Token-based discovery without CA pinning
|
||||
_This was the default in Kubernetes 1.7 and earlier_, but comes with some
|
||||
important caveats. This mode relies only on the symmetric token to sign
|
||||
(HMAC-SHA256) the discovery information that establishes the root of trust for
|
||||
the master. It's still possible in Kubernetes 1.8 and above using the
|
||||
`--discovery-token-unsafe-skip-ca-verification` flag, but you should consider
|
||||
using one of the other modes if possible.
|
||||
|
||||
**Example `kubeadm join` command:**
|
||||
|
||||
- `kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-unsafe-skip-ca-verification 1.2.3.4:6443`
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Still protects against many network-level attacks.
|
||||
|
||||
- The token can be generated ahead of time and shared with the master and
|
||||
worker nodes, which can then bootstrap in parallel without coordination. This
|
||||
allows it to be used in many provisioning scenarios.
|
||||
|
||||
**Disadvantages:**
|
||||
|
||||
- If an attacker is able to steal a bootstrap token via some vulnerability,
|
||||
they can use that token (along with network-level access) to impersonate the
|
||||
master to other bootstrapping nodes. This may or may not be an appropriate
|
||||
tradeoff in your environment.
|
||||
|
||||
### File or HTTPS-based discovery
|
||||
This provides an out-of-band way to establish a root of trust between the master
|
||||
and bootstrapping nodes. Consider using this mode if you are building automated provisioning
|
||||
using kubeadm.
|
||||
|
||||
**Example `kubeadm join` commands:**
|
||||
|
||||
- `kubeadm join --discovery-file path/to/file.conf` (local file)
|
||||
|
||||
- `kubeadm join --discovery-file https://url/file.conf` (remote HTTPS URL)
|
||||
|
||||
**Advantages:**
|
||||
- Allows bootstrapping nodes to securely discover a root of trust for the
|
||||
master even if other worker nodes or the network are compromised.
|
||||
|
||||
**Disadvantages:**
|
||||
- Requires that you have some way to carry the discovery information from
|
||||
the master to the bootstrapping nodes. This might be possible, for example,
|
||||
via your cloud provider or provisioning tool. The information in this file is
|
||||
not secret, but HTTPS or equivalent is required to ensure its integrity.
|
||||
|
||||
- Less convenient to use manually since the file is difficult to copy and paste
|
||||
between nodes.
|
||||
|
||||
## Use Kubeadm with other CRI runtimes
|
||||
|
||||
Since [Kubernetes 1.6 release](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#node-components-1), Kubernetes container runtimes have been transferred to using CRI by default. Currently, the build-in container runtime is Docker which is enabled by build-in `dockershim` in `kubelet`.
|
||||
@@ -492,17 +637,17 @@ the following images are required for the cluster works will be automatically
|
||||
pulled by the kubelet if they don't exist locally while `kubeadm init` is initializing
|
||||
your master:
|
||||
|
||||
| Image Name | v1.6 release branch version | v1.7 release branch version
|
||||
| Image Name | v1.7 release branch version | v1.8 release branch version
|
||||
|---|---|---|
|
||||
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.6.x | v1.7.x
|
||||
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.6.x | v1.7.x
|
||||
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.6.x | v1.7.x
|
||||
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.6.x | v1.7.x
|
||||
| gcr.io/google_containers/kube-apiserver-${ARCH} | v1.7.x | v1.8.x
|
||||
| gcr.io/google_containers/kube-controller-manager-${ARCH} | v1.7.x | v1.8.x
|
||||
| gcr.io/google_containers/kube-scheduler-${ARCH} | v1.7.x | v1.8.x
|
||||
| gcr.io/google_containers/kube-proxy-${ARCH} | v1.7.x | v1.8.x
|
||||
| gcr.io/google_containers/etcd-${ARCH} | 3.0.17 | 3.0.17
|
||||
| gcr.io/google_containers/pause-${ARCH} | 3.0 | 3.0
|
||||
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.1 | 1.14.4
|
||||
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.1 | 1.14.4
|
||||
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.1 | 1.14.4
|
||||
| gcr.io/google_containers/k8s-dns-sidecar-${ARCH} | 1.14.4 | 1.14.4
|
||||
| gcr.io/google_containers/k8s-dns-kube-dns-${ARCH} | 1.14.4 | 1.14.4
|
||||
| gcr.io/google_containers/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.4 | 1.14.4
|
||||
|
||||
Here `v1.7.x` means the "latest patch release of the v1.7 branch".
|
||||
|
||||
|
||||
Reference in New Issue
Block a user