whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reword kubelet live reconfiguration task (#18629)

Browse Source 
- Revise version requirements
- Use glossary tooltips in summary
- Use sentence case for headings
- Write kubelet in lowercase where appropriate
- Add “What's next” section
This commit is contained in:
Tim Bannister
2020-02-07 19:13:46 +00:00
committed by GitHub
parent 40869a3695
commit ecda7af470
1 changed files with 100 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/administer-cluster/reconfigure-kubelet.md
+100 -64
View File
@@ -4,17 +4,20 @@ reviewers:
- dawnchen - dawnchen
title: Reconfigure a Node's Kubelet in a Live Cluster title: Reconfigure a Node's Kubelet in a Live Cluster
content_template: templates/task content_template: templates/task
min-kubernetes-server-version: v1.11
--- ---
{{% capture overview %}} {{% capture overview %}}
{{< feature-state for_k8s_version="v1.11" state="beta" >}} {{< feature-state for_k8s_version="v1.11" state="beta" >}}
[Dynamic Kubelet Configuration](https://github.com/kubernetes/enhancements/issues/281) [Dynamic Kubelet Configuration](https://github.com/kubernetes/enhancements/issues/281)
allows you to change the configuration of each Kubelet in a live Kubernetes allows you to change the configuration of each
cluster by deploying a ConfigMap and configuring each Node to use it. {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} in a running Kubernetes cluster,
by deploying a {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}} and configuring
each {{< glossary_tooltip term_id="node" >}} to use it.
{{< warning >}} {{< warning >}}
All Kubelet configuration parameters can be changed dynamically, All kubelet configuration parameters can be changed dynamically,
but this is unsafe for some parameters. Before deciding to change a parameter but this is unsafe for some parameters. Before deciding to change a parameter
dynamically, you need a strong understanding of how that change will affect your dynamically, you need a strong understanding of how that change will affect your
cluster's behavior. Always carefully test configuration changes on a small set cluster's behavior. Always carefully test configuration changes on a small set
@@ -25,38 +28,49 @@ fields is available in the inline `KubeletConfiguration`
{{% /capture %}} {{% /capture %}}
{{% capture prerequisites %}} {{% capture prerequisites %}}
- Kubernetes v1.11 or higher on both the Master and the Nodes You need to have a Kubernetes cluster.
- kubectl v1.11 or higher, configured to communicate with the cluster You also need kubectl v1.11 or higher, configured to communicate with your cluster.
- The Kubelet's `--dynamic-config-dir` flag must be set to a writable {{< version-check >}}
directory on the Node. Your cluster API server version (eg v1.12) must be no more than one minor
version away from the version of kubectl that you are using. For example,
if your cluster is running v1.16 then you can use kubectl v1.15, v1.16
or v1.17; other combinations
[aren't supported](/docs/setup/release/version-skew-policy/#kubectl).
Some of the examples use the commandline tool
[jq](https://stedolan.github.io/jq/). You do not need `jq` to complete the task,
because there are manual alternatives.
For each node that you're reconfiguring, you must set the kubelet
`--dynamic-config-dir` flag to a writable directory.
{{% /capture %}} {{% /capture %}}
{{% capture steps %}} {{% capture steps %}}
## Reconfiguring the Kubelet on a Live Node in your Cluster ## Reconfiguring the kubelet on a running node in your cluster
### Basic Workflow Overview ### Basic workflow overview
The basic workflow for configuring a Kubelet in a live cluster is as follows: The basic workflow for configuring a kubelet in a live cluster is as follows:
1. Write a YAML or JSON configuration file containing the 1. Write a YAML or JSON configuration file containing the
Kubelet's configuration. kubelet's configuration.
2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane. 2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
3. Update the Kubelet's corresponding Node object to use this ConfigMap. 3. Update the kubelet's corresponding Node object to use this ConfigMap.
Each Kubelet watches a configuration reference on its respective Node object. Each kubelet watches a configuration reference on its respective Node object.
When this reference changes, the Kubelet downloads the new configuration, When this reference changes, the kubelet downloads the new configuration,
updates a local reference to refer to the file, and exits. updates a local reference to refer to the file, and exits.
For the feature to work correctly, you must be running an OS-level service For the feature to work correctly, you must be running an OS-level service
manager (such as systemd), which will restart the Kubelet if it exits. When the manager (such as systemd), which will restart the kubelet if it exits. When the
Kubelet is restarted, it will begin using the new configuration. kubelet is restarted, it will begin using the new configuration.
The new configuration completely overrides configuration provided by `--config`, The new configuration completely overrides configuration provided by `--config`,
and is overridden by command-line flags. Unspecified values in the new configuration and is overridden by command-line flags. Unspecified values in the new configuration
will receive default values appropriate to the configuration version will receive default values appropriate to the configuration version
(e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags. (e.g. `kubelet.config.k8s.io/v1beta1`), unless overridden by flags.
The status of the Node's Kubelet configuration is reported via The status of the Node's kubelet configuration is reported via
`Node.Spec.Status.Config`. Once you have updated a Node to use the new `Node.Spec.Status.Config`. Once you have updated a Node to use the new
ConfigMap, you can observe this status to confirm that the Node is using the ConfigMap, you can observe this status to confirm that the Node is using the
intended configuration. intended configuration.
@@ -70,7 +84,7 @@ mind that it is also valid for multiple Nodes to consume the same ConfigMap.
{{< warning >}} {{< warning >}}
While it is *possible* to change the configuration by While it is *possible* to change the configuration by
updating the ConfigMap in-place, this causes all Kubelets configured with updating the ConfigMap in-place, this causes all kubelets configured with
that ConfigMap to update simultaneously. It is much safer to treat ConfigMaps that ConfigMap to update simultaneously. It is much safer to treat ConfigMaps
as immutable by convention, aided by `kubectl`'s `--append-hash` option, as immutable by convention, aided by `kubectl`'s `--append-hash` option,
and incrementally roll out updates to `Node.Spec.ConfigSource`. and incrementally roll out updates to `Node.Spec.ConfigSource`.
@@ -91,23 +105,35 @@ and debug issues. The compromise, however, is that you must start with knowledge
of the existing configuration to ensure that you only change the fields you of the existing configuration to ensure that you only change the fields you
intend to change. intend to change.
Ideally, the Kubelet would be bootstrapped from a file on disk The kubelet loads settings from its configuration file, but you can set command
and you could edit this file (which could also be version-controlled), line flags to override the configuration in the file. This means that if you
to create the first Kubelet ConfigMap only know the contents of the configuration file, and you don't know the
(see [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)), command line overrides, then you do not know the running configuration either.
Currently, the Kubelet is bootstrapped with **a combination of this file and command-line flags**
that can override the configuration in the file. Because you need to know the running configuration in order to override it,
As a workaround, you can generate a config file containing a Node's current you can fetch the running configuration from the kubelet. You can generate a
configuration by accessing the Kubelet server's `configz` endpoint via the config file containing a Node's current configuration by accessing the kubelet's
kubectl proxy. This endpoint, in its current implementation, is intended to be `configz` endpoint, through `kubectl proxy`. The next section explains how to
used only as a debugging aid. Do not rely on the behavior of this endpoint for do this.
production scenarios. The examples below use the `jq` command to streamline
working with JSON. To follow the tasks as written, you need to have `jq` {{< caution >}}
installed, but you can adapt the tasks if you prefer to extract the The kubelet's `configz` endpoint is there to help with debugging, and is not
`kubeletconfig` subobject manually. a stable part of kubelet behavior.
Do not rely on the behavior of this endpoint for production scenarios or for
use with automated tools.
{{< /caution >}}
For more information on configuring the kubelet via a configuration file, see
[Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file)).
#### Generate the configuration file #### Generate the configuration file
{{< note >}}
The steps below use the `jq` command to streamline working with JSON.
To follow the tasks as written, you need to have `jq` installed. You can
adapt the steps if you prefer to extract the `kubeletconfig` subobject manually.
{{< /note >}}
1. Choose a Node to reconfigure. In this example, the name of this Node is 1. Choose a Node to reconfigure. In this example, the name of this Node is
referred to as `NODE_NAME`. referred to as `NODE_NAME`.
2. Start the kubectl proxy in the background using the following command: 2. Start the kubectl proxy in the background using the following command:
@@ -122,20 +148,22 @@ installed, but you can adapt the tasks if you prefer to extract the
For example: `${NODE_NAME}` will be rewritten as `$\{NODE_NAME\}` during the paste. For example: `${NODE_NAME}` will be rewritten as `$\{NODE_NAME\}` during the paste.
You must remove the backslashes before running the command, or the command will fail. You must remove the backslashes before running the command, or the command will fail.
```bash ```bash
NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME} NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}
``` ```
{{< note >}} {{< note >}}
You need to manually add the `kind` and `apiVersion` to the downloaded You need to manually add the `kind` and `apiVersion` to the downloaded
object, because they are not reported by the `configz` endpoint. object, because those fields are not reported by the `configz` endpoint.
{{< /note >}} {{< /note >}}
#### Edit the configuration file #### Edit the configuration file
Using a text editor, change one of the parameters in the Using a text editor, change one of the parameters in the
file generated by the previous procedure. For example, you file generated by the previous procedure. For example, you
might edit the QPS parameter `eventRecordQPS`. might edit the parameter `eventRecordQPS`, that controls
rate limiting for event recording.
#### Push the configuration file to the control plane #### Push the configuration file to the control plane
@@ -162,12 +190,12 @@ data:
{...} {...}
``` ```
The ConfigMap is created in the `kube-system` namespace because this You created that ConfigMap inside the `kube-system` namespace because the kubelet
ConfigMap configures a Kubelet, which is a Kubernetes system component. is a Kubernetes system component.
The `--append-hash` option appends a short checksum of the ConfigMap contents The `--append-hash` option appends a short checksum of the ConfigMap contents
to the name. This is convenient for an edit-then-push workflow, because it to the name. This is convenient for an edit-then-push workflow, because it
automatically, yet deterministically, generates new names for new ConfigMaps. automatically, yet deterministically, generates new names for new resources.
The name that includes this generated hash is referred to as `CONFIG_MAP_NAME` The name that includes this generated hash is referred to as `CONFIG_MAP_NAME`
in the following examples. in the following examples.
@@ -185,13 +213,13 @@ In your text editor, add the following YAML under `spec`:
```yaml ```yaml
configSource: configSource:
configMap: configMap:
name: CONFIG_MAP_NAME name: CONFIG_MAP_NAME # replace CONFIG_MAP_NAME with the name of the ConfigMap
namespace: kube-system namespace: kube-system
kubeletConfigKey: kubelet kubeletConfigKey: kubelet
``` ```
You must specify all three of `name`, `namespace`, and `kubeletConfigKey`. You must specify all three of `name`, `namespace`, and `kubeletConfigKey`.
The `kubeletConfigKey` parameter shows the Kubelet which key of the ConfigMap The `kubeletConfigKey` parameter shows the kubelet which key of the ConfigMap
contains its config. contains its config.
#### Observe that the Node begins using the new configuration #### Observe that the Node begins using the new configuration
@@ -200,16 +228,16 @@ Retrieve the Node using the `kubectl get node ${NODE_NAME} -o yaml` command and
`Node.Status.Config`. The config sources corresponding to the `active`, `Node.Status.Config`. The config sources corresponding to the `active`,
`assigned`, and `lastKnownGood` configurations are reported in the status. `assigned`, and `lastKnownGood` configurations are reported in the status.
- The `active` configuration is the version the Kubelet is currently running with. - The `active` configuration is the version the kubelet is currently running with.
- The `assigned` configuration is the latest version the Kubelet has resolved based on - The `assigned` configuration is the latest version the kubelet has resolved based on
`Node.Spec.ConfigSource`. `Node.Spec.ConfigSource`.
- The `lastKnownGood` configuration is the version the - The `lastKnownGood` configuration is the version the
Kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`. kubelet will fall back to if an invalid config is assigned in `Node.Spec.ConfigSource`.
The`lastKnownGood` configuration might not be present if it is set to its default value, The`lastKnownGood` configuration might not be present if it is set to its default value,
the local config deployed with the node. The status will update `lastKnownGood` to the local config deployed with the node. The status will update `lastKnownGood` to
match a valid `assigned` config after the Kubelet becomes comfortable with the config. match a valid `assigned` config after the kubelet becomes comfortable with the config.
The details of how the Kubelet determines a config should become the `lastKnownGood` are The details of how the kubelet determines a config should become the `lastKnownGood` are
not guaranteed by the API, but is currently implemented as a 10-minute grace period. not guaranteed by the API, but is currently implemented as a 10-minute grace period.
You can use the following command (using `jq`) to filter down You can use the following command (using `jq`) to filter down
@@ -254,16 +282,19 @@ The following is an example response:
``` ```
If an error occurs, the Kubelet reports it in the `Node.Status.Config.Error` (if you do not have `jq`, you can look at the whole response and find `Node.Status.Config`
by eye).
If an error occurs, the kubelet reports it in the `Node.Status.Config.Error`
structure. Possible errors are listed in structure. Possible errors are listed in
[Understanding Node.Status.Config.Error messages](#understanding-node-status-config-error-messages). [Understanding Node.Status.Config.Error messages](#understanding-node-status-config-error-messages).
You can search for the identical text in the Kubelet log for additional details You can search for the identical text in the kubelet log for additional details
and context about the error. and context about the error.
#### Make more changes #### Make more changes
Follow the workflow above to make more changes and push them again. Each time Follow the workflow above to make more changes and push them again. Each time
you push a ConfigMap with new contents, the --append-hash kubectl option creates you push a ConfigMap with new contents, the `--append-hash` kubectl option creates
the ConfigMap with a new name. The safest rollout strategy is to first create a the ConfigMap with a new name. The safest rollout strategy is to first create a
new ConfigMap, and then update the Node to use the new ConfigMap. new ConfigMap, and then update the Node to use the new ConfigMap.
@@ -283,7 +314,7 @@ error is reported.
{{% /capture %}} {{% /capture %}}
{{% capture discussion %}} {{% capture discussion %}}
## Kubectl Patch Example ## `kubectl patch` example
You can change a Node's configSource using several different mechanisms. You can change a Node's configSource using several different mechanisms.
This example uses `kubectl patch`: This example uses `kubectl patch`:
@@ -292,25 +323,25 @@ This example uses `kubectl patch`:
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}" kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
``` ```
## Understanding how the Kubelet checkpoints config ## Understanding how the kubelet checkpoints config
When a new config is assigned to the Node, the Kubelet downloads and unpacks the When a new config is assigned to the Node, the kubelet downloads and unpacks the
config payload as a set of files on the local disk. The Kubelet also records metadata config payload as a set of files on the local disk. The kubelet also records metadata
that locally tracks the assigned and last-known-good config sources, so that the that locally tracks the assigned and last-known-good config sources, so that the
Kubelet knows which config to use across restarts, even if the API server becomes kubelet knows which config to use across restarts, even if the API server becomes
unavailable. After checkpointing a config and the relevant metadata, the Kubelet unavailable. After checkpointing a config and the relevant metadata, the kubelet
exits if it detects that the assigned config has changed. When the Kubelet is exits if it detects that the assigned config has changed. When the kubelet is
restarted by the OS-level service manager (such as `systemd`), it reads the new restarted by the OS-level service manager (such as `systemd`), it reads the new
metadata and uses the new config. metadata and uses the new config.
The recorded metadata is fully resolved, meaning that it contains all necessary The recorded metadata is fully resolved, meaning that it contains all necessary
information to choose a specific config version - typically a `UID` and `ResourceVersion`. information to choose a specific config version - typically a `UID` and `ResourceVersion`.
This is in contrast to `Node.Spec.ConfigSource`, where the intended config is declared This is in contrast to `Node.Spec.ConfigSource`, where the intended config is declared
via the idempotent `namespace/name` that identifies the target ConfigMap; the Kubelet via the idempotent `namespace/name` that identifies the target ConfigMap; the kubelet
tries to use the latest version of this ConfigMap. tries to use the latest version of this ConfigMap.
When you are debugging problems on a node, you can inspect the Kubelet's config When you are debugging problems on a node, you can inspect the kubelet's config
metadata and checkpoints. The structure of the Kubelet's checkpointing directory is: metadata and checkpoints. The structure of the kubelet's checkpointing directory is:
```none ```none
- --dynamic-config-dir (root for managing dynamic config) - --dynamic-config-dir (root for managing dynamic config)
@@ -334,13 +365,18 @@ in the Kubelet log for additional details and context about the error.
Error Message | Possible Causes Error Message | Possible Causes
:-------------| :-------------- :-------------| :--------------
failed to load config, see Kubelet log for details | The Kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk. failed to load config, see Kubelet log for details | The kubelet likely could not parse the downloaded config payload, or encountered a filesystem error attempting to load the payload from disk.
failed to validate config, see Kubelet log for details | The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the Kubelet. failed to validate config, see Kubelet log for details | The configuration in the payload, combined with any command-line flag overrides, and the sum of feature gates from flags, the config file, and the remote payload, was determined to be invalid by the kubelet.
invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil | Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the Kubelet is older than the API server and does not recognize a newer source type. invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil | Since Node.Spec.ConfigSource is validated by the API server to contain at least one non-nil subfield, this likely means that the kubelet is older than the API server and does not recognize a newer source type.
failed to sync: failed to download config, see Kubelet log for details | The Kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The Kubelet will retry the download when in this error state. failed to sync: failed to download config, see Kubelet log for details | The kubelet could not download the config. It is possible that Node.Spec.ConfigSource could not be resolved to a concrete API object, or that network errors disrupted the download attempt. The kubelet will retry the download when in this error state.
failed to sync: internal failure, see Kubelet log for details | The Kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache. failed to sync: internal failure, see Kubelet log for details | The kubelet encountered some internal problem and failed to update its config as a result. Examples include filesystem errors and reading objects from the internal informer cache.
internal failure, see Kubelet log for details | The Kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop. internal failure, see Kubelet log for details | The kubelet encountered some internal problem while manipulating config, outside of the configuration sync loop.
{{< /table >}} {{< /table >}}
{{% /capture %}} {{% /capture %}}
{{% capture whatsnext %}}
- For more information on configuring the kubelet via a configuration file, see
[Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file).
- See the reference documentation for [`NodeConfigSource`](https://kubernetes.io/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#nodeconfigsource-v1-core)
{{% /capture %}}