diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md index 2b6939bac6..ed17375ade 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md @@ -143,8 +143,8 @@ install them selectively. {{< tab name="kube-proxy" include="generated/kubeadm_init_phase_addon_kube-proxy.md" />}} {{< /tabs >}} -For more details on each field in the `v1beta2` configuration you can navigate to our -[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) +For more details on each field in the `v1beta3` configuration you can navigate to our +[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index 2260c2ca22..e787d2f1d2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -104,6 +104,10 @@ sudo kubeadm init --skip-phases=control-plane,etcd --config=configfile.yaml What this example would do is write the manifest files for the control plane and etcd in `/etc/kubernetes/manifests` based on the configuration in `configfile.yaml`. This allows you to modify the files and then skip these phases using `--skip-phases`. By calling the last command you will create a control plane node with the custom manifest files. +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + +Alternatively, you can use the `skipPhases` field under `InitConfiguration`. + ### Using kubeadm init with a configuration file {#config-file} {{< caution >}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md index 53ca4a789b..248832c6b3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md @@ -66,6 +66,10 @@ For example: sudo kubeadm join --skip-phases=preflight --config=config.yaml ``` +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + +Alternatively, you can use the `skipPhases` field in `JoinConfiguration`. + ### Discovering what cluster CA to trust The kubeadm discovery has several options, each with security tradeoffs. diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md b/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md index 8a9828a7ad..b5a2831426 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md @@ -1,79 +1,108 @@ --- reviewers: - sig-cluster-lifecycle -title: Customizing control plane configuration with kubeadm +title: Customizing components with the kubeadm API content_type: concept weight: 40 --- +This page covers how to customize the components that kubeadm deploys. For control plane components +you can use flags in the `ClusteConfiguration` structure or patches per-node. For the kubelet +and kube-proxy you can use `KubeletConfiguration` and `KubeProxyConfiguration`, accordingly. + +All of these options are possible via the kubeadm configuration API. +For more details on each field in the configuration you can navigate to our +[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3). + +{{< note >}} +Customizing the CoreDNS deployment of kubeadm is currently not supported. You must manually +patch the `kube-system/coredns` {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}} +and recreate the CoreDNS {{< glossary_tooltip text="Pods" term_id="pod" >}} after that. Alternatively, +you can skip the default CoreDNS deployment and deploy your own variant. +For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases). +{{< /note >}} + + + {{< feature-state for_k8s_version="v1.12" state="stable" >}} -The kubeadm `ClusterConfiguration` object exposes the field `extraArgs` that can override the default flags passed to control plane -components such as the APIServer, ControllerManager and Scheduler. The components are defined using the following fields: +## Customizing the control plane with flags in `ClusterConfiguration` + +The kubeadm `ClusterConfiguration` object exposes a way for users to override the default +flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd. +The components are defined using the following structures: - `apiServer` - `controllerManager` - `scheduler` +- `etcd` -The `extraArgs` field consist of `key: value` pairs. To override a flag for a control plane component: +These structures contain a common `extraArgs` field, that consists of `key: value` pairs. +To override a flag for a control plane component: -1. Add the appropriate fields to your configuration. -2. Add the flags to override to the field. +1. Add the appropriate `extraArgs` to your configuration. +2. Add flags to the `extraArgs` field. 3. Run `kubeadm init` with `--config `. -For more details on each field in the configuration you can navigate to our -[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#ClusterConfiguration). - {{< note >}} -You can generate a `ClusterConfiguration` object with default values by running `kubeadm config print init-defaults` and saving the output to a file of your choice. +You can generate a `ClusterConfiguration` object with default values by running `kubeadm config print init-defaults` +and saving the output to a file of your choice. {{< /note >}} +{{< note >}} +The `ClusterConfiguration` object is currently global in kubeadm clusters. This means that any flags that you add, +will apply to all instances of the same component on different nodes. To apply individual configuration per component +on different nodes you can use [patches](#patches). +{{< /note >}} +{{< note >}} +Duplicate flags (keys), or passing the same flag `--foo` multiple times, is currently not supported. +To workaround that you must use [patches](#patches). +{{< /note >}} - - -## APIServer flags +### APIServer flags For details, see the [reference documentation for kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/). Example usage: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 apiServer: extraArgs: - advertise-address: 192.168.0.103 anonymous-auth: "false" enable-admission-plugins: AlwaysPullImages,DefaultStorageClass audit-log-path: /home/johndoe/audit.log ``` -## ControllerManager flags +### ControllerManager flags For details, see the [reference documentation for kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/). Example usage: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 controllerManager: extraArgs: cluster-signing-key-file: /home/johndoe/keys/ca.key - bind-address: 0.0.0.0 deployment-controller-sync-period: "50" ``` -## Scheduler flags +### Scheduler flags For details, see the [reference documentation for kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/). Example usage: + ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.16.0 scheduler: @@ -87,4 +116,95 @@ scheduler: pathType: "File" ``` +### Etcd flags +For details, see the [etcd server documentation](https://etcd.io/docs/). + +Example usage: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: ClusterConfiguration +etcd: + local: + extraArgs: + election-timeout: 1000 +``` + +## Customizing the control plane with patches {#patches} + +{{< feature-state for_k8s_version="v1.22" state="beta" >}} + +Kubeadm allows you to pass a directory with patch files to `InitConfiguration` and `JoinConfiguration` +on individual nodes. These patches can be used as the last customization step before the control +plane component manifests are written to disk. + +You can pass this file to `kubeadm init` with `--config `: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: InitConfiguration +nodeRegistration: + patches: + directory: /home/user/somedir +``` + +{{< note >}} +For `kubeadm init` you can pass a file containing both a `ClusterConfiguration` and `InitConfiguration` +separated by `---`. +{{< /note >}} + +You can pass this file to `kubeadm join` with `--config `: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta3 +kind: JoinConfiguration +nodeRegistration: + patches: + directory: /home/user/somedir +``` + +The directory must contain files named `target[suffix][+patchtype].extension`. +For example, `kube-apiserver0+merge.yaml` or just `etcd.json`. + +- `target` can be one of `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `etcd`. +- `patchtype` can be one of `strategic`, `merge` or `json` and these must match the patching formats +[supported by kubectl](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch). +The default `patchtype` is `strategic`. +- `extension` must be either `json` or `yaml`. +- `suffix` is an optional string that can be used to determine which patches are applied first +alpha-numerically. + +{{< note >}} +If you are using `kubeadm upgrade` to upgrade your kubeadm nodes you must again provide the same +patches, so that the customization is preserved after upgrade. To do that you can use the `--patches` +flag, which must point to the same directory. `kubeadm upgrade` currently does not support a configuration +API structure that can be used for the same purpose. +{{< /note >}} + +## Customizing the kubelet + +To customize the kubelet you can add a `KubeletConfiguration` next to the `ClusterConfiguration` or +`InitConfiguration` separated by `---` within the same configuration file. This file can then be passed to `kubeadm init`. + +{{< note >}} +kubeadm applies the same `KubeletConfiguration` to all nodes in the cluster. To apply node +specific settings you can use kubelet flags as overrides by passing them in the `nodeRegistration.kubeletExtraArgs` +field supported by both `InitConfiguration` and `JoinConfiguration`. Some kubelet flags are deprecated, +so check their status in the [kubelet reference documentation](/docs/reference/command-line-tools-reference/kubelet) +before using them. +{{< /note >}} + +For more details see [Configuring each kubelet in your cluster using kubeadm](/docs/setup/production-environment/tools/kubeadm/kubelet-integration) + +## Customizing kube-proxy + +To customize kube-proxy you can pass a `KubeProxyConfiguration` next your `ClusterConfiguration` or +`InitConfiguration` to `kubeadm init` separated by `---`. + +For more details you can navigate to our [API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3). + +{{< note >}} +kubeadm deploys kube-proxy as a {{< glossary_tooltip text="DaemonSet" term_id="daemonset" >}}, which means +that the `KubeProxyConfiguration` would apply to all instances of kube-proxy in the cluster. +{{< /note >}} diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md b/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md index 767787179c..70f86d5af0 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md @@ -21,7 +21,7 @@ For each server that you want to use as a {{< glossary_tooltip text="node" term_ You need to have an IPv4 and and IPv6 address range to use. Cluster operators typically use private address ranges for IPv4. For IPv6, a cluster operator typically chooses a global -unicast address block from within `2000::/3`, using a range that is assigned to the operator. +unicast address block from within `2000::/3`, using a range that is assigned to the operator. You don't have to route the cluster's IP address ranges to the public internet. The size of the IP address allocations should be suitable for the number of Pods and @@ -30,7 +30,7 @@ Services that you are planning to run. {{< note >}} If you are upgrading an existing cluster then, by default, the `kubeadm upgrade` command changes the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -`IPv6DualStack` to `true` if that is not already enabled. +`IPv6DualStack` to `true` if that is not already enabled. However, `kubeadm` does not support making modifications to the pod IP address range (“cluster CIDR”) nor to the cluster's Service address range (“Service CIDR”). {{< /note >}} @@ -45,11 +45,11 @@ similar to the following example: kubeadm init --pod-network-cidr=10.244.0.0/16,2001:db8:42:0::/56 --service-cidr=10.96.0.0/16,2001:db8:42:1::/112 ``` -To make things clearer, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for the primary dual-stack control plane node. +To make things clearer, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for the primary dual-stack control plane node. ```yaml --- -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration featureGates: IPv6DualStack: true @@ -57,7 +57,7 @@ networking: podSubnet: 10.244.0.0/16,2001:db8:42:0::/56 serviceSubnet: 10.96.0.0/16,2001:db8:42:1::/112 --- -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: "10.100.0.1" @@ -85,10 +85,10 @@ The `--apiserver-advertise-address` flag does not support dual-stack. Before joining a node, make sure that the node has IPv6 routable network interface and allows IPv6 forwarding. -Here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for joining a worker node to the cluster. +Here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for joining a worker node to the cluster. ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: bootstrapToken: @@ -98,9 +98,9 @@ nodeRegistration: node-ip: 10.100.0.3,fd00:1:2:3::3 ``` -Also, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for joining another control plane node to the cluster. +Also, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for joining another control plane node to the cluster. ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration controlPlane: localAPIEndpoint: @@ -124,7 +124,7 @@ kubeadm join --config=kubeadm-config.yaml ... ### Create a single-stack cluster {{< note >}} -Enabling the dual-stack feature doesn't mean that you need to use dual-stack addressing. +Enabling the dual-stack feature doesn't mean that you need to use dual-stack addressing. You can deploy a single-stack cluster that has the dual-stack networking feature enabled. {{< /note >}} @@ -134,10 +134,10 @@ In 1.21 the `IPv6DualStack` feature is Beta and the feature gate is defaulted to kubeadm init --feature-gates IPv6DualStack=false ``` -To make things more clear, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for the single-stack control plane node. +To make things more clear, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for the single-stack control plane node. ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration featureGates: IPv6DualStack: false diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md b/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md index e387e2d41c..1fed8ce77a 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md @@ -115,7 +115,7 @@ option. Your cluster requirements may need a different configuration. {{< note >}} The `kubeadm init` flags `--config` and `--certificate-key` cannot be mixed, therefore if you want - to use the [kubeadm configuration](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) + to use the [kubeadm configuration](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) you must add the `certificateKey` field in the appropriate config locations (under `InitConfiguration` and `JoinConfiguration: controlPlane`). {{< /note >}} @@ -230,7 +230,7 @@ in the kubeadm config file. 1. Create a file called `kubeadm-config.yaml` with the following contents: - apiVersion: kubeadm.k8s.io/v1beta2 + apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: stable controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm.md index 9e95a69f47..0319ee17fc 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm.md @@ -95,7 +95,7 @@ this example. HOST=${ETCDHOSTS[$i]} NAME=${NAMES[$i]} cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml - apiVersion: "kubeadm.k8s.io/v1beta2" + apiVersion: "kubeadm.k8s.io/v1beta3" kind: ClusterConfiguration etcd: local: diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 85f32a4f02..edaa061b16 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -367,23 +367,6 @@ kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": The tracking issue for this problem is [here](https://github.com/kubernetes/kubeadm/issues/1027). -## The NodeRegistration.Taints field is omitted when marshalling kubeadm configuration - -*Note: This [issue](https://github.com/kubernetes/kubeadm/issues/1358) only applies to tools that marshal kubeadm types (e.g. to a YAML configuration file). It will be fixed in kubeadm API v1beta2.* - -By default, kubeadm applies the `node-role.kubernetes.io/master:NoSchedule` taint to control-plane nodes. -If you prefer kubeadm to not taint the control-plane node, and set `InitConfiguration.NodeRegistration.Taints` to an empty slice, -the field will be omitted when marshalling. When the field is omitted, kubeadm applies the default taint. - -There are at least two workarounds: - -1. Use the `node-role.kubernetes.io/master:PreferNoSchedule` taint instead of an empty slice. [Pods will get scheduled on masters](/docs/concepts/scheduling-eviction/taint-and-toleration/), unless other nodes have capacity. - -2. Remove the taint after kubeadm init exits: -```bash -kubectl taint nodes NODE_NAME node-role.kubernetes.io/master:NoSchedule- -``` - ## `/usr` is mounted read-only on nodes {#usr-mounted-read-only} On Linux distributions such as Fedora CoreOS or Flatcar Container Linux, the directory `/usr` is mounted as a read-only filesystem. @@ -393,19 +376,19 @@ Kubernetes components like the kubelet and kube-controller-manager use the defau for the feature to work. To workaround this issue you can configure the flex-volume directory using the kubeadm -[configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2). +[configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3). On the primary control-plane Node (created using `kubeadm init`) pass the following file using `--config`: ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" --- -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration controllerManager: extraArgs: @@ -415,7 +398,7 @@ controllerManager: On joining Nodes: ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index 28fdc60ced..31a9ff0e33 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -42,7 +42,7 @@ A minimal example of configuring the field explicitly: ```yaml # kubeadm-config.yaml kind: ClusterConfiguration -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kubernetesVersion: v1.21.0 --- kind: KubeletConfiguration diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index be98558659..556c30a197 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -161,10 +161,10 @@ The built-in signer is part of [`kube-controller-manager`](/docs/reference/comma To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags. -If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2): +If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3): ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration controllerManager: extraArgs: @@ -223,7 +223,7 @@ To configure the kubelets in a new kubeadm cluster to obtain properly signed ser certificates you must pass the following minimal configuration to `kubeadm init`: ```yaml -apiVersion: kubeadm.k8s.io/v1beta2 +apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration --- apiVersion: kubelet.config.k8s.io/v1beta1