Merge pull request #28360 from neolit123/1.22-add-v1beta3

kubeadm: use the new v1beta3 instead of v1beta2

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md

@@ -143,8 +143,8 @@ install them selectively.
 {{< tab name="kube-proxy" include="generated/kubeadm_init_phase_addon_kube-proxy.md" />}}
 {{< /tabs >}}
 
-For more details on each field in the `v1beta2` configuration you can navigate to our
+For more details on each field in the `v1beta3` configuration you can navigate to our
-[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2)
+[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3)
 
 ## {{% heading "whatsnext" %}}
 

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

@@ -104,6 +104,10 @@ sudo kubeadm init --skip-phases=control-plane,etcd --config=configfile.yaml
 
 What this example would do is write the manifest files for the control plane and etcd in `/etc/kubernetes/manifests` based on the configuration in `configfile.yaml`. This allows you to modify the files and then skip these phases using `--skip-phases`. By calling the last command you will create a control plane node with the custom manifest files.
 
+{{< feature-state for_k8s_version="v1.22" state="beta" >}}
+
+Alternatively, you can use the `skipPhases` field under `InitConfiguration`.
+
 ### Using kubeadm init with a configuration file {#config-file}
 
 {{< caution >}}

content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md

@@ -66,6 +66,10 @@ For example:
 sudo kubeadm join --skip-phases=preflight --config=config.yaml
 ```
 
+{{< feature-state for_k8s_version="v1.22" state="beta" >}}
+
+Alternatively, you can use the `skipPhases` field in `JoinConfiguration`.
+
 ### Discovering what cluster CA to trust
 
 The kubeadm discovery has several options, each with security tradeoffs.

content/en/docs/setup/production-environment/tools/kubeadm/control-plane-flags.md

@@ -1,79 +1,108 @@
 ---
 reviewers:
 - sig-cluster-lifecycle
-title: Customizing control plane configuration with kubeadm
+title: Customizing components with the kubeadm API
 content_type: concept
 weight: 40
 ---
 
 <!-- overview -->
 
+This page covers how to customize the components that kubeadm deploys. For control plane components
+you can use flags in the `ClusteConfiguration` structure or patches per-node. For the kubelet
+and kube-proxy you can use `KubeletConfiguration` and `KubeProxyConfiguration`, accordingly.
+
+All of these options are possible via the kubeadm configuration API.
+For more details on each field in the configuration you can navigate to our
+[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3).
+
+{{< note >}}
+Customizing the CoreDNS deployment of kubeadm is currently not supported. You must manually
+patch the `kube-system/coredns` {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}}
+and recreate the CoreDNS {{< glossary_tooltip text="Pods" term_id="pod" >}} after that. Alternatively,
+you can skip the default CoreDNS deployment and deploy your own variant.
+For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases).
+{{< /note >}}
+
+<!-- body -->
+
 {{< feature-state for_k8s_version="v1.12" state="stable" >}}
 
-The kubeadm `ClusterConfiguration` object exposes the field `extraArgs` that can override the default flags passed to control plane
+## Customizing the control plane with flags in `ClusterConfiguration`
-components such as the APIServer, ControllerManager and Scheduler. The components are defined using the following fields:
+
+The kubeadm `ClusterConfiguration` object exposes a way for users to override the default
+flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd.
+The components are defined using the following structures:
 
 - `apiServer`
 - `controllerManager`
 - `scheduler`
+- `etcd`
 
-The `extraArgs` field consist of `key: value` pairs. To override a flag for a control plane component:
+These structures contain a common `extraArgs` field, that consists of `key: value` pairs.
+To override a flag for a control plane component:
 
-1.  Add the appropriate fields to your configuration.
+1.  Add the appropriate `extraArgs` to your configuration.
-2.  Add the flags to override to the field.
+2.  Add flags to the `extraArgs` field.
 3.  Run `kubeadm init` with `--config <YOUR CONFIG YAML>`.
 
-For more details on each field in the configuration you can navigate to our
-[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#ClusterConfiguration).
-
 {{< note >}}
-You can generate a `ClusterConfiguration` object with default values by running `kubeadm config print init-defaults` and saving the output to a file of your choice.
+You can generate a `ClusterConfiguration` object with default values by running `kubeadm config print init-defaults`
+and saving the output to a file of your choice.
 {{< /note >}}
 
+{{< note >}}
+The `ClusterConfiguration` object is currently global in kubeadm clusters. This means that any flags that you add,
+will apply to all instances of the same component on different nodes. To apply individual configuration per component
+on different nodes you can use [patches](#patches).
+{{< /note >}}
 
+{{< note >}}
+Duplicate flags (keys), or passing the same flag `--foo` multiple times, is currently not supported.
+To workaround that you must use [patches](#patches).
+{{< /note >}}
 
-<!-- body -->
+### APIServer flags
-
-## APIServer flags
 
 For details, see the [reference documentation for kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/).
 
 Example usage:
+
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 kubernetesVersion: v1.16.0
 apiServer:
   extraArgs:
-    advertise-address: 192.168.0.103
     anonymous-auth: "false"
     enable-admission-plugins: AlwaysPullImages,DefaultStorageClass
     audit-log-path: /home/johndoe/audit.log
 ```
 
-## ControllerManager flags
+### ControllerManager flags
 
 For details, see the [reference documentation for kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/).
 
 Example usage:
+
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 kubernetesVersion: v1.16.0
 controllerManager:
   extraArgs:
     cluster-signing-key-file: /home/johndoe/keys/ca.key
-    bind-address: 0.0.0.0
     deployment-controller-sync-period: "50"
 ```
 
-## Scheduler flags
+### Scheduler flags
 
 For details, see the [reference documentation for kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/).
 
 Example usage:
+
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 kubernetesVersion: v1.16.0
 scheduler:
@@ -87,4 +116,95 @@ scheduler:
       pathType: "File"
 ```
 
+### Etcd flags
 
+For details, see the [etcd server documentation](https://etcd.io/docs/).
+
+Example usage:
+
+```yaml
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: ClusterConfiguration
+etcd:
+  local:
+    extraArgs:
+      election-timeout: 1000
+```
+
+## Customizing the control plane with patches {#patches}
+
+{{< feature-state for_k8s_version="v1.22" state="beta" >}}
+
+Kubeadm allows you to pass a directory with patch files to `InitConfiguration` and `JoinConfiguration`
+on individual nodes. These patches can be used as the last customization step before the control
+plane component manifests are written to disk.
+
+You can pass this file to `kubeadm init` with `--config <YOUR CONFIG YAML>`:
+
+```yaml
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: InitConfiguration
+nodeRegistration:
+  patches:
+    directory: /home/user/somedir
+```
+
+{{< note >}}
+For `kubeadm init` you can pass a file containing both a `ClusterConfiguration` and `InitConfiguration`
+separated by `---`.
+{{< /note >}}
+
+You can pass this file to `kubeadm join` with `--config <YOUR CONFIG YAML>`:
+
+```yaml
+apiVersion: kubeadm.k8s.io/v1beta3
+kind: JoinConfiguration
+nodeRegistration:
+  patches:
+    directory: /home/user/somedir
+```
+
+The directory must contain files named `target[suffix][+patchtype].extension`.
+For example, `kube-apiserver0+merge.yaml` or just `etcd.json`.
+
+- `target` can be one of `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `etcd`.
+- `patchtype` can be one of `strategic`, `merge` or `json` and these must match the patching formats
+[supported by kubectl](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch).
+The default `patchtype` is `strategic`.
+- `extension` must be either `json` or `yaml`.
+- `suffix` is an optional string that can be used to determine which patches are applied first
+alpha-numerically.
+
+{{< note >}}
+If you are using `kubeadm upgrade` to upgrade your kubeadm nodes you must again provide the same
+patches, so that the customization is preserved after upgrade. To do that you can use the `--patches`
+flag, which must point to the same directory. `kubeadm upgrade` currently does not support a configuration
+API structure that can be used for the same purpose.
+{{< /note >}}
+
+## Customizing the kubelet
+
+To customize the kubelet you can add a `KubeletConfiguration` next to the `ClusterConfiguration` or
+`InitConfiguration` separated by `---` within the same configuration file. This file can then be passed to `kubeadm init`.
+
+{{< note >}}
+kubeadm applies the same `KubeletConfiguration` to all nodes in the cluster. To apply node
+specific settings you can use kubelet flags as overrides by passing them in the `nodeRegistration.kubeletExtraArgs`
+field supported by both `InitConfiguration` and `JoinConfiguration`. Some kubelet flags are deprecated,
+so check their status in the [kubelet reference documentation](/docs/reference/command-line-tools-reference/kubelet)
+before using them.
+{{< /note >}}
+
+For more details see [Configuring each kubelet in your cluster using kubeadm](/docs/setup/production-environment/tools/kubeadm/kubelet-integration)
+
+## Customizing kube-proxy
+
+To customize kube-proxy you can pass a `KubeProxyConfiguration` next your `ClusterConfiguration` or
+`InitConfiguration` to `kubeadm init` separated by `---`.
+
+For more details you can navigate to our [API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3).
+
+{{< note >}}
+kubeadm deploys kube-proxy as a {{< glossary_tooltip text="DaemonSet" term_id="daemonset" >}}, which means
+that the `KubeProxyConfiguration` would apply to all instances of kube-proxy in the cluster.
+{{< /note >}}

content/en/docs/setup/production-environment/tools/kubeadm/dual-stack-support.md

@@ -21,7 +21,7 @@ For each server that you want to use as a {{< glossary_tooltip text="node" term_
 
 You need to have an IPv4 and and IPv6 address range to use. Cluster operators typically
 use private address ranges for IPv4. For IPv6, a cluster operator typically chooses a global
-unicast address block from within `2000::/3`, using a range that is assigned to the operator.  
+unicast address block from within `2000::/3`, using a range that is assigned to the operator.
 You don't have to route the cluster's IP address ranges to the public internet.
 
 The size of the IP address allocations should be suitable for the number of Pods and
@@ -30,7 +30,7 @@ Services that you are planning to run.
 {{< note >}}
 If you are upgrading an existing cluster then, by default, the `kubeadm upgrade` command
 changes the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-`IPv6DualStack` to `true` if that is not already enabled.  
+`IPv6DualStack` to `true` if that is not already enabled.
 However, `kubeadm` does not support making modifications to the pod IP address range
 (“cluster CIDR”) nor to the cluster's Service address range (“Service CIDR”).
 {{< /note >}}
@@ -45,11 +45,11 @@ similar to the following example:
 kubeadm init --pod-network-cidr=10.244.0.0/16,2001:db8:42:0::/56 --service-cidr=10.96.0.0/16,2001:db8:42:1::/112
 ```
 
-To make things clearer, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for the primary dual-stack control plane node.
+To make things clearer, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for the primary dual-stack control plane node.
 
 ```yaml
 ---
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 featureGates:
   IPv6DualStack: true
@@ -57,7 +57,7 @@ networking:
   podSubnet: 10.244.0.0/16,2001:db8:42:0::/56
   serviceSubnet: 10.96.0.0/16,2001:db8:42:1::/112
 ---
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: InitConfiguration
 localAPIEndpoint:
   advertiseAddress: "10.100.0.1"
@@ -85,10 +85,10 @@ The `--apiserver-advertise-address` flag does not support dual-stack.
 
 Before joining a node, make sure that the node has IPv6 routable network interface and allows IPv6 forwarding.
 
-Here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for joining a worker node to the cluster.
+Here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for joining a worker node to the cluster.
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: JoinConfiguration
 discovery:
   bootstrapToken:
@@ -98,9 +98,9 @@ nodeRegistration:
     node-ip: 10.100.0.3,fd00:1:2:3::3
 ```
 
-Also, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for joining another control plane node to the cluster.
+Also, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for joining another control plane node to the cluster.
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: JoinConfiguration
 controlPlane:
   localAPIEndpoint:
@@ -124,7 +124,7 @@ kubeadm join --config=kubeadm-config.yaml ...
 ### Create a single-stack cluster
 
 {{< note >}}
-Enabling the dual-stack feature doesn't mean that you need to use dual-stack addressing.  
+Enabling the dual-stack feature doesn't mean that you need to use dual-stack addressing.
 You can deploy a single-stack cluster that has the dual-stack networking feature enabled.
 {{< /note >}}
 
@@ -134,10 +134,10 @@ In 1.21 the `IPv6DualStack` feature is Beta and the feature gate is defaulted to
 kubeadm init --feature-gates IPv6DualStack=false
 ```
 
-To make things more clear, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) `kubeadm-config.yaml` for the single-stack control plane node.
+To make things more clear, here is an example kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3) `kubeadm-config.yaml` for the single-stack control plane node.
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 featureGates:
   IPv6DualStack: false

content/en/docs/setup/production-environment/tools/kubeadm/high-availability.md

@@ -115,7 +115,7 @@ option. Your cluster requirements may need a different configuration.
 
     {{< note >}}
     The `kubeadm init` flags `--config` and `--certificate-key` cannot be mixed, therefore if you want
-    to use the [kubeadm configuration](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2)
+    to use the [kubeadm configuration](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3)
     you must add the `certificateKey` field in the appropriate config locations
     (under `InitConfiguration` and `JoinConfiguration: controlPlane`).
     {{< /note >}}
@@ -230,7 +230,7 @@ in the kubeadm config file.
 
 1.  Create a file called `kubeadm-config.yaml` with the following contents:
 
-        apiVersion: kubeadm.k8s.io/v1beta2
+        apiVersion: kubeadm.k8s.io/v1beta3
         kind: ClusterConfiguration
         kubernetesVersion: stable
         controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"

content/en/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm.md

@@ -95,7 +95,7 @@ this example.
     HOST=${ETCDHOSTS[$i]}
     NAME=${NAMES[$i]}
     cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
-    apiVersion: "kubeadm.k8s.io/v1beta2"
+    apiVersion: "kubeadm.k8s.io/v1beta3"
     kind: ClusterConfiguration
     etcd:
         local:

content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md

@@ -367,23 +367,6 @@ kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec":
 
 The tracking issue for this problem is [here](https://github.com/kubernetes/kubeadm/issues/1027).
 
-## The NodeRegistration.Taints field is omitted when marshalling kubeadm configuration
-
-*Note: This [issue](https://github.com/kubernetes/kubeadm/issues/1358) only applies to tools that marshal kubeadm types (e.g. to a YAML configuration file). It will be fixed in kubeadm API v1beta2.*
-
-By default, kubeadm applies the `node-role.kubernetes.io/master:NoSchedule` taint to control-plane nodes.
-If you prefer kubeadm to not taint the control-plane node, and set `InitConfiguration.NodeRegistration.Taints` to an empty slice,
-the field will be omitted when marshalling. When the field is omitted, kubeadm applies the default taint.
-
-There are at least two workarounds:
-
-1. Use the `node-role.kubernetes.io/master:PreferNoSchedule` taint instead of an empty slice. [Pods will get scheduled on masters](/docs/concepts/scheduling-eviction/taint-and-toleration/), unless other nodes have capacity.
-
-2. Remove the taint after kubeadm init exits:
-```bash
-kubectl taint nodes NODE_NAME node-role.kubernetes.io/master:NoSchedule-
-```
-
 ## `/usr` is mounted read-only on nodes {#usr-mounted-read-only}
 
 On Linux distributions such as Fedora CoreOS or Flatcar Container Linux, the directory `/usr` is mounted as a read-only filesystem.
@@ -393,19 +376,19 @@ Kubernetes components like the kubelet and kube-controller-manager use the defau
 for the feature to work.
 
 To workaround this issue you can configure the flex-volume directory using the kubeadm
-[configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2).
+[configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3).
 
 On the primary control-plane Node (created using `kubeadm init`) pass the following
 file using `--config`:
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: InitConfiguration
 nodeRegistration:
   kubeletExtraArgs:
     volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
 ---
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 controllerManager:
   extraArgs:
@@ -415,7 +398,7 @@ controllerManager:
 On joining Nodes:
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: JoinConfiguration
 nodeRegistration:
   kubeletExtraArgs:

content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md

@@ -42,7 +42,7 @@ A minimal example of configuring the field explicitly:
 ```yaml
 # kubeadm-config.yaml
 kind: ClusterConfiguration
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kubernetesVersion: v1.21.0
 ---
 kind: KubeletConfiguration

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md

@@ -161,10 +161,10 @@ The built-in signer is part of [`kube-controller-manager`](/docs/reference/comma
 
 To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags.
 
-If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2):
+If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3):
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 controllerManager:
   extraArgs:
@@ -223,7 +223,7 @@ To configure the kubelets in a new kubeadm cluster to obtain properly signed ser
 certificates you must pass the following minimal configuration to `kubeadm init`:
 
 ```yaml
-apiVersion: kubeadm.k8s.io/v1beta2
+apiVersion: kubeadm.k8s.io/v1beta3
 kind: ClusterConfiguration
 ---
 apiVersion: kubelet.config.k8s.io/v1beta1