Merge remote-tracking branch 'upstream/main' into dev-1.24

2022-04-11 09:31:28 -07:00
parent ca3737922a 80e2e7d547
commit e65201a5b3
34 changed files with 1273 additions and 1690 deletions
@@ -29,7 +29,7 @@ This page lists some of the available add-ons and links to their respective inst
 * [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking
 * [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
 * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring.
-* [Romana](https://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize).
+* **Romana** is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize).
 * [Weave Net](https://www.weave.works/docs/net/latest/kubernetes/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.

 ## Service Discovery
@@ -27,6 +27,8 @@ API or the `kube-*` components from the upstream code, see the following instruc
  - [Golang](https://golang.org/doc/install) version 1.13+
  - [Docker](https://docs.docker.com/engine/installation/)
  - [etcd](https://github.com/coreos/etcd/)
+  - [make](https://www.gnu.org/software/make/)
+  - [gcc compiler/linker](https://gcc.gnu.org/)

 - Your `GOPATH` environment variable must be set, and the location of `etcd`
  must be in your `PATH` environment variable.
@@ -142,8 +144,9 @@ Run `git status` to see what was generated.
 On branch master
 ...
    modified:   api/openapi-spec/swagger.json
+    modified:   api/openapi-spec/v3/apis__apps__v1_openapi.json
+    modified:   pkg/generated/openapi/zz_generated.openapi.go
    modified:   staging/src/k8s.io/api/apps/v1/generated.proto
-    modified:   staging/src/k8s.io/api/apps/v1/types.go
    modified:   staging/src/k8s.io/api/apps/v1/types_swagger_doc_generated.go
 ```

@@ -778,7 +778,7 @@ This admission controller will deny any Pod that attempts to set certain escalat
 fields, as shown in the
 [Configure a Security Context for a Pod or Container](/docs/tasks/configure-pod-container/security-context/)
 task.
-If you don't use [Pod Security admission]((/docs/concepts/security/pod-security-admission/),
+If you don't use [Pod Security admission](/docs/concepts/security/pod-security-admission/),
 [PodSecurityPolicies](/docs/concepts/security/pod-security-policy/), nor any external enforcement mechanism,
 then you could use this admission controller to restrict the set of values a security context can take.

@@ -109,14 +109,16 @@ IngressSpec describes the Ingress the user wishes to exist.

      - **rules.http.paths.pathType** (string), required

-        PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching is
+        PathType determines the interpretation of the Path matching. PathType can be one of the following enum values: 
+          - `"Exact"` Matches the URL path exactly.
+          - `"Prefix"` Matches based on a URL path prefix split by '/'. Matching is
          done on a path element by element basis. A path element refers is the
          list of labels in the path split by the '/' separator. A request is a
          match for path p if every p is an element-wise prefix of p of the
          request path. Note that if the last element of the path is a substring
          of the last element in request path, it is not a match (e.g. /foo/bar
          matches /foo/bar/baz, but does not match /foo/barbaz).
-        * ImplementationSpecific: Interpretation of the Path matching is up to
+          - `"ImplementationSpecific"` Interpretation of the Path matching is up to
          the IngressClass. Implementations can treat this as a separate PathType
          or treat it identically to Prefix or Exact path types.
        Implementations are required to support all path types.
@@ -23,7 +23,7 @@ Kubernetes requires PKI for the following operations:

 * Client certificates for the kubelet to authenticate to the API server
 * Kubelet [server certificates](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates)
-  for the the API server to talk to the kubelets
+  for the API server to talk to the kubelets
 * Server certificate for the API server endpoint
 * Client certificates for administrators of the cluster to authenticate to the API server
 * Client certificates for the API server to talk to the kubelets
@@ -91,7 +91,7 @@ Required certificates:
 the load balancer stable IP and/or DNS name, `kubernetes`, `kubernetes.default`, `kubernetes.default.svc`,
 `kubernetes.default.svc.cluster`, `kubernetes.default.svc.cluster.local`)

-where `kind` maps to one or more of the [x509 key usage](https://godoc.org/k8s.io/api/certificates/v1beta1#KeyUsage) types:
+where `kind` maps to one or more of the [x509 key usage](https://pkg.go.dev/k8s.io/api/certificates/v1beta1#KeyUsage) types:

 | kind   | Key usage                                                                       |
 |--------|---------------------------------------------------------------------------------|
@@ -24,9 +24,12 @@ you can skip the default CoreDNS deployment and deploy your own variant.
 For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases).
 {{< /note >}}

-<!-- body -->
+{{< note >}}
+To reconfigure a cluster that has already been created see
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
+{{< /note >}}

-{{< feature-state for_k8s_version="v1.12" state="stable" >}}
+<!-- body -->

 ## Customizing the control plane with flags in `ClusterConfiguration`

@@ -161,6 +161,9 @@ To customize control plane components, including optional IPv6 assignment to liv
 for control plane components and etcd server, provide extra arguments to each component as documented in
 [custom arguments](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/).

+To reconfigure a cluster that has already been created see
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
+
 To run `kubeadm init` again, you must first [tear down the cluster](#tear-down).

 If you join a node with a different architecture to your cluster, make sure that your deployed DaemonSets
@@ -173,7 +173,7 @@ The built-in signer is part of [`kube-controller-manager`](/docs/reference/comma

 To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags.

-If you're creating a new cluster, you can use a kubeadm [configuration file](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3):
+If you're creating a new cluster, you can use a kubeadm [configuration file](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3):

 ```yaml
 apiVersion: kubeadm.k8s.io/v1beta3
@@ -0,0 +1,281 @@
+---
+reviewers:
+- sig-cluster-lifecycle
+title: Reconfiguring a kubeadm cluster
+content_type: task
+weight: 10
+---
+
+<!-- overview -->
+
+kubeadm does not support automated ways of reconfiguring components that
+were deployed on managed nodes. One way of automating this would be
+by using a custom [operator](/docs/concepts/extend-kubernetes/operator/).
+
+To modify the components configuration you must manually edit associated cluster
+objects and files on disk.
+
+This guide shows the correct sequence of steps that need to be performed
+to achieve kubeadm cluster reconfiguration.
+
+## {{% heading "prerequisites" %}}
+
+- You need a cluster that was deployed using kubeadm
+- Have administrator credentials (`/etc/kubernetes/admin.conf`) and network connectivity
+to a running kube-apiserver in the cluster from a host that has kubectl installed
+- Have a text editor installed on all hosts
+
+<!-- steps -->
+
+## Reconfiguring the cluster
+
+kubeadm writes a set of cluster wide component configuration options in
+ConfigMaps and other objects. These objects must be manually edited. The command `kubectl edit`
+can be used for that.
+
+The `kubectl edit` command will open a text editor where you can edit and save the object directly.
+
+You can use the environment variables `KUBECONFIG` and `KUBE_EDITOR` to specify the location of
+the kubectl consumed kubeconfig file and preferred text editor.
+
+For example:
+```
+KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit <parameters>
+```
+
+{{< note >}}
+Upon saving any changes to these cluster objects, components running on nodes may not be
+automatically updated. The steps below instruct you on how to perform that manually.
+{{< /note >}}
+
+{{< warning >}}
+Component configuration in ConfigMaps is stored as unstructured data (YAML string).
+This means that validation will not be performed upon updating the contents of a ConfigMap.
+You have to be careful to follow the documented API format for a particular
+component configuration and avoid introducing typos and YAML indentation mistakes.
+{{< /warning >}}
+
+### Applying cluster configuration changes
+
+#### Updating the `ClusterConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`ClusterConfiguration`](/docs/reference/config-api/kubeadm-config.v1beta3/)
+in a ConfigMap called `kubeadm-config` in the `kube-system` namespace.
+
+To change a particular option in the `ClusterConfiguration` you can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kubeadm-config
+```
+
+The configuration is located under the `data.ClusterConfiguration` key.
+
+{{< note >}}
+The `ClusterConfiguration` includes a variety of options that affect the configuration of individual
+components such as kube-apiserver, kube-scheduler, kube-controller-manager, CoreDNS, etcd and kube-proxy.
+Changes to the configuration must be reflected on node components manually.
+{{< /note >}}
+
+#### Reflecting `ClusterConfiguration` changes on control plane nodes
+
+kubeadm manages the control plane components as static Pod manifests located in
+the directory `/etc/kubernetes/manifests`.
+Any changes to the `ClusterConfiguration` under the `apiServer`, `controllerManager`, `scheduler` or `etcd`
+keys must be reflected in the associated files in the manifests directory on a control plane node.
+
+Such changes may include:
+- `extraArgs` - requires updating the list of flags passed to a component container
+- `extraMounts` - requires updated the volume mounts for a component container
+- `*SANs` - requires writing new certificates with updated Subject Alternative Names.
+
+Before proceeding with these changes, make sure you have backed up the directory `/etc/kubernetes/`.
+
+To write new certificates you can use:
+```shell
+kubeadm init phase certs <component-name> --config <config-file>
+```
+
+To write new manifest files in `/etc/kubernetes/manifests` you can use:
+
+```shell
+kubeadm init phase control-plane <component-name> --config <config-file>
+```
+
+The `<config-file>` contents must match the updated `ClusterConfiguration`.
+The `<component-name>` value must be the name of the component.
+
+{{< note >}}
+Updating a file in `/etc/kubernetes/manifests` will tell the kubelet to restart the static Pod for the corresponding component.
+Try doing these changes one node at a time to leave the cluster without downtime.
+{{< /note >}}
+
+### Applying kubelet configuration changes
+
+#### Updating the `KubeletConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/)
+in a ConfigMap called `kubelet-config` in the `kube-system` namespace.
+
+You can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kubelet-config
+```
+
+The configuration is located under the `data.kubelet` key.
+
+#### Reflecting the kubelet changes
+
+To reflect the change on kubeadm nodes you must do the following:
+- Log in to a kubeadm node
+- Run `kubeadm upgrade node phase kubelet-config` to download the latest `kubelet-config`
+ConfigMap contents into the local file `/var/lib/kubelet/config.conf`
+- Edit the file `/var/lib/kubelet/kubeadm-flags.env` to apply additional configuration with
+flags
+- Restart the kubelet service with `systemctl restart kubelet`
+
+{{< note >}}
+Do these changes one node at a time to allow workloads to be rescheduled properly.
+{{< /note >}}
+
+{{< note >}}
+During `kubeadm upgrade`, kubeadm downloads the `KubeletConfiguration` from the
+`kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.conf`.
+This means that node local configuration must be applied either by flags in
+`/var/lib/kubelet/kubeadm-flags.env` or by manually updating the contents of
+`/var/lib/kubelet/config.conf` after `kubeadm upgrade`, and then restarting the kubelet.
+{{< /note >}}
+
+### Applying kube-proxy configuration changes
+
+#### Updating the `KubeProxyConfiguration`
+
+During cluster creation and upgrade, kubeadm writes its
+[`KubeProxyConfiguration`](/docs/reference/config-api/kube-proxy-config.v1alpha1/)
+in a ConfigMap in the `kube-system` namespace called `kube-proxy`.
+
+This ConfigMap is used by the `kube-proxy` DaemonSet in the `kube-system` namespace.
+
+To change a particular option in the `KubeProxyConfiguration`, you can edit the ConfigMap with this command:
+
+```shell
+kubectl edit cm -n kube-system kube-proxy
+```
+
+The configuration is located under the `data.config.conf` key.
+
+#### Reflecting the kube-proxy changes
+
+Once the `kube-proxy` ConfigMap is updated, you can restart all kube-proxy Pods:
+
+Obtain the Pod names:
+
+```shell
+kubectl get po -n kube-system | grep kube-proxy
+```
+
+Delete a Pod with:
+
+```shell
+kubectl delete po -n kube-system <pod-name>
+```
+
+New Pods that use the updated ConfigMap will be created.
+
+{{< note >}}
+Because kubeadm deploys kube-proxy as a DaemonSet, node specific configuration is unsupported.
+{{< /note >}}
+
+### Applying CoreDNS configuration changes
+
+#### Updating the CoreDNS Deployment and Service
+
+kubeadm deploys CoreDNS as a Deployment called `coredns` and with a Service `kube-dns`,
+both in the `kube-system` namespace.
+
+To update any of the CoreDNS settings, you can edit the Deployment and
+Service objects:
+
+```shell
+kubectl edit deployment -n kube-system coredns
+kubectl edit service -n kube-system kube-dns
+```
+
+#### Reflecting the CoreDNS changes
+
+Once the CoreDNS changes are applied you can delete the CoreDNS Pods:
+
+Obtain the Pod names:
+
+```shell
+kubectl get po -n kube-system | grep coredns
+```
+
+Delete a Pod with:
+
+```shell
+kubectl delete po -n kube-system <pod-name>
+```
+
+New Pods with the updated CoreDNS configuration will be created.
+
+{{< note >}}
+kubeadm does not allow CoreDNS configuration during cluster creation and upgrade.
+This means that if you execute `kubeadm upgrade apply`, your changes to the CoreDNS
+objects will be lost and must be reapplied.
+{{< /note >}}
+
+## Persisting the reconfiguration
+
+During the execution of `kubeadm upgrade` on a managed node, kubeadm might overwrite configuration
+that was applied after the cluster was created (reconfiguration).
+
+### Persisting Node object reconfiguration
+
+kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particular
+Kubernetes node. To change any of the contents of this Node object you can use:
+
+```shell
+kubectl edit no <node-name>
+```
+
+During `kubeadm upgrade` the contents of such a Node might get overwritten.
+If you would like to persist your modifications to the Node object after upgrade,
+you can prepare a [kubectl patch](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/)
+and apply it to the Node object:
+
+```shell
+kubectl patch no <node-name> --patch-file <patch-file>
+```
+
+#### Persisting control plane component reconfiguration
+
+The main source of control plane configuration is the `ClusterConfiguration`
+object stored in the cluster. To extend the static Pod manifests configuration,
+[patches](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches) can be used.
+
+These patch files must remain as files on the control plane nodes to ensure that
+they can be used by the `kubeadm upgrade ... --patches <directory>`.
+
+If reconfiguration is done to the `ClusterConfiguration` and static Pod manifests on disk,
+the set of node specific patches must be updated accordingly.
+
+#### Persisting kubelet reconfiguration
+
+Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.conf` will be overwritten on
+`kubeadm upgrade` by downloading the contents of the cluster wide `kubelet-config` ConfigMap.
+To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.conf`
+has to be updated manually post-upgrade or the file `/var/lib/kubelet/kubeadm-flags.env` can include flags.
+The kubelet flags override the associated `KubeletConfiguration` options, but note that
+some of the flags are deprecated.
+
+A kubelet restart will be required after changing `/var/lib/kubelet/config.conf` or
+`/var/lib/kubelet/kubeadm-flags.env`.
+
+{{% heading "whatsnext" %}}
+
+- [Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade)
+- [Customizing components with the kubeadm API](/docs/setup/production-environment/tools/kubeadm/control-plane-flags)
+- [Certificate management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs)
@@ -43,7 +43,12 @@ first drain the node (or nodes) that you are upgrading. In the case of control p
 they could be running CoreDNS Pods or other critical workloads. For more information see
 [Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/).
 - All containers are restarted after upgrade, because the container spec hash value is changed.
- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded, you can execute `systemctl status kubelet`  or view the service logs with `journalctl -xeu kubelet`.
+- To verify that the kubelet service has successfully restarted after the kubelet has been upgraded,
+you can execute `systemctl status kubelet`  or view the service logs with `journalctl -xeu kubelet`.
+- Usage of the `--config` flag of `kubeadm upgrade` with
+[kubeadm configuration API types](/docs/reference/config-api/kubeadm-config.v1beta3)
+with the purpose of reconfiguring the cluster is not recommended and can have unexpected results. Follow the steps in
+[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure) instead.

 <!-- steps -->

@@ -243,7 +243,7 @@ like the `kube-system` namespace, because those pods can gain access to service
 or run with elevated permissions if those service accounts are granted access to permissive
 [PodSecurityPolicies](/docs/concepts/security/pod-security-policy/).

-If you use [Pod Security admission]((/docs/concepts/security/pod-security-admission/) and allow
+If you use [Pod Security admission](/docs/concepts/security/pod-security-admission/) and allow
 any component to create Pods within a namespace that permits privileged Pods, those Pods may
 be able to escape their containers and use this widened access to elevate their privileges.

@@ -20,7 +20,7 @@ the direct installation of host services.

 Administrative tasks such as installation of security patches, event
 log collection, and more can be performed without requiring cluster operators to
-log onto each Window node. HostProcess containers can run as any user that is
+log onto each Windows node. HostProcess containers can run as any user that is
 available on the host or is in the domain of the host machine, allowing administrators
 to restrict resource access through user permissions. While neither filesystem or process
 isolation are supported, a new volume is created on the host upon starting the container
@@ -47,7 +47,7 @@ some extra application configuration. You will need to add the CA certificate
 bundle to the list of CA certificates that the TLS client or server trusts. For
 example, you would do this with a golang TLS config by parsing the certificate
 chain and adding the parsed certificates to the `RootCAs` field in the
-[`tls.Config`](https://godoc.org/crypto/tls#Config) struct.
+[`tls.Config`](https://pkg.go.dev/crypto/tls#Config) struct.

 {{< note >}}
 Even though the custom CA certificate may be included in the filesystem (in the
@@ -152,7 +152,7 @@ sudo yum install -y kubectl

 {{< tabs name="other_kubectl_install" >}}
 {{% tab name="Snap" %}}
-If you are on Ubuntu or another Linux distribution that support [snap](https://snapcraft.io/docs/core/install) package manager, kubectl is available as a [snap](https://snapcraft.io/) application.
+If you are on Ubuntu or another Linux distribution that supports the [snap](https://snapcraft.io/docs/core/install) package manager, kubectl is available as a [snap](https://snapcraft.io/) application.

 ```shell
 snap install kubectl --classic