Merge master into dev-1.22 to keep in sync

2021-05-20 19:17:30 +02:00
parent 09a6795d12 d398b75e19
commit 5cb4f973a4
290 changed files with 14292 additions and 10271 deletions
@@ -47,6 +47,15 @@ and other ecosystem groups to ensure a smooth transition and will evaluate thing
 as the situation evolves.


+### Can I still use dockershim after it is removed from Kubernetes?
+
+Update:
+Mirantis and Docker have [committed][mirantis] to maintaining the dockershim after
+it is removed from Kubernetes.
+
+[mirantis]: https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/
+
+
 ### Will my existing Docker images still work?

 Yes, the images produced from `docker build` will work with all CRI implementations.
@@ -178,4 +187,3 @@ discussion of the changes.

 Always and whenever you want!  🤗🤗

-
@@ -0,0 +1,268 @@
+---
+layout: blog
+title: 'Using Finalizers to Control Deletion'
+date: 2021-05-14
+slug: using-finalizers-to-control-deletion
+---
+
+**Authors:** Aaron Alpar (Kasten)
+
+Deleting objects in Kubernetes can be challenging. You may think you’ve deleted something, only to find it still persists. While issuing a `kubectl delete` command and hoping for the best might work for day-to-day operations, understanding how Kubernetes `delete` commands operate will help you understand why some objects linger after deletion. 
+
+In this post, I’ll look at:
+
+- What properties of a resource govern deletion
+- How finalizers and owner references impact object deletion
+- How the propagation policy can be used to change the order of deletions
+- How deletion works, with examples
+
+For simplicity, all examples will use ConfigMaps and basic shell commands to demonstrate the process. We’ll explore how the commands work and discuss repercussions and results from using them in practice.
+
+## The basic `delete`
+
+Kubernetes has several different commands you can use that allow you to create, read, update, and delete objects. For the purpose of this blog post, we’ll focus on four `kubectl` commands: `create`, `get`, `patch`, and `delete`. 
+
+Here are examples of the basic `kubectl delete` command:
+
+```
+kubectl create configmap mymap
+configmap/mymap created
+```
+
+```
+kubectl get configmap/mymap
+NAME    DATA   AGE
+mymap   0      12s
+```
+
+```
+kubectl delete configmap/mymap
+configmap "mymap" deleted
+```
+
+```
+kubectl get configmap/mymap
+Error from server (NotFound): configmaps "mymap" not found
+```
+
+Shell commands preceded by `$` are followed by their output. You can see that we begin with a `kubectl create configmap mymap`, which will create the empty configmap `mymap`. Next, we need to `get` the configmap to prove it exists. We can then delete that configmap. Attempting to `get` it again produces an HTTP 404 error, which means the configmap is not found.
+
+The state diagram for the basic `delete` command is very simple:
+
+
+{{<figure width="495" src="/images/blog/2021-05-14-using-finalizers-to-control-deletion/state-diagram-delete.png" caption="State diagram for delete">}}
+
+Although this operation is straightforward, other factors may interfere with the deletion, including finalizers and owner references.
+
+## Understanding Finalizers
+
+When it comes to understanding resource deletion in Kubernetes, knowledge of how finalizers work is helpful and can help you understand why some objects don’t get deleted. 
+
+Finalizers are keys on resources that signal pre-delete operations. They control the garbage collection on resources, and are designed to alert controllers what cleanup operations to perform prior to removing a resource. However, they don’t necessarily name code that should be executed; finalizers on resources are basically just lists of keys much like annotations. Like annotations, they can be manipulated.
+
+Some common finalizers you’ve likely encountered are:
+
+- `kubernetes.io/pv-protection`
+- `kubernetes.io/pvc-protection`
+
+The finalizers above are used on volumes to prevent accidental deletion. Similarly, some finalizers can be used to prevent deletion of any resource but are not managed by any controller.
+
+Below with a custom configmap, which has no properties but contains a finalizer:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: mymap
+  finalizers:
+  - kubernetes
+EOF
+```
+
+The configmap resource controller doesn't understand what to do with the `kubernetes` finalizer key. I term these “dead” finalizers for configmaps as it is normally used on namespaces. Here’s what happen upon attempting to delete the configmap:
+
+```
+kubectl delete configmap/mymap &
+configmap "mymap" deleted
+jobs
+[1]+  Running kubectl delete configmap/mymap
+```
+
+Kubernetes will report back that the object has been deleted, however, it hasn’t been deleted in a traditional sense. Rather, it’s in the process of deletion. When we attempt to `get` that object again, we discover the object has been modified to include the deletion timestamp. 
+
+```
+kubectl get configmap/mymap -o yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  creationTimestamp: "2020-10-22T21:30:18Z"
+  deletionGracePeriodSeconds: 0
+  deletionTimestamp: "2020-10-22T21:30:34Z"
+  finalizers:
+  - kubernetes
+  name: mymap
+  namespace: default
+  resourceVersion: "311456"
+  selfLink: /api/v1/namespaces/default/configmaps/mymap
+  uid: 93a37fed-23e3-45e8-b6ee-b2521db81638
+```
+
+In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and put it into a read-only state. The deletion timestamp signals that the object can only be read, with the exception of removing the finalizer key updates. In other words, the deletion will not be complete until we edit the object and remove the finalizer.
+
+Here's a demonstration of using the `patch` command to remove finalizers. If we want to delete an object, we can simply patch it on the command line to remove the finalizers. In this way, the deletion that was running in the background will complete and the object will be deleted. When we attempt to `get` that configmap, it will be gone. 
+
+```
+kubectl patch configmap/mymap \
+    --type json \
+    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
+configmap/mymap patched
+[1]+  Done  kubectl delete configmap/mymap
+
+kubectl get configmap/mymap -o yaml
+Error from server (NotFound): configmaps "mymap" not found
+```
+
+Here's a state diagram for finalization: 
+
+{{<figure width="617" src="/images/blog/2021-05-14-using-finalizers-to-control-deletion/state-diagram-finalize.png" caption="State diagram for finalize">}}
+
+So, if you attempt to delete an object that has a finalizer on it, it will remain in finalization until the controller has removed the finalizer keys or the finalizers are removed using Kubectl. Once that finalizer list is empty, the object can actually be reclaimed by Kubernetes and put into a queue to be deleted from the registry.
+
+## Owner References
+
+Owner references describe how groups of objects are related. They are properties on resources that specify the relationship to one another, so entire trees of resources can be deleted.
+
+Finalizer rules are processed when there are owner references. An owner reference consists of a name and a UID. Owner references link resources within the same namespace, and it also needs a UID for that reference to work. Pods typically have owner references to the owning replica set. So, when deployments or stateful sets are deleted, then the child replica sets and pods are deleted in the process. 
+
+Here are some examples of owner references and how they work. In the first example, we create a parent object first, then the child. The result is a very simple configmap that contains an owner reference to its parent:
+
+```
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: mymap-parent
+EOF
+CM_UID=$(kubectl get configmap mymap-parent -o jsonpath="{.metadata.uid}")
+
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: mymap-child
+  ownerReferences:
+  - apiVersion: v1
+    kind: ConfigMap
+    name: mymap-parent
+    uid: $CM_UID
+EOF
+```
+
+Deleting the child object when an owner reference is involved does not delete the parent:
+
+```
+kubectl get configmap
+NAME           DATA   AGE
+mymap-child    0      12m4s
+mymap-parent   0      12m4s
+
+kubectl delete configmap/mymap-child
+configmap "mymap-child" deleted
+
+kubectl get configmap
+NAME           DATA   AGE
+mymap-parent   0      12m10s
+```
+
+In this example, we re-created the parent-child configmaps from above. Now, when deleting from the parent (instead of the child) with an owner reference from the child to the parent, when we `get` the configmaps, none are in the namespace:
+
+```
+kubectl get configmap
+NAME           DATA   AGE
+mymap-child    0      10m2s
+mymap-parent   0      10m2s
+
+kubectl delete configmap/mymap-parent
+configmap "mymap-parent" deleted
+
+kubectl get configmap
+No resources found in default namespace.
+```
+
+To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=false option for `kubectl delete` to delete an object and orphan its children. 
+
+In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=false, the parent is deleted but the child still exists:
+
+```
+kubectl get configmap
+NAME           DATA   AGE
+mymap-child    0      13m8s
+mymap-parent   0      13m8s
+
+kubectl delete --cascade=false configmap/mymap-parent
+configmap "mymap-parent" deleted
+
+kubectl get configmap
+NAME          DATA   AGE
+mymap-child   0      13m21s
+```
+
+The --cascade option links to the propagation policy in the API, which allows you to change the order in which objects are deleted within a tree. In the following example uses API access to craft a custom delete API call with the background propagation policy:
+
+```
+kubectl proxy --port=8080 &
+Starting to serve on 127.0.0.1:8080
+
+curl -X DELETE \
+  localhost:8080/api/v1/namespaces/default/configmaps/mymap-parent \
+  -d '{ "kind":"DeleteOptions", "apiVersion":"v1", "propagationPolicy":"Background" }' \
+  -H "Content-Type: application/json"
+{
+  "kind": "Status",
+  "apiVersion": "v1",
+  "metadata": {},
+  "status": "Success",
+  "details": { ... }
+}
+```
+
+Note that the propagation policy cannot be specified on the command line using kubectl. You have to specify it using a custom API call. Simply create a proxy, so you have access to the API server from the client, and execute a `curl` command with just a URL to execute that `delete` command. 
+
+There are three different options for the propagation policy:
+
+- `Foreground`: Children are deleted before the parent (post-order)
+- `Background`: Parent is deleted before the children (pre-order)
+- `Orphan`: Owner references are ignored
+
+Keep in mind that when you delete an object and owner references have been specified, finalizers will be honored in the process. This can result in trees of objects persisting, and you end up with a partial deletion. At that point, you have to look at any existing owner references on your objects, as well as any finalizers, to understand what’s happening. 
+
+## Forcing a Deletion of a Namespace
+
+There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, `finalize`. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
+
+```
+cat <<EOF | curl -X PUT \
+  localhost:8080/api/v1/namespaces/test/finalize \
+  -H "Content-Type: application/json" \
+  --data-binary @-
+{
+  "kind": "Namespace",
+  "apiVersion": "v1",
+  "metadata": {
+    "name": "test"
+  },
+  "spec": {
+    "finalizers": null
+  }
+}
+EOF
+```
+
+This should be done with caution as it may delete the namespace only and leave orphan objects within the, now non-exiting, namespace - a confusing state for Kubernetes. If this happens, the namespace can be re-created manually and sometimes the orphaned objects will re-appear under the just-created namespace which will allow manual cleanup and recovery.
+
+## Key Takeaways
+
+As these examples demonstrate, finalizers can get in the way of deleting resources in Kubernetes, especially when there are parent-child relationships between objects. Often, there is a reason for adding a finalizer into the code, so you should always investigate before manually deleting it. Owner references allow you to specify and remove trees of resources, although finalizers will be honored in the process. Finally, the propagation policy can be used to specify the order of deletion via a custom API call, giving you control over how objects are deleted. Now that you know a little more about how deletions work in Kubernetes, we recommend you try it out on your own, using a test cluster. 
+
+{{< youtube class="youtube-quote-sm" id="F7-ZxWwf4sY" title="Clean Up Your Room! What Does It Mean to Delete Something in K8s">}}
@@ -24,7 +24,8 @@ cid: community
 <a href="#videos">Videos</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <a href="#discuss">Discussions</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <a href="#events">Events and meetups</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
-<a href="#news">News</a>
+<a href="#news">News</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="/releases">Releases</a>

 </div>
 <br class="mobile"><br class="mobile">
@@ -23,7 +23,7 @@ This page lists some of the available add-ons and links to their respective inst
 * [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave.
 * [Contiv](https://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](https://github.com/contiv). The [installer](https://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options.
 * [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
-* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md) is an overlay network provider that can be used with Kubernetes.
+* [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes.
 * [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod.
 * [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
 * [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) is a networking provider for Kubernetes based on [OVN (Open Virtual Network)](https://github.com/ovn-org/ovn/), a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy.
@@ -328,13 +328,13 @@ kubectl create secret tls my-tls-secret \
  --key=path/to/key/file
 ```

-The public/private key pair must exist before hand. The public key certificate
+The public/private key pair must exist beforehand. The public key certificate
 for `--cert` must be .PEM encoded (Base64-encoded DER format), and match the
 given private key for `--key`.
 The private key must be in what is commonly called PEM private key format,
 unencrypted. In both cases, the initial and the last lines from PEM (for
 example, `--------BEGIN CERTIFICATE-----` and `-------END CERTIFICATE----` for
-a cetificate) are *not* included.
+a certificate) are *not* included.

 ### Bootstrap token Secrets

@@ -25,7 +25,7 @@ This page provides an outline of the container image concept.

 Container images are usually given a name such as `pause`, `example/mycontainer`, or `kube-apiserver`.
 Images can also include a registry hostname; for example: `fictional.registry.example/imagename`,
-and possible a port number as well; for example: `fictional.registry.example:10443/imagename`.
+and possibly a port number as well; for example: `fictional.registry.example:10443/imagename`.

 If you don't specify a registry hostname, Kubernetes assumes that you mean the Docker public registry.

@@ -113,11 +113,13 @@ Operator.

 {{% thirdparty-content %}}

+* [Charmed Operator Framework](https://juju.is/)
 * [kubebuilder](https://book.kubebuilder.io/)
 * [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator)
 * [Metacontroller](https://metacontroller.app/) along with WebHooks that
  you implement yourself
 * [Operator Framework](https://operatorframework.io)
+* [shell-operator](https://github.com/flant/shell-operator)

 ## {{% heading "whatsnext" %}}

@@ -21,7 +21,7 @@ This page is an overview of Kubernetes.
 <!-- body -->
 Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

-The name Kubernetes originates from Greek, meaning helmsman or pilot. Google open-sourced the Kubernetes project in 2014. Kubernetes combines [over 15 years of Google's experience](/blog/2015/04/borg-predecessor-to-kubernetes/) running production workloads at scale with best-of-breed ideas and practices from the community.
+The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation results from counting the eight letters between the "K" and the "s". Google open-sourced the Kubernetes project in 2014. Kubernetes combines [over 15 years of Google's experience](/blog/2015/04/borg-predecessor-to-kubernetes/) running production workloads at scale with best-of-breed ideas and practices from the community.

 ## Going back in time

@@ -39,7 +39,8 @@ on every resource object.
 | `app.kubernetes.io/version`         | The current version of the application (e.g., a semantic version, revision hash, etc.) | `5.7.21` | string |
 | `app.kubernetes.io/component`       | The component within the architecture | `database` | string |
 | `app.kubernetes.io/part-of`         | The name of a higher level application this one is part of | `wordpress` | string |
-| `app.kubernetes.io/managed-by`  | The tool being used to manage the operation of an application | `helm` | string |
+| `app.kubernetes.io/managed-by`      | The tool being used to manage the operation of an application | `helm` | string |
+| `app.kubernetes.io/created-by`      | The controller/user who created this resource | `controller-manager` | string |

 To illustrate these labels in action, consider the following StatefulSet object:

@@ -54,6 +55,7 @@ metadata:
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: wordpress
    app.kubernetes.io/managed-by: helm
+    app.kubernetes.io/created-by: controller-manager
 ```

 ## Applications And Instances Of Applications
@@ -170,4 +172,3 @@ metadata:

 With the MySQL `StatefulSet` and `Service` you'll notice information about both MySQL and WordPress, the broader application, are included.

-
@@ -39,7 +39,7 @@ Creation and deletion of namespaces are described in the
 [Admin Guide documentation for namespaces](/docs/tasks/administer-cluster/namespaces).

 {{< note >}}
-    Avoid creating namespace with prefix `kube-`, since it is reserved for Kubernetes system namespaces.
+    Avoid creating namespaces with the prefix `kube-`, since it is reserved for Kubernetes system namespaces.
 {{< /note >}}

 ### Viewing namespaces
@@ -1,8 +1,11 @@
 ---
-title: "Scheduling and Eviction"
+title: "Scheduling, Preemption and Eviction"
 weight: 90
 description: >
-  In Kubernetes, scheduling refers to making sure that Pods are matched to Nodes so that the kubelet can run them.
-  Eviction is the process of proactively failing one or more Pods on resource-starved Nodes.
+  In Kubernetes, scheduling refers to making sure that Pods are matched to Nodes
+  so that the kubelet can run them. Preemption is the process of terminating 
+  Pods with lower Priority so that Pods with higher Priority can schedule on 
+  Nodes. Eviction is the process of proactively terminating one or more Pods on
+  resource-starved Nodes.
 ---

@@ -113,7 +113,7 @@ enforced/disallowed:
 			</td>
 		</tr>
 		<tr>
-			<td>AppArmor <em>(optional)</em></td>
+			<td>AppArmor</td>
 			<td>
 				On supported hosts, the 'runtime/default' AppArmor profile is applied by default.
 				The baseline policy should prevent overriding or disabling the default AppArmor
@@ -124,14 +124,26 @@ enforced/disallowed:
 			</td>
 		</tr>
 		<tr>
-			<td>SELinux <em>(optional)</em></td>
+			<td>SELinux</td>
 			<td>
-				Setting custom SELinux options should be disallowed.<br>
+				Setting the SELinux type is restricted, and setting a custom SELinux user or role option is forbidden.<br>
 				<br><b>Restricted Fields:</b><br>
-				spec.securityContext.seLinuxOptions<br>
-				spec.containers[*].securityContext.seLinuxOptions<br>
-				spec.initContainers[*].securityContext.seLinuxOptions<br>
-				<br><b>Allowed Values:</b> undefined/nil<br>
+				spec.securityContext.seLinuxOptions.type<br>
+				spec.containers[*].securityContext.seLinuxOptions.type<br>
+				spec.initContainers[*].securityContext.seLinuxOptions.type<br>
+				<br><b>Allowed Values:</b><br>
+				undefined/empty<br>
+				container_t<br>
+				container_init_t<br>
+				container_kvm_t<br>
+				<br><b>Restricted Fields:</b><br>
+				spec.securityContext.seLinuxOptions.user<br>
+				spec.containers[*].securityContext.seLinuxOptions.user<br>
+				spec.initContainers[*].securityContext.seLinuxOptions.user<br>
+				spec.securityContext.seLinuxOptions.role<br>
+				spec.containers[*].securityContext.seLinuxOptions.role<br>
+				spec.initContainers[*].securityContext.seLinuxOptions.role<br>
+				<br><b>Allowed Values:</b> undefined/empty<br>
 			</td>
 		</tr>
 		<tr>
@@ -540,11 +540,11 @@ spec:

 ### Access Modes

-Claims use the same conventions as volumes when requesting storage with specific access modes.
+Claims use [the same conventions as volumes](#access-modes) when requesting storage with specific access modes.

 ### Volume Modes

-Claims use the same convention as volumes to indicate the consumption of the volume as either a filesystem or block device.
+Claims use [the same convention as volumes](#volume-mode) to indicate the consumption of the volume as either a filesystem or block device.

 ### Resources

@@ -154,9 +154,9 @@ the class or PV. If a mount option is invalid, the PV mount fails.
 ### Volume Binding Mode

 The `volumeBindingMode` field controls when [volume binding and dynamic
-provisioning](/docs/concepts/storage/persistent-volumes/#provisioning) should occur.
+provisioning](/docs/concepts/storage/persistent-volumes/#provisioning) should occur. When unset, "Immediate" mode is used by default.

-By default, the `Immediate` mode indicates that volume binding and dynamic
+The `Immediate` mode indicates that volume binding and dynamic
 provisioning occurs once the PersistentVolumeClaim is created. For storage
 backends that are topology-constrained and not globally accessible from all Nodes
 in the cluster, PersistentVolumes will be bound or provisioned without knowledge of the Pod's scheduling
@@ -188,6 +188,36 @@ The following plugins support `WaitForFirstConsumer` with pre-created Persistent
 and pre-created PVs, but you'll need to look at the documentation for a specific CSI driver
 to see its supported topology keys and examples.

+{{< note >}}
+   If you choose to use `waitForFirstConsumer`, do not use `nodeName` in the Pod spec
+   to specify node affinity. If `nodeName` is used in this case, the scheduler will be bypassed and PVC will remain in `pending` state.
+
+   Instead, you can use node selector for hostname in this case as shown below.
+{{< /note >}}
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: task-pv-pod
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: kube-01
+  volumes:
+    - name: task-pv-storage
+      persistentVolumeClaim:
+        claimName: task-pv-claim
+  containers:
+    - name: task-pv-container
+      image: nginx
+      ports:
+        - containerPort: 80
+          name: "http-server"
+      volumeMounts:
+        - mountPath: "/usr/share/nginx/html"
+          name: task-pv-storage
+```
+
 ### Allowed Topologies

 When a cluster operator specifies the `WaitForFirstConsumer` volume binding mode, it is no longer necessary
@@ -80,12 +80,14 @@ Here are some ways to mitigate involuntary disruptions:
  [multi-zone cluster](/docs/setup/multiple-zones).)

 The frequency of voluntary disruptions varies.  On a basic Kubernetes cluster, there are
-no voluntary disruptions at all.  However, your cluster administrator or hosting provider
+no automated voluntary disruptions (only user-triggered ones).  However, your cluster administrator or hosting provider
 may run some additional services which cause voluntary disruptions. For example,
 rolling out node software updates can cause voluntary disruptions. Also, some implementations
 of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes.
 Your cluster administrator or hosting provider should have documented what level of voluntary
-disruptions, if any, to expect.
+disruptions, if any, to expect. Certain configuration options, such as
+[using PriorityClasses](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/)
+in your pod spec can also cause voluntary (and involuntary) disruptions.


 ## Pod disruption budgets
@@ -246,7 +246,7 @@ myapp-pod   1/1       Running   0          9m
 ```

 This simple example should provide some inspiration for you to create your own
-init containers. [What's next](#whats-next) contains a link to a more detailed example.
+init containers. [What's next](#what-s-next) contains a link to a more detailed example.

 ## Detailed behavior

@@ -326,7 +326,6 @@ Kubernetes, consult the documentation for the version you are using.

 ## {{% heading "whatsnext" %}}

-
 * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container)
 * Learn how to [debug init containers](/docs/tasks/debug-application-cluster/debug-init-containers/)

@@ -13,7 +13,7 @@ obsolete -->
 You can use _topology spread constraints_ to control how {{< glossary_tooltip text="Pods" term_id="Pod" >}} are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.

 {{< note >}}
-In versions of Kubernetes before v1.19, you must enable the `EvenPodsSpread`
+In versions of Kubernetes before v1.18, you must enable the `EvenPodsSpread`
 [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on
 the [API server](/docs/concepts/overview/components/#kube-apiserver) and the
 [scheduler](/docs/reference/generated/kube-scheduler/) in order to use Pod
@@ -90,8 +90,8 @@ This section shows how to generate the
 For example:

 ```shell
-export K8S_WEBROOT=$(GOPATH)/src/github.com/<your-username>/website
-export K8S_ROOT=$(GOPATH)/src/k8s.io/kubernetes
+export K8S_WEBROOT=${GOPATH}/src/github.com/<your-username>/website
+export K8S_ROOT=${GOPATH}/src/k8s.io/kubernetes
 export K8S_RELEASE=1.17.0
 ```

@@ -16,18 +16,21 @@ card:
 This page shows you how to [localize](https://blog.mozilla.org/l10n/2011/12/14/i18n-vs-l10n-whats-the-diff/) the docs for a different language.


-
 <!-- body -->

-## Getting started
+## Contribute to an existing localization

-Because contributors can't approve their own pull requests, you need at least two contributors to begin a localization.
+You can help add or improve content to an existing localization. In [Kubernetes Slack](https://slack.k8s.io/) you'll find a channel for each localization. There is also a general [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations) where you can say hello.

-All localization teams must be self-sustaining with their own resources. The Kubernetes website is happy to host your work, but it's up to you to translate it.
+{{< note >}}
+If you want to work on a localization that already exists, check
+this page in that localization (if it exists), rather than the
+English original. You might see extra details there.
+{{< /note >}}

 ### Find your two-letter language code

-First, consult the [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) to find your localization's two-letter country code. For example, the two-letter code for Korean is `ko`.
+First, consult the [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) to find your localization's two-letter language code. For example, the two-letter code for Korean is `ko`.

 ### Fork and clone the repo

@@ -40,13 +43,54 @@ git clone https://github.com/<username>/website
 cd website
 ```

-### Open a pull request
+The website content directory includes sub-directories for each language. The localization you want to help out with is inside `content/<two-letter-code>`.

-Next, [open a pull request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) to add a localization to the `kubernetes/website` repository.
+### Suggest changes

-The PR must include all of the [minimum required content](#minimum-required-content) before it can be approved.
+Create or update your chosen localized page based on the English original. See
+[translating content](#translating-content) for more details.

-For an example of adding a new localization, see the PR to enable [docs in French](https://github.com/kubernetes/website/pull/12548).
+If you notice a technical inaccuracy or other problem with the upstream (English)
+documentation, you should fix the upstream documentation first and then repeat the
+equivalent fix by updating the localization you're working on.
+
+Please limit pull requests to a single localization, since pull requests that change
+content in multiple localizations could be difficult to review.
+
+Follow [Suggesting Content Improvements](/docs/contribute/suggest-improvements/) to propose changes to
+that localization. The process is very similar to proposing changes to the upstream (English) content.
+
+## Start a new localization
+
+If you want the Kubernetes documentation localized into a new language, here's what
+you need to do.
+
+Because contributors can't approve their own pull requests, you need _at least two contributors_
+to begin a localization.
+
+All localization teams must be self-sustaining. The Kubernetes website is happy to host your work, but
+it's up to you to translate it and keep existing localized content current.
+
+You'll need to know the two-letter language code for your language. Consult the
+[ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) to find your
+localization's two-letter language code. For example, the two-letter code for Korean is
+`ko`.
+
+When you start a new localization, you must localize all the
+[minimum required content](#minimum-required-content) before
+the Kubernetes project can publish your changes to the live
+website.
+
+SIG Docs can help you work on a separate branch so that you
+can incrementally work towards that goal.
+
+### Find community
+
+Let Kubernetes SIG Docs know you're interested in creating a localization! Join the [SIG Docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) and the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). Other localization teams are happy to help you get started and answer any questions you have.
+
+Please also consider participating in the [SIG Docs Localization Subgroup meeting](https://github.com/kubernetes/community/tree/master/sig-docs).  The mission of the SIG Docs localization subgroup is to work across the SIG Docs localization teams to collaborate on defining and documenting the processes for creating localized contribution guides. In addition, the SIG Docs localization subgroup will look for opportunities for the creation and sharing of common tools across localization teams and also serve to identify new requirements to the SIG Docs Leadership team.  If you have questions about this meeting, please inquire on the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations).
+
+You can also create a Slack channel for your localization in the `kubernetes/community` repository. For an example of adding a Slack channel, see the PR for [adding a channel for Persian](https://github.com/kubernetes/community/pull/4980).

 ### Join the Kubernetes GitHub organization

@@ -70,15 +114,6 @@ Next, add a GitHub label for your localization in the `kubernetes/test-infra` re

 For an example of adding a label, see the PR for adding the [Italian language label](https://github.com/kubernetes/test-infra/pull/11316).

-### Find community
-
-Let Kubernetes SIG Docs know you're interested in creating a localization! Join the [SIG Docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) and the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations). Other localization teams are happy to help you get started and answer any questions you have.
-
-Please also consider participating in the [SIG Docs Localization Subgroup meeting](https://github.com/kubernetes/community/tree/master/sig-docs).  The mission of the SIG Docs localization subgroup is to work across the SIG Docs localization teams to collaborate on defining and documenting the processes for creating localized contribution guides. In addition, the SIG Docs localization subgroup will look for opportunities for the creation and sharing of common tools across localization teams and also serve to identify new requirements to the SIG Docs Leadership team.  If you have questions about this meeting, please inquire on the [SIG Docs Localizations Slack channel](https://kubernetes.slack.com/messages/sig-docs-localizations).
-
-You can also create a Slack channel for your localization in the `kubernetes/community` repository. For an example of adding a Slack channel, see the PR for [adding a channel for Persian](https://github.com/kubernetes/community/pull/4980).
-
-## Minimum required content

 ### Modify the site configuration

@@ -107,20 +142,20 @@ Add a language-specific subdirectory to the [`content`](https://github.com/kuber
 mkdir content/de
 ```

+You also need to create a directory inside `data/i18n` for
+[localized strings](#site-strings-in-i18n); look at existing localizations
+for an example. To use these new strings, you must also create a symbolic link
+from `i18n/<localization>.toml` to the actual string configuration in
+`data/i18n/<localization>/<localization>.toml` (remember to commit the symbolic
+link).
+
+For example, for German the strings live in `data/i18n/de/de.toml`, and
+`i18n/de.toml` is a symbolic link to `data/i18n/de/de.toml`.
+
 ### Localize the community code of conduct

 Open a PR against the [`cncf/foundation`](https://github.com/cncf/foundation/tree/master/code-of-conduct-languages) repository to add the code of conduct in your language.

-### Add a localized README file
-
-To guide other localization contributors, add a new [`README-**.md`](https://help.github.com/articles/about-readmes/) to the top level of k/website, where `**` is the two-letter language code. For example, a German README file would be `README-de.md`.
-
-Provide guidance to localization contributors in the localized `README-**.md` file. Include the same information contained in `README.md` as well as:
-
- A point of contact for the localization project
- Any information specific to the localization
-
-After you create the localized README, add a link to the file from the main English `README.md`, and include contact information in English. You can provide a GitHub ID, email address, [Slack channel](https://slack.com/), or other method of contact. You must also provide a link to your localized Community Code of Conduct.

 ### Setting up the OWNERS files

@@ -174,10 +209,38 @@ For each team, add the list of GitHub users requested in [Add your localization
     - remyleone
 ```

+### Open a pull request
+
+Next, [open a pull request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) to add a localization to the `kubernetes/website` repository.
+
+The PR must include all of the [minimum required content](#minimum-required-content) before it can be approved.
+
+For an example of adding a new localization, see the PR to enable [docs in French](https://github.com/kubernetes/website/pull/12548).
+
+### Add a localized README file
+
+To guide other localization contributors, add a new [`README-**.md`](https://help.github.com/articles/about-readmes/) to the top level of k/website, where `**` is the two-letter language code. For example, a German README file would be `README-de.md`.
+
+Provide guidance to localization contributors in the localized `README-**.md` file. Include the same information contained in `README.md` as well as:
+
+- A point of contact for the localization project
+- Any information specific to the localization
+
+After you create the localized README, add a link to the file from the main English `README.md`, and include contact information in English. You can provide a GitHub ID, email address, [Slack channel](https://slack.com/), or other method of contact. You must also provide a link to your localized Community Code of Conduct.
+
+### Launching your new localization
+
+Once a localization meets requirements for workflow and minimum output, SIG Docs will:
+
+- Enable language selection on the website
+- Publicize the localization's availability through [Cloud Native Computing Foundation](https://www.cncf.io/about/) (CNCF) channels, including the [Kubernetes blog](https://kubernetes.io/blog/).
+
 ## Translating content

 Localizing *all* of the Kubernetes documentation is an enormous task. It's okay to start small and expand over time.

+### Minimum required content
+
 At a minimum, all localizations must include:

 Description | URLs
@@ -185,7 +248,7 @@ Description | URLs
 Home | [All heading and subheading URLs](/docs/home/)
 Setup | [All heading and subheading URLs](/docs/setup/)
 Tutorials | [Kubernetes Basics](/docs/tutorials/kubernetes-basics/), [Hello Minikube](/docs/tutorials/hello-minikube/)
-Site strings | [All site strings in a new localized TOML file](https://github.com/kubernetes/website/tree/master/i18n)
+Site strings | [All site strings](#Site-strings-in-i18n) in a new localized TOML file

 Translated documents must reside in their own `content/**/` subdirectory, but otherwise follow the same URL path as the English source. For example, to prepare the [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) tutorial for translation into German, create a subfolder under the `content/de/` folder and copy the English source:

@@ -213,27 +276,30 @@ To find source files for your target version:
 2. Select a branch for your target version from the following table:
    Target version | Branch
    -----|-----
-    Next version | [`dev-{{< skew nextMinorVersion >}}`](https://github.com/kubernetes/website/tree/dev-{{< skew nextMinorVersion >}})
    Latest version | [`master`](https://github.com/kubernetes/website/tree/master)
-    Previous version | `release-*.**`
+    Previous version | [`release-{{< skew prevMinorVersion >}}`](https://github.com/kubernetes/website/tree/release-{{< skew prevMinorVersion >}})
+    Next version | [`dev-{{< skew nextMinorVersion >}}`](https://github.com/kubernetes/website/tree/dev-{{< skew nextMinorVersion >}})

-The `master` branch holds content for the current release `{{< latest-version >}}`. The release team will create `{{< release-branch >}}` branch shortly before the next release: v{{< skew nextMinorVersion >}}.
+The `master` branch holds content for the current release `{{< latest-version >}}`. The release team will create a `{{< release-branch >}}` branch before the next release: v{{< skew nextMinorVersion >}}.

 ### Site strings in i18n

-Localizations must include the contents of [`i18n/en.toml`](https://github.com/kubernetes/website/blob/master/i18n/en.toml) in a new language-specific file. Using German as an example: `i18n/de.toml`.
+Localizations must include the contents of [`data/i18n/en/en.toml`](https://github.com/kubernetes/website/blob/master/data/i18n/en/en.toml) in a new language-specific file. Using German as an example: `data/i18n/de/de.toml`.

-Add a new localization file to `i18n/`. For example, with German (`de`):
+Add a new localization directory and file to `data/i18n/`. For example, with German (`de`):

-```shell
-cp i18n/en.toml i18n/de.toml
+```bash
+mkdir -p data/i18n/de
+cp data/i18n/en/en.toml data/i18n/de/de.toml
 ```

-Then translate the value of each string:
+Revise the comments at the top of the file to suit your localization,
+then translate the value of each string. For example, this is the German-language
+placeholder text for the search form:

-```TOML
-[docs_label_i_am]
-other = "ICH BIN..."
+```toml
+[ui_search_placeholder]
+other = "Suchen"
 ```

 Localizing site strings lets you customize site-wide text and features: for example, the legal copyright text in the footer on each page.
@@ -244,7 +310,9 @@ Some language teams have their own language-specific style guide and glossary. F

 ## Branching strategy

-Because localization projects are highly collaborative efforts, we encourage teams to work in shared localization branches.
+Because localization projects are highly collaborative efforts, we
+encourage teams to work in shared localization branches - especially
+when starting out and the localization is not yet live.

 To collaborate on a localization branch:

@@ -288,16 +356,4 @@ For more information about working from forks or directly from the repository, s

 SIG Docs welcomes upstream contributions and corrections to the English source.

-## Help an existing localization

-You can also help add or improve content to an existing localization. Join the [Slack channel](https://kubernetes.slack.com/messages/C1J0BPD2M/) for the localization, and start opening PRs to help. Please limit pull requests to a single localization since pull requests that change content in multiple localizations could be difficult to review.
-
-
-
-## {{% heading "whatsnext" %}}
-
-
-Once a localization meets requirements for workflow and minimum output, SIG docs will:
-
- Enable language selection on the website
- Publicize the localization's availability through [Cloud Native Computing Foundation](https://www.cncf.io/about/) (CNCF) channels, including the [Kubernetes blog](https://kubernetes.io/blog/).
@@ -132,7 +132,7 @@ different Kubernetes components.
 | `IPv6DualStack` | `true` | Beta | 1.21 | |
 | `KubeletCredentialProviders` | `false` | Alpha | 1.20 | |
 | `LegacyNodeRoleBehavior` | `false` | Alpha | 1.16 | 1.18 |
-| `LegacyNodeRoleBehavior` | `true` | Beta | 1.19 |  |
+| `LegacyNodeRoleBehavior` | `true` | Beta | 1.19 | 1.20 |
 | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 |
 | `LocalStorageCapacityIsolation` | `true` | Beta | 1.10 | |
 | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | |
@@ -142,7 +142,7 @@ different Kubernetes components.
 | `NamespaceDefaultLabelName` | `true` | Beta | 1.21 | |
 | `NetworkPolicyEndPort` | `false` | Alpha | 1.21 |  |
 | `NodeDisruptionExclusion` | `false` | Alpha | 1.16 | 1.18 |
-| `NodeDisruptionExclusion` | `true` | Beta | 1.19 | |
+| `NodeDisruptionExclusion` | `true` | Beta | 1.19 | 1.20 |
 | `NonPreemptingPriority` | `false` | Alpha | 1.15 | 1.18 |
 | `NonPreemptingPriority` | `true` | Beta | 1.19 | |
 | `PodDeletionCost` | `false` | Alpha | 1.21 | |
@@ -162,7 +162,7 @@ different Kubernetes components.
 | `ServiceLBNodePortControl` | `false` | Alpha | 1.20 | |
 | `ServiceLoadBalancerClass` | `false` | Alpha | 1.21 | |
 | `ServiceNodeExclusion` | `false` | Alpha | 1.8 | 1.18 |
-| `ServiceNodeExclusion` | `true` | Beta | 1.19 | |
+| `ServiceNodeExclusion` | `true` | Beta | 1.19 | 1.20 |
 | `ServiceTopology` | `false` | Alpha | 1.17 | |
 | `SetHostnameAsFQDN` | `false` | Alpha | 1.19 | 1.19 |
 | `SetHostnameAsFQDN` | `true` | Beta | 1.20 | |
@@ -171,7 +171,8 @@ different Kubernetes components.
 | `StorageVersionHash` | `false` | Alpha | 1.14 | 1.14 |
 | `StorageVersionHash` | `true` | Beta | 1.15 | |
 | `SuspendJob` | `false` | Alpha | 1.21 | |
-| `TTLAfterFinished` | `false` | Alpha | 1.12 | |
+| `TTLAfterFinished` | `false` | Alpha | 1.12 | 1.20 |
+| `TTLAfterFinished` | `true` | Beta | 1.21 | |
 | `TopologyAwareHints` | `false` | Alpha | 1.21 | |
 | `TopologyManager` | `false` | Alpha | 1.16 | 1.17 |
 | `TopologyManager` | `true` | Beta | 1.18 | |
@@ -264,6 +265,7 @@ different Kubernetes components.
 | `EvenPodsSpread` | `true` | Beta | 1.18 | 1.18 |
 | `EvenPodsSpread` | `true` | GA | 1.19 | - |
 | `ExecProbeTimeout` | `true` | GA | 1.20 | - |
+| `ExternalPolicyForExternalIP` | `true` | GA | 1.18 | - |
 | `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | 1.12 |
 | `GCERegionalPersistentDisk` | `true` | GA | 1.13 | - |
 | `HugePages` | `false` | Alpha | 1.8 | 1.9 |
@@ -284,11 +286,13 @@ different Kubernetes components.
 | `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 |
 | `KubeletPodResources` | `true` | Beta | 1.15 | |
 | `KubeletPodResources` | `true` | GA | 1.20 | |
+| `LegacyNodeRoleBehavior` | `false` | GA | 1.21 | - |
 | `MountContainers` | `false` | Alpha | 1.9 | 1.16 |
 | `MountContainers` | `false` | Deprecated | 1.17 | - |
 | `MountPropagation` | `false` | Alpha | 1.8 | 1.9 |
 | `MountPropagation` | `true` | Beta | 1.10 | 1.11 |
 | `MountPropagation` | `true` | GA | 1.12 | - |
+| `NodeDisruptionExclusion` | `true` | GA | 1.21 | - |
 | `NodeLease` | `false` | Alpha | 1.12 | 1.13 |
 | `NodeLease` | `true` | Beta | 1.14 | 1.16 |
 | `NodeLease` | `true` | GA | 1.17 | - |
@@ -342,6 +346,7 @@ different Kubernetes components.
 | `ServiceLoadBalancerFinalizer` | `false` | Alpha | 1.15 | 1.15 |
 | `ServiceLoadBalancerFinalizer` | `true` | Beta | 1.16 | 1.16 |
 | `ServiceLoadBalancerFinalizer` | `true` | GA | 1.17 | - |
+| `ServiceNodeExclusion` | `true` | GA | 1.21 | - |
 | `StartupProbe` | `false` | Alpha | 1.16 | 1.17 |
 | `StartupProbe` | `true` | Beta | 1.18 | 1.19 |
 | `StartupProbe` | `true` | GA | 1.20 | - |
@@ -636,6 +641,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
  host mounts, or containers that are privileged or using specific non-namespaced
  capabilities (e.g. `MKNODE`, `SYS_MODULE` etc.). This should only be enabled
  if user namespace remapping is enabled in the Docker daemon.
+- `ExternalPolicyForExternalIP`: Fix a bug where ExternalTrafficPolicy is not applied to Service ExternalIPs.
 - `GCERegionalPersistentDisk`: Enable the regional PD feature on GCE.
 - `GenericEphemeralVolume`: Enables ephemeral, inline volumes that support all features
  of normal volumes (can be provided by third-party storage vendors, storage capacity tracking,
@@ -216,6 +216,10 @@ kubectl get nodes -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'

 # Produce a period-delimited tree of all keys returned for pods, etc
 kubectl get pods -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'
+
+# Produce ENV for all pods, assuming you have a default container for the pods, default namespace and the `env` command is supported.
+# Helpful when running any supported command across all pods, not just `env`
+for pod in $(kubectl get po --output=jsonpath={.items..metadata.name}); do echo $pod && kubectl exec -it $pod env; done
 ```

 ## Updating resources
@@ -265,7 +265,7 @@ nginx-app    1/1     1            1           2m
 ```

 ```shell
-kubectl get po -l run=nginx-app
+kubectl get po -l app=nginx-app
 ```
 ```
 NAME                         READY     STATUS    RESTARTS   AGE
@@ -279,7 +279,7 @@ deployment "nginx-app" deleted
 ```

 ```shell
-kubectl get po -l run=nginx-app
+kubectl get po -l app=nginx-app
 # Return nothing
 ```

@@ -96,7 +96,7 @@ ServiceSpec describes the attributes that a user creates on a service.

  - **ports.nodePort** (int32)

-    The port on each node on which this service is exposed when type is NodePort or LoadBalancer.  Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail.  If not specified, a port will be allocated if this Service requires one.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
+    The port on each node on which this service is exposed when type is NodePort or LoadBalancer.  Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail.  If not specified, a port will be allocated if this Service requires one.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport

  - **ports.appProtocol** (string)

@@ -43,21 +43,6 @@ The following *predicates* implement filtering:
 - `MaxCSIVolumeCount`: Decides how many {{< glossary_tooltip text="CSI" term_id="csi" >}}
  volumes should be attached, and whether that's over a configured limit.

- `CheckNodeMemoryPressure`: If a Node is reporting memory pressure, and there's no
-  configured exception, the Pod won't be scheduled there.
-
- `CheckNodePIDPressure`: If a Node is reporting that process IDs are scarce, and
-  there's no configured exception, the Pod won't be scheduled there.
-
- `CheckNodeDiskPressure`: If a Node is reporting storage pressure (a filesystem that
-   is full or nearly full), and there's no configured exception, the Pod won't be
-   scheduled there.
-
- `CheckNodeCondition`: Nodes can report that they have a completely full filesystem,
-  that networking isn't available or that kubelet is otherwise not ready to run Pods.
-  If such a condition is set for a Node, and there's no configured exception, the Pod
-  won't be scheduled there.
-
 - `PodToleratesNodeTaints`: checks if a Pod's {{< glossary_tooltip text="tolerations" term_id="toleration" >}}
  can tolerate the Node's {{< glossary_tooltip text="taints" term_id="taint" >}}.

@@ -59,7 +59,7 @@ kubeadm init phase control-plane all [flags]
 <td colspan="2">--apiserver-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the API Server or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the API Server or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 <tr>
@@ -87,7 +87,7 @@ kubeadm init phase control-plane all [flags]
 <td colspan="2">--controller-manager-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Controller Manager or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Controller Manager or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 <tr>
@@ -136,7 +136,7 @@ kubeadm init phase control-plane all [flags]
 <td colspan="2">--scheduler-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Scheduler or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Scheduler or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 <tr>
@@ -48,7 +48,7 @@ kubeadm init phase control-plane apiserver [flags]
 <td colspan="2">--apiserver-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the API Server or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the API Server or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 <tr>
@@ -48,7 +48,7 @@ kubeadm init phase control-plane controller-manager [flags]
 <td colspan="2">--controller-manager-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Controller Manager or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Controller Manager or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 <tr>
@@ -76,7 +76,7 @@ kubeadm init phase control-plane scheduler [flags]
 <td colspan="2">--scheduler-extra-args &lt;comma-separated 'key=value' pairs&gt;</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Scheduler or override default ones in form of <!-- raw HTML omitted -->=<!-- raw HTML omitted --></p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>A set of extra flags to pass to the Scheduler or override default ones in form of &lt;flagname&gt;=&lt;value&gt;</p></td>
 </tr>

 </tbody>
@@ -147,7 +147,7 @@ kubeadm join [api-server-endpoint] [flags]
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -83,7 +83,7 @@ kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags]
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -69,7 +69,7 @@ kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags]
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -69,7 +69,7 @@ kubeadm join phase control-plane-prepare download-certs [api-server-endpoint] [f
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -69,7 +69,7 @@ kubeadm join phase control-plane-prepare kubeconfig [api-server-endpoint] [flags
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -62,7 +62,7 @@ kubeadm join phase kubelet-start [api-server-endpoint] [flags]
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -97,7 +97,7 @@ kubeadm join phase preflight [api-server-endpoint] [flags]
 <td colspan="2">--discovery-token-ca-cert-hash strings</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;<!-- raw HTML omitted -->:<!-- raw HTML omitted -->&quot;).</p></td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>For token-based discovery, validate that the root CA public key matches this hash (format: &quot;&lt;type&gt;:&lt;value&gt;&quot;).</p></td>
 </tr>

 <tr>
@@ -24,7 +24,7 @@ on how your cluster is deployed.

 To avoid running into cloud provider quota issues, when creating a cluster with many nodes,
 consider:
-* Request a quota increase for cloud resources such as:
+* Requesting a quota increase for cloud resources such as:
    * Computer instances
    * CPUs
    * Storage volumes
@@ -33,7 +33,7 @@ consider:
    * Number of load balancers
    * Network subnets
    * Log streams
-* Gate the cluster scaling actions to brings up new nodes in batches, with a pause
+* Gating the cluster scaling actions to bring up new nodes in batches, with a pause
  between batches, because some cloud providers rate limit the creation of new instances.

 ## Control plane components
@@ -66,6 +66,10 @@ When creating a cluster, you can (using custom tooling):
 * start and configure additional etcd instance
 * configure the {{< glossary_tooltip term_id="kube-apiserver" text="API server" >}} to use it for storing events

+See [Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) and
+[Set up a High Availability etcd cluster with kubeadm](docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/)
+for details on configuring and managing etcd for a large cluster.
+
 ## Addon resources

 Kubernetes [resource limits](/docs/concepts/configuration/manage-resources-containers/)
@@ -78,9 +78,13 @@ kind: ClusterConfiguration
 kubernetesVersion: v1.16.0
 scheduler:
  extraArgs:
-    bind-address: 0.0.0.0
-    config: /home/johndoe/schedconfig.yaml
-    kubeconfig: /home/johndoe/kubeconfig.yaml
+    config: /etc/kubernetes/scheduler-config.yaml
+  extraVolumes:
+    - name: schedulerconfig
+      hostPath: /home/johndoe/schedconfig.yaml
+      mountPath: /etc/kubernetes/scheduler-config.yaml
+      readOnly: true
+      pathType: "File"
 ```


@@ -246,8 +246,8 @@ this example.

    ```sh
    root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
-    root@HOST1 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
-    root@HOST2 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
+    root@HOST1 $ kubeadm init phase etcd local --config=/tmp/${HOST1}/kubeadmcfg.yaml
+    root@HOST2 $ kubeadm init phase etcd local --config=/tmp/${HOST2}/kubeadmcfg.yaml
    ```

 1. Optional: Check the cluster health
@@ -1,4 +0,0 @@
---
-title: "Release notes and version skew"
-weight: 10
---
@@ -196,13 +196,6 @@ the slightly simpler syntax:
 kubectl port-forward deployment/mongo :27017
 ```

-The output is similar to this:
-
-```
-Forwarding from 127.0.0.1:63753 -> 27017
-Forwarding from [::1]:63753 -> 27017
-```
-
 The `kubectl` tool finds a local port number that is not in use (avoiding low ports numbers,
 because these might be used by other applications). The output is similar to:

@@ -34,7 +34,7 @@ Dashboard also provides information on the state of Kubernetes resources in your
 The Dashboard UI is not deployed by default. To deploy it, run the following command:

 ```
-kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
 ```

 ## Accessing the Dashboard UI
@@ -229,7 +229,7 @@ serverTLSBootstrap: true

 If you have already created the cluster you must adapt it by doing the following:
 - Find and edit the `kubelet-config-{{< skew latestVersion >}}` ConfigMap in the `kube-system` namespace.
-In that ConfigMap, the `config` key has a
+In that ConfigMap, the `kubelet` key has a
 [KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
 document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`.
 - On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml`
@@ -79,7 +79,7 @@ If you are using the sample manifest from the previous point, this will require
  * If kube-proxy is running in IPVS mode:

    ``` bash
-     sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
+     sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
    ```
     In this mode, node-local-dns pods listen only on `<node-local-address>`. The node-local-dns interface cannot bind the kube-dns cluster IP since the interface used for IPVS loadbalancing already uses this address.
     `__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
@@ -131,6 +131,8 @@ The output is similar to:

 ```yaml
 apiVersion: v1
+data:
+  config.yaml: YXBpVXJsOiAiaHR0cHM6Ly9teS5hcGkuY29tL2FwaS92MSIKdXNlcm5hbWU6IHt7dXNlcm5hbWV9fQpwYXNzd29yZDoge3twYXNzd29yZH19
 kind: Secret
 metadata:
  creationTimestamp: 2018-11-15T20:40:59Z
@@ -139,8 +141,6 @@ metadata:
  resourceVersion: "7225"
  uid: c280ad2e-e916-11e8-98f2-025000000001
 type: Opaque
-data:
-  config.yaml: YXBpVXJsOiAiaHR0cHM6Ly9teS5hcGkuY29tL2FwaS92MSIKdXNlcm5hbWU6IHt7dXNlcm5hbWV9fQpwYXNzd29yZDoge3twYXNzd29yZH19
 ```

 The commands `kubectl get` and `kubectl describe` avoid showing the contents of a `Secret` by
@@ -168,6 +168,8 @@ Results in the following Secret:

 ```yaml
 apiVersion: v1
+data:
+  username: YWRtaW5pc3RyYXRvcg==
 kind: Secret
 metadata:
  creationTimestamp: 2018-11-15T20:46:46Z
@@ -176,8 +178,6 @@ metadata:
  resourceVersion: "7579"
  uid: 91460ecb-e917-11e8-98f2-025000000001
 type: Opaque
-data:
-  username: YWRtaW5pc3RyYXRvcg==
 ```

 Where `YWRtaW5pc3RyYXRvcg==` decodes to `administrator`.
@@ -1,5 +1,5 @@
 ---
-title: Managing Secret using kubectl
+title: Managing Secrets using kubectl
 content_type: task
 weight: 10
 description: Creating Secret objects using kubectl command line.
@@ -15,7 +15,7 @@ description: Creating Secret objects using kubectl command line.

 ## Create a Secret

-A `Secret` can contain user credentials required by Pods to access a database.
+A `Secret` can contain user credentials required by pods to access a database.
 For example, a database connection string consists of a username and password.
 You can store the username in a file `./username.txt` and the password in a
 file `./password.txt` on your local machine.
@@ -24,11 +24,10 @@ file `./password.txt` on your local machine.
 echo -n 'admin' > ./username.txt
 echo -n '1f2d1e2e67df' > ./password.txt
 ```
-
-The `-n` flag in the above two commands ensures that the generated files will
-not contain an extra newline character at the end of the text. This is
-important because when `kubectl` reads a file and encode the content into
-base64 string, the extra newline character gets encoded too.
+In these commands, the `-n` flag ensures that the generated files do not have
+an extra newline character at the end of the text. This is important because
+when `kubectl` reads a file and encodes the content into a base64 string, the
+extra newline character gets encoded too.

 The `kubectl create secret` command packages these files into a Secret and creates
 the object on the API server.
@@ -45,7 +44,7 @@ The output is similar to:
 secret/db-user-pass created
 ```

-Default key name is the filename. You may optionally set the key name using
+The default key name is the filename. You can optionally set the key name using
 `--from-file=[key=]source`. For example:

 ```shell
@@ -54,17 +53,18 @@ kubectl create secret generic db-user-pass \
  --from-file=password=./password.txt
 ```

-You do not need to escape special characters in passwords from files
-(`--from-file`).
+You do not need to escape special characters in password strings that you 
+include in a file.

 You can also provide Secret data using the `--from-literal=<key>=<value>` tag.
 This tag can be specified more than once to provide multiple key-value pairs.
 Note that special characters such as `$`, `\`, `*`, `=`, and `!` will be
 interpreted by your [shell](https://en.wikipedia.org/wiki/Shell_(computing))
 and require escaping.
+
 In most shells, the easiest way to escape the password is to surround it with
-single quotes (`'`).  For example, if your actual password is `S!B\*d$zDsb=`,
-you should execute the command this way:
+single quotes (`'`). For example, if your password is `S!B\*d$zDsb=`,
+run the following command:

 ```shell
 kubectl create secret generic dev-db-secret \
@@ -74,7 +74,7 @@ kubectl create secret generic dev-db-secret \

 ## Verify the Secret

-You can check that the secret was created:
+Check that the Secret was created:

 ```shell
 kubectl get secrets
@@ -111,7 +111,7 @@ username:    5 bytes

 The commands `kubectl get` and `kubectl describe` avoid showing the contents
 of a `Secret` by default. This is to protect the `Secret` from being exposed
-accidentally to an onlooker, or from being stored in a terminal log.
+accidentally, or from being stored in a terminal log.

 ## Decoding the Secret  {#decoding-secret}

@@ -141,7 +141,7 @@ The output is similar to:

 ## Clean Up

-To delete the Secret you have created:
+Delete the Secret you created:

 ```shell
 kubectl delete secret db-user-pass
@@ -152,5 +152,5 @@ kubectl delete secret db-user-pass
 ## {{% heading "whatsnext" %}}

 - Read more about the [Secret concept](/docs/concepts/configuration/secret/)
- Learn how to [manage Secret using config file](/docs/tasks/configmap-secret/managing-secret-using-config-file/)
- Learn how to [manage Secret using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/)
+- Learn how to [manage Secrets using config files](/docs/tasks/configmap-secret/managing-secret-using-config-file/)
+- Learn how to [manage Secrets using kustomize](/docs/tasks/configmap-secret/managing-secret-using-kustomize/)
@@ -48,7 +48,19 @@ secretGenerator:
  - password=1f2d1e2e67df
 ```

-Note that in both cases, you don't need to base64 encode the values.
+You can also define the `secretGenerator` in the `kustomization.yaml`
+file by providing `.env` files.
+For example, the following `kustomization.yaml` file pulls in data from
+`.env.secret` file:
+
+```yaml
+secretGenerator:
+- name: db-user-pass
+  envs:
+  - .env.secret
+```
+
+Note that in all cases, you don't need to base64 encode the values.

 ## Create the Secret

@@ -197,21 +197,70 @@ As Pod specs with GMSA fields populated (as described above) are applied in a cl

 1. The container runtime configures each Windows container with the specified GMSA credential spec so that the container can assume the identity of the GMSA in Active Directory and access services in the domain using that identity.

+## Containerd 
+
+On Windows Server 2019, in order to use GMSA with containerd, you must be running OS Build 17763.1817 (or later) which can be installed using the patch [KB5000822](https://support.microsoft.com/en-us/topic/march-9-2021-kb5000822-os-build-17763-1817-2eb6197f-e3b1-4f42-ab51-84345e063564). 
+
+There is also a known issue with containerd that occurs when trying to connect to SMB shares from Pods. Once you have configured GMSA, the pod will be unable to connect to the share using the hostname or FQDN, but connecting to the share using an IP address works as expected. 
+
+```PowerShell
+ping adserver.ad.local
+```
+and correctly resolves the hostname to an IPv4 address. The output is similar to:
+
+```
+Pinging adserver.ad.local [192.168.111.18] with 32 bytes of data:
+Reply from 192.168.111.18: bytes=32 time=6ms TTL=124
+Reply from 192.168.111.18: bytes=32 time=5ms TTL=124
+Reply from 192.168.111.18: bytes=32 time=5ms TTL=124
+Reply from 192.168.111.18: bytes=32 time=5ms TTL=124
+```
+
+However, when attempting to browse the directory using the hostname
+
+```PowerShell
+cd \\adserver.ad.local\test
+```
+
+you see an error that implies the target share doesn't exist:
+
+```
+cd : Cannot find path '\\adserver.ad.local\test' because it does not exist.
+At line:1 char:1
+ cd \\adserver.ad.local\test
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    + CategoryInfo          : ObjectNotFound: (\\adserver.ad.local\test:String) [Set-Location], ItemNotFoundException
+    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand
+```
+
+but you notice that the error disappears if you browse to the share using its IPv4 address instead; for example:
+
+```PowerShell
+cd \\192.168.111.18\test
+```
+
+After you change into a directory within the share, you see a prompt similar to:
+
+```
+Microsoft.PowerShell.Core\FileSystem::\\192.168.111.18\test>
+```
+
+To correct the behaviour you must run the following on the node `reg add "HKLM\SYSTEM\CurrentControlSet\Services\hns\State" /v EnableCompartmentNamespace /t REG_DWORD /d 1` to add the required registry key. This node change will only take effect in newly created pods, meaning you must now recreate any running pods which require access to SMB shares.
+
 ## Troubleshooting

 If you are having difficulties getting GMSA to work in your environment, there are a few troubleshooting steps you can take.

-First, make sure the credspec has been passed to the Pod.  To do this you will need to `exec` into one of your Pods and check the output of the `nltest.exe /parentdomain` command.  In the example below the Pod did not get the credspec correctly:
+First, make sure the credspec has been passed to the Pod.  To do this you will need to `exec` into one of your Pods and check the output of the `nltest.exe /parentdomain` command.  

-```shell
+In the example below the Pod did not get the credspec correctly:
+
+```PowerShell
 kubectl exec -it iis-auth-7776966999-n5nzr powershell.exe
-
-Windows PowerShell
-Copyright (C) Microsoft Corporation. All rights reserved.
-
-PS C:\> nltest.exe /parentdomain
+```
+nltest.exe /parentdomain` results in the following error:
+```
 Getting parent domain failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE
-PS C:\>
 ```

 If your Pod did get the credspec correctly, then next check communication with the domain.  First, from inside of your Pod, quickly do an nslookup to find the root of your domain.
@@ -224,23 +273,30 @@ This will tell us 3 things:

 If the DNS and communication test passes, next you will need to check if the Pod has established secure channel communication with the domain.  To do this, again, `exec` into your Pod and run the `nltest.exe /query` command.

-```shell
-PS C:\> nltest.exe /query
+```PowerShell
+nltest.exe /query
+```
+
+Results in the following output:
+```
 I_NetLogonControl failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE
 ```

-This tells us that for some reason, the Pod was unable to logon to the domain using the account specified in the credspec.  You can try to repair the secure channel by running the `nltest.exe /sc_reset:domain.example` command.
+This tells us that for some reason, the Pod was unable to logon to the domain using the account specified in the credspec.  You can try to repair the secure channel by running the following:

-```shell
-PS C:\> nltest /sc_reset:domain.example
+```PowerShell
+nltest /sc_reset:domain.example
+```
+
+If the command is successful you will see and output similar to this:
+```
 Flags: 30 HAS_IP  HAS_TIMESERV
 Trusted DC Name \\dc10.domain.example
 Trusted DC Connection Status Status = 0 0x0 NERR_Success
 The command completed successfully
-PS C:\>
 ```

-If the above command corrects the error, you can automate the step by adding the following lifecycle hook to your Pod spec.  If it did not correct the error, you will need to examine your credspec again and confirm that it is correct and complete.
+If the above corrects the error, you can automate the step by adding the following lifecycle hook to your Pod spec.  If it did not correct the error, you will need to examine your credspec again and confirm that it is correct and complete.

 ```yaml
        image: registry.domain.example/iis-auth:1809v1
@@ -252,6 +308,3 @@ If the above command corrects the error, you can automate the step by adding the
 ```

 If you add the `lifecycle` section show above to your Pod spec, the Pod will execute the commands listed to restart the `netlogon` service until the `nltest.exe /query` command exits without error.
-
-## GMSA limitations
-When using the [ContainerD runtime for Windows](/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#cri-containerd) accessing restricted network shares via the GMSA domain identity fails. The container will receive the identity of and calls from `nltest.exe /query` will work.  It is recommended to use the [Docker EE runtime](/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#docker-ee) if access to network shares is required.  The Windows Server team is working on resolving the issue in the Windows Kernel and will release a patch to resolve this issue in the future. Look for updates on the [Microsoft Windows Containers issue tracker](https://github.com/microsoft/Windows-Containers/issues/44).
@@ -624,9 +624,20 @@ Like before, all previous files in the `/etc/config/` directory will be deleted.
 You can project keys to specific paths and specific permissions on a per-file
 basis. The [Secrets](/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod) user guide explains the syntax.

+### Optional References
+
+A ConfigMap reference may be marked "optional".  If the ConfigMap is non-existent, the mounted volume will be empty. If the ConfigMap exists, but the referenced
+key is non-existent the path will be absent beneath the mount point.
+
 ### Mounted ConfigMaps are updated automatically

-When a ConfigMap already being consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted ConfigMap is fresh on every periodic sync. However, it is using its local ttl-based cache for getting the current value of the ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + ttl of ConfigMaps cache (1 minute by default) in kubelet. You can trigger an immediate refresh by updating one of the pod's annotations.
+When a mounted ConfigMap is updated, the projected content is eventually updated too.  This applies in the case where an optionally referenced ConfigMap comes into
+existence after a pod has started.
+
+Kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, it uses its local TTL-based cache for getting the current value of the
+ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as
+kubelet sync period (1 minute by default) + TTL of ConfigMaps cache (1 minute by default) in kubelet. You can trigger an immediate refresh by updating one of
+the pod's annotations.

 {{< note >}}
 A container using a ConfigMap as a [subPath](/docs/concepts/storage/volumes/#using-subpath) volume will not receive ConfigMap updates.
@@ -146,7 +146,7 @@ All modifications to a cron job, especially its `.spec`, are applied only to the
 The `.spec.schedule` is a required field of the `.spec`.
 It takes a [Cron](https://en.wikipedia.org/wiki/Cron) format string, such as `0 * * * *` or `@hourly`, as schedule time of its jobs to be created and executed.

-The format also includes extended `vixie cron` step values. As explained in the
+The format also includes extended "Vixie cron" step values. As explained in the
 [FreeBSD manual](https://www.freebsd.org/cgi/man.cgi?crontab%285%29):

 > Step values can be	used in	conjunction with ranges.  Following a range
@@ -86,6 +86,43 @@ metadata:
  name: example-configmap-1-8mbdf7882g
 ```

+To generate a ConfigMap from an env file, add an entry to the `envs` list in `configMapGenerator`. Here is an example of generating a ConfigMap with a data item from a `.env` file:
+
+```shell
+# Create a .env file
+cat <<EOF >.env
+FOO=Bar
+EOF
+
+cat <<EOF >./kustomization.yaml
+configMapGenerator:
+- name: example-configmap-1
+  envs:
+  - .env
+EOF
+```
+
+The generated ConfigMap can be examined with the following command:
+
+```shell
+kubectl kustomize ./
+```
+
+The generated ConfigMap is:
+
+```yaml
+apiVersion: v1
+data:
+    FOO=Bar
+kind: ConfigMap
+metadata:
+  name: example-configmap-1-8mbdf7882g
+```
+
+{{< note >}}
+Each variable in the `.env` file becomes a separate key in the ConfigMap that you generate. This is different from the previous example which embeds a file named `.properties` (and all its entries) as the value for a single key.
+{{< /note >}}
+
 ConfigMaps can also be generated from literal key-value pairs. To generate a ConfigMap from a literal key-value pair, add an entry to the `literals` list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a key-value pair:

 ```shell
@@ -975,4 +1012,3 @@ deployment.apps "dev-my-nginx" deleted
 * [Kubectl Command Reference](/docs/reference/generated/kubectl/kubectl-commands/)
 * [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/)

-
@@ -310,10 +310,10 @@ Patch your Deployment:

 {{< tabs name="kubectl_retainkeys_example" >}}
 {{{< tab name="Bash" codelang="bash" >}}
-kubectl patch deployment retainkeys-demo --patch "$(cat patch-file-no-retainkeys.yaml)"
+kubectl patch deployment retainkeys-demo --type merge --patch "$(cat patch-file-no-retainkeys.yaml)"
 {{< /tab >}}
 {{< tab name="PowerShell" codelang="posh" >}}
-kubectl patch deployment retainkeys-demo --patch $(Get-Content patch-file-no-retainkeys.yaml -Raw)
+kubectl patch deployment retainkeys-demo --type merge --patch $(Get-Content patch-file-no-retainkeys.yaml -Raw)
 {{< /tab >}}}
 {{< /tabs >}}

@@ -341,10 +341,10 @@ Patch your Deployment again with this new patch:

 {{< tabs name="kubectl_retainkeys2_example" >}}
 {{{< tab name="Bash" codelang="bash" >}}
-kubectl patch deployment retainkeys-demo --patch "$(cat patch-file-retainkeys.yaml)"
+kubectl patch deployment retainkeys-demo --type merge --patch "$(cat patch-file-retainkeys.yaml)"
 {{< /tab >}}
 {{< tab name="PowerShell" codelang="posh" >}}
-kubectl patch deployment retainkeys-demo --patch $(Get-Content patch-file-retainkeys.yaml -Raw)
+kubectl patch deployment retainkeys-demo --type merge --patch $(Get-Content patch-file-retainkeys.yaml -Raw)
 {{< /tab >}}}
 {{< /tabs >}}

@@ -43,14 +43,14 @@ You may need to delete the associated headless service separately after the Stat
 kubectl delete service <service-name>
 ```

-When deleting a StatefulSet through `kubectl`, the StatefulSet scales down to 0. All Pods that are part of this workload are also deleted. If you want to delete only the StatefulSet and not the Pods, use `--cascade=false`.
+When deleting a StatefulSet through `kubectl`, the StatefulSet scales down to 0. All Pods that are part of this workload are also deleted. If you want to delete only the StatefulSet and not the Pods, use `--cascade=orphan`.
 For example:

 ```shell
-kubectl delete -f <file.yaml> --cascade=false
+kubectl delete -f <file.yaml> --cascade=orphan
 ```

-By passing `--cascade=false` to `kubectl delete`, the Pods managed by the StatefulSet are left behind even after the StatefulSet object itself is deleted. If the pods have a label `app=myapp`, you can then delete them as follows:
+By passing `--cascade=orphan` to `kubectl delete`, the Pods managed by the StatefulSet are left behind even after the StatefulSet object itself is deleted. If the pods have a label `app=myapp`, you can then delete them as follows:

 ```shell
 kubectl delete pods -l app=myapp
@@ -22,7 +22,7 @@ spec:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
-        - image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent:v0.0.12
+        - image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent:v0.0.16
          name: konnectivity-agent
          command: ["/proxy-agent"]
          args: [
@@ -8,7 +8,7 @@ spec:
  hostNetwork: true
  containers:
  - name: konnectivity-server-container
-    image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.12
+    image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.16
    command: ["/proxy-server"]
    args: [
            "--logtostderr=true",
@@ -8,7 +8,7 @@ host="redis"
 # import os
 # host = os.getenv("REDIS_SERVICE_HOST")

-q = rediswq.RedisWQ(name="job2", host="redis")
+q = rediswq.RedisWQ(name="job2", host=host)
 print("Worker with sessionID: " +  q.sessionID())
 print("Initial queue state: empty=" + str(q.empty()))
 while not q.empty():
@@ -18,6 +18,7 @@ spec:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
+        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
@@ -0,0 +1,17 @@
+# See the OWNERS docs at https://go.k8s.io/owners
+
+# This is the directory for English source content.
+# Teams and members are visible at https://github.com/orgs/kubernetes/teams.
+
+reviewers:
+  - sig-docs-en-reviews
+  - release-engineering-reviewers
+
+approvers:
+  - sig-docs-en-owners
+  - sig-release-leads
+  - release-engineering-approvers
+
+labels:
+- sig/release
+- area/release-eng
@@ -0,0 +1,27 @@
+---
+linktitle: Release History
+title: Releases
+type: docs
+---
+
+
+<!-- overview -->
+
+The Kubernetes project maintains release branches for the most recent three minor releases ({{< skew latestVersion >}}, {{< skew prevMinorVersion >}}, {{< skew oldestMinorVersion >}}).  Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.
+
+Kubernetes versions are expressed as **x.y.z**,
+where **x** is the major version, **y** is the minor version, and **z** is the patch version, following [Semantic Versioning](https://semver.org/) terminology.
+
+More information in the [version skew policy](/releases/version-skew-policy/) document.
+
+<!-- body -->
+
+## Release History
+
+{{< release-data >}}
+
+## Upcoming Release
+
+Check out the [schedule](https://github.com/kubernetes/sig-release/tree/master/releases/release-{{< skew nextMinorVersion >}}) for the upcoming **{{< skew nextMinorVersion >}}** Kubernetes release!
+
+## Helpful Resources
@@ -0,0 +1,26 @@
+---
+title: Download Kubernetes
+type: docs
+---
+## Core Kubernetes components
+
+Find links to download Kubernetes components (and their checksums) in the [CHANGELOG](https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG) files.
+
+Alternately, use [downloadkubernetes.com](https://www.downloadkubernetes.com/) to filter by version and architecture.
+
+## kubectl
+
+<!-- overview -->
+The Kubernetes command-line tool, [kubectl](/docs/reference/kubectl/kubectl/), allows
+you to run commands against Kubernetes clusters.
+
+You can use kubectl to deploy applications, inspect and manage cluster resources,
+and view logs. For more information including a complete list of kubectl operations, see the
+[`kubectl` reference documentation](/docs/reference/kubectl/).
+
+kubectl is installable on a variety of Linux platforms, macOS and Windows. 
+Find your preferred operating system below.
+
+- [Install kubectl on Linux](/docs/tasks/tools/install-kubectl-linux)
+- [Install kubectl on macOS](/docs/tasks/tools/install-kubectl-macos)
+- [Install kubectl on Windows](/docs/tasks/tools/install-kubectl-windows)
@@ -0,0 +1,13 @@
+---
+linktitle: Release Notes
+title: Notes
+type: docs
+description: >
+  Kubernetes release notes.
+sitemap:
+  priority: 0.5
+---
+
+Release notes can be found by reading the [Changelog](https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG) that matches your Kubernetes version. View the changelog for {{< skew latestVersion >}} on [GitHub](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-{{< skew latestVersion >}}.md).
+
+Alternately, release notes can be searched and filtered online at: [relnotes.k8s.io](https://relnotes.k8s.io). View filtered release notes for {{< skew latestVersion >}} on [relnotes.k8s.io](https://relnotes.k8s.io/?releaseVersions={{< skew latestVersion >}}.0).
@@ -0,0 +1,164 @@
+---
+title: Patch Releases
+type: docs
+---
+
+Schedule and team contact information for Kubernetes patch releases.
+
+For general information about Kubernetes release cycle, see the
+[release process description].
+
+## Cadence
+
+Our typical patch release cadence is monthly.  It is
+commonly a bit faster (1 to 2 weeks) for the earliest patch releases
+after a 1.X minor release.  Critical bug fixes may cause a more
+immediate release outside of the normal cadence.  We also aim to not make
+releases during major holiday periods.
+
+## Contact
+
+See the [Release Managers page][release-managers] for full contact details on the Patch Release Team.
+
+Please give us a business day to respond - we may be in a different timezone!
+
+In between releases the team is looking at incoming cherry pick
+requests on a weekly basis.  The team will get in touch with
+submitters via GitHub PR, SIG channels in Slack, and direct messages
+in Slack and [email](mailto:release-managers-private@kubernetes.io)
+if there are questions on the PR.
+
+## Cherry picks
+
+Please follow the [cherry pick process][cherry-picks].
+
+Cherry picks must be merge-ready in GitHub with proper labels (e.g.,
+`approved`, `lgtm`, `release-note`) and passing CI tests ahead of the
+cherry pick deadline.  This is typically two days before the target
+release, but may be more.  Earlier PR readiness is better, as we
+need time to get CI signal after merging your cherry picks ahead
+of the actual release.
+
+Cherry pick PRs which miss merge criteria will be carried over and tracked
+for the next patch release.
+
+## Support Period
+
+In accordance with the [yearly support KEP][yearly-support], the Kubernetes
+Community will support active patch release series for a period of roughly
+fourteen (14) months.
+
+The first twelve months of this timeframe will be considered the standard
+period.
+
+Towards the end of the twelve month, the following will happen:
+
+- [Release Managers][release-managers] will cut a release
+- The patch release series will enter maintenance mode
+
+During the two-month maintenance mode period, Release Managers may cut
+additional maintenance releases to resolve:
+
+- CVEs (under the advisement of the Product Security Committee)
+- dependency issues (including base image updates)
+- critical core component issues
+
+At the end of the two-month maintenance mode period, the patch release series
+will be considered EOL (end of life) and cherry picks to the associated branch
+are to be closed soon afterwards.
+
+Note that the 28th of the month was chosen for maintenance mode and EOL target
+dates for simplicity (every month has it).
+
+## Upcoming Monthly Releases
+
+Timelines may vary with the severity of bug fixes, but for easier planning we
+will target the following monthly release points.  Unplanned, critical
+releases may also occur in between these.
+
+| Monthly Patch Release | Target date |
+| --- | --- |
+| June 2021 | 2021-06-16 |
+| July 2021 | 2021-07-14 |
+| August 2021 | 2021-08-11 |
+| September 2021 | 2021-09-15 |
+
+## Detailed Release History for Active Branches
+
+### 1.21
+
+**1.21** enters maintenance mode on **2022-04-28**
+
+End of Life for **1.21** is **2022-06-28**
+
+| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
+|--- |--- |--- |
+| 1.21.2        | 2021-06-12           | 2021-06-16  |
+| 1.21.1        | 2021-05-07           | 2021-05-12  |
+
+### 1.20
+
+**1.20** enters maintenance mode on **2021-12-28**
+
+End of Life for **1.20** is **2022-02-28**
+
+| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
+|--- |--- |--- |
+| 1.20.8        | 2021-06-12           | 2021-06-16  |
+| 1.20.7        | 2021-05-07           | 2021-05-12  |
+| 1.20.6        | 2021-04-09           | 2021-04-14  |
+| 1.20.5        | 2021-03-12           | 2021-03-17  |
+| 1.20.4        | 2021-02-12           | 2021-02-18  |
+| 1.20.3        | [Conformance Tests Issue](https://groups.google.com/g/kubernetes-dev/c/oUpY9vWgzJo) | 2021-02-17  |
+| 1.20.2        | 2021-01-08           | 2021-01-13  |
+| 1.20.1        | [Tagging Issue](https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA) | 2020-12-18  |
+
+### 1.19
+
+**1.19** enters maintenance mode on **2021-08-28**
+
+End of Life for **1.19** is **2021-10-28**
+
+| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
+|--- |--- |--- |
+| 1.19.12       | 2021-06-12           | 2021-06-16  |
+| 1.19.11       | 2021-05-07           | 2021-05-12  |
+| 1.19.10       | 2021-04-09           | 2021-04-14  |
+| 1.19.9        | 2021-03-12           | 2021-03-17  |
+| 1.19.8        | 2021-02-12           | 2021-02-17  |
+| 1.19.7        | 2021-01-08           | 2021-01-13  |
+| 1.19.6        | [Tagging Issue](https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA) | 2020-12-18  |
+| 1.19.5        | 2020-12-04           | 2020-12-09  |
+| 1.19.4        | 2020-11-06           | 2020-11-11  |
+| 1.19.3        | 2020-10-09           | 2020-10-14  |
+| 1.19.2        | 2020-09-11           | 2020-09-16  |
+| 1.19.1        | 2020-09-04           | 2020-09-09  |
+
+## Non-Active Branch History
+
+These releases are no longer supported.
+
+| Minor Version | Final Patch Release | EOL date |
+| --- | --- | --- |
+| 1.18 | 1.18.19 | 2021-05-12 |
+| 1.17 | 1.17.17 | 2021-01-13 |
+| 1.16 | 1.16.15 | 2020-09-02 |
+| 1.15 | 1.15.12 | 2020-05-06 |
+| 1.14 | 1.14.10 | 2019-12-11 |
+| 1.13 | 1.13.12 | 2019-10-15 |
+| 1.12 | 1.12.10 | 2019-07-08 |
+| 1.11 | 1.11.10 | 2019-05-01 |
+| 1.10 | 1.10.13 | 2019-02-13 |
+| 1.9  | 1.9.11  | 2018-09-29 |
+| 1.8  | 1.8.15  | 2018-07-12 |
+| 1.7  | 1.7.16  | 2018-04-04 |
+| 1.6  | 1.6.13  | 2017-11-23 |
+| 1.5  | 1.5.8   | 2017-10-01 |
+| 1.4  | 1.4.12  | 2017-04-21 |
+| 1.3  | 1.3.10  | 2016-11-01 |
+| 1.2  | 1.2.7   | 2016-10-23 |
+
+[cherry-picks]: https://git.k8s.io/community/contributors/devel/sig-release/cherry-picks.md
+[release-managers]: /release-managers.md
+[release process description]: /release.md
+[yearly-support]: https://git.k8s.io/enhancements/keps/sig-release/1498-kubernetes-yearly-support-period/README.md
@@ -0,0 +1,213 @@
+---
+title: Release Managers
+type: docs
+---
+
+"Release Managers" is an umbrella term that encompasses the set of Kubernetes
+contributors responsible for maintaining release branches, tagging releases,
+and building/packaging Kubernetes.
+
+The responsibilities of each role are described below.
+
+- [Contact](#contact)
+- [Handbooks](#handbooks)
+- [Release Managers](#release-managers)
+  - [Becoming a Release Manager](#becoming-a-release-manager)
+- [Release Manager Associates](#release-manager-associates)
+  - [Becoming a Release Manager Associate](#becoming-a-release-manager-associate)
+- [Build Admins](#build-admins)
+- [SIG Release Leads](#sig-release-leads)
+  - [Chairs](#chairs)
+  - [Technical Leads](#technical-leads)
+
+## Contact
+
+| Mailing List | Slack | Visibility | Usage | Membership |
+| --- | --- | --- | --- | --- |
+| [release-managers@kubernetes.io](mailto:release-managers@kubernetes.io) | [#release-management](https://kubernetes.slack.com/messages/CJH2GBF7Y) (channel) / @release-managers (user group) | Public | Public discussion for Release Managers | All Release Managers (including Associates, Build Admins, and SIG Chairs) |
+| [release-managers-private@kubernetes.io](mailto:release-managers-private@kubernetes.io) | N/A | Private | Private discussion for privileged Release Managers | Release Managers, SIG Release leadership |
+| [security-release-team@kubernetes.io](mailto:security-release-team@kubernetes.io) | [#security-release-team](https://kubernetes.slack.com/archives/G0162T1RYHG) (channel) / @security-rel-team (user group) | Private | Security release coordination with the Product Security Committee | [security-discuss-private@kubernetes.io](mailto:security-discuss-private@kubernetes.io), [release-managers-private@kubernetes.io](mailto:release-managers-private@kubernetes.io) |
+
+## Handbooks
+
+**NOTE: The Patch Release Team and Branch Manager handbooks will be de-duplicated at a later date.**
+
+- [Patch Release Team][handbook-patch-release]
+- [Branch Managers][handbook-branch-mgmt]
+- [Build Admins][handbook-packaging]
+
+## Release Managers
+
+**Note:** The documentation might refer to the Patch Release Team and the
+Branch Management role. Those two roles were consolidated into the
+Release Managers role.
+
+Minimum requirements for Release Managers and Release Manager Associates are:
+
+- Familiarity with basic Unix commands and able to debug shell scripts.
+- Familiarity with branched source code workflows via `git` and associated
+  `git` command line invocations.
+- General knowledge of Google Cloud (Cloud Build and Cloud Storage).
+- Open to seeking help and communicating clearly.
+- Kubernetes Community [membership][community-membership]
+
+Release Managers are responsible for:
+
+- Coordinating and cutting Kubernetes releases:
+  - Patch releases (`x.y.z`, where `z` > 0)
+  - Minor releases (`x.y.z`, where `z` = 0)
+  - Pre-releases (alpha, beta, and release candidates)
+  - Working with the [Release Team][release-team] through each
+  release cycle
+  - Setting the [schedule and cadence for patch releases][patches]
+- Maintaining the release branches:
+  - Reviewing cherry picks
+  - Ensuring the release branch stays healthy and that no unintended patch
+    gets merged
+- Mentoring the [Release Manager Associates](#associates) group
+- Actively developing features and maintaining the code in k/release
+- Supporting Release Manager Associates and contributors through actively
+  participating in the Buddy program
+  - Check in monthly with Associates and delegate tasks, empower them to cut
+    releases, and mentor
+  - Being available to support Associates in onboarding new contributors e.g.,
+    answering questions and suggesting appropriate work for them to do
+
+This team at times works in close conjunction with the
+[Product Security Committee][psc] and therefore should abide by the guidelines
+set forth in the [Security Release Process][security-release-process].
+
+GitHub Access Controls: [@kubernetes/release-managers](https://github.com/orgs/kubernetes/teams/release-managers)
+
+GitHub Mentions: [@kubernetes/release-engineering](https://github.com/orgs/kubernetes/teams/release-engineering)
+
+- Adolfo García Veytia ([@puerco](https://github.com/puerco))
+- Carlos Panato ([@cpanato](https://github.com/cpanato))
+- Daniel Mangum ([@hasheddan](https://github.com/hasheddan))
+- Marko Mudrinić ([@xmudrii](https://github.com/xmudrii))
+- Sascha Grunert ([@saschagrunert](https://github.com/saschagrunert))
+- Stephen Augustus ([@justaugustus](https://github.com/justaugustus))
+
+### Becoming a Release Manager
+
+To become a Release Manager, one must first serve as a Release Manager
+Associate. Associates graduate to Release Manager by actively working on
+releases over several cycles and:
+
+- demonstrating the willingness to lead
+- tag-teaming with Release Managers on patches, to eventually cut a release
+  independently
+  - because releases have a limiting function, we also consider substantial
+    contributions to image promotion and other core Release Engineering tasks
+- questioning how Associates work, suggesting improvements, gathering feedback,
+  and driving change
+- being reliable and responsive
+- leaning into advanced work that requires Release Manager-level access and
+  privileges to complete
+
+## Release Manager Associates
+
+Release Manager Associates are apprentices to the Release Managers, formerly
+referred to as Release Manager shadows. They are responsible for:
+
+- Patch release work, cherry pick review
+- Contributing to k/release: updating dependencies and getting used to the
+  source codebase
+- Contributing to the documentation: maintaining the handbooks, ensuring that
+  release processes are documented
+- With help from a release manager: working with the Release Team during the
+  release cycle and cutting Kubernetes releases
+- Seeking opportunities to help with prioritization and communication
+  - Sending out pre-announcements and updates about patch releases
+  - Updating the calendar, helping with the release dates and milestones from
+    the [release cycle timeline][k-sig-release-releases]
+- Through the Buddy program, onboarding new contributors and pairing up with
+  them on tasks
+
+GitHub Mentions: @kubernetes/release-engineering
+
+- Arnaud Meukam ([@ameukam](https://github.com/ameukam))
+- Jim Angel ([@jimangel](https://github.com/jimangel))
+- Joyce Kung ([@thejoycekung](https://github.com/thejoycekung))
+- Max Körbächer ([@mkorbi](https://github.com/mkorbi))
+- Nabarun Pal ([@palnabarun](https://github.com/palnabarun))
+- Seth McCombs ([@sethmccombs](https://github.com/sethmccombs))
+- Taylor Dolezal ([@onlydole](https://github.com/onlydole))
+- Verónica López ([@verolop](https://github.com/verolop))
+- Wilson Husin ([@wilsonehusin](https://github.com/wilsonehusin))
+
+### Becoming a Release Manager Associate
+
+Contributors can become Associates by demonstrating the following:
+
+- consistent participation, including 6-12 months of active release
+  engineering-related work
+- experience fulfilling a technical lead role on the Release Team during a
+  release cycle
+  - this experience provides a solid baseline for understanding how SIG Release
+    works overall—including our expectations regarding technical skills,
+    communications/responsiveness, and reliability
+- working on k/release items that improve our interactions with Testgrid,
+  cleaning up libraries, etc.
+  - these efforts require interacting and pairing with Release Managers and
+    Associates
+
+## Build Admins
+
+Build Admins are (currently) Google employees with the requisite access to
+Google build systems/tooling to publish deb/rpm packages on behalf of the
+Kubernetes project. They are responsible for:
+
+- Building, signing, and publishing the deb/rpm packages
+- Being the interlock with Release Managers (and Associates) on the final steps
+of each minor (1.Y) and patch (1.Y.Z) release
+
+GitHub team: [@kubernetes/build-admins](https://github.com/orgs/kubernetes/teams/build-admins)
+
+- Aaron Crickenberger ([@spiffxp](https://github.com/spiffxp))
+- Amit Watve ([@amwat](https://github.com/amwat))
+- Benjamin Elder ([@BenTheElder](https://github.com/BenTheElder))
+- Grant McCloskey ([@MushuEE](https://github.com/MushuEE))
+
+## SIG Release Leads
+
+SIG Release Chairs and Technical Leads are responsible for:
+
+- The governance of SIG Release
+- Leading knowledge exchange sessions for Release Managers and Associates
+- Coaching on leadership and prioritization
+
+They are mentioned explicitly here as they are owners of the various
+communications channels and permissions groups (GitHub teams, GCP access) for
+each role. As such, they are highly privileged community members and privy to
+some private communications, which can at times relate to Kubernetes security
+disclosures.
+
+GitHub team: [@kubernetes/sig-release-leads](https://github.com/orgs/kubernetes/teams/sig-release-leads)
+
+### Chairs
+
+- Sascha Grunert ([@saschagrunert](https://github.com/saschagrunert))
+- Stephen Augustus ([@justaugustus](https://github.com/justaugustus))
+
+### Technical Leads
+
+- Daniel Mangum ([@hasheddan](https://github.com/hasheddan))
+- Jeremy Rickard ([@jeremyrickard](https://github.com/jeremyrickard))
+
+---
+
+Past Branch Managers, can be found in the [releases directory][k-sig-release-releases]
+of the kubernetes/sig-release repository within `release-x.y/release_team.md`.
+
+Example: [1.15 Release Team](https://git.k8s.io/sig-release/releases/release-1.15/release_team.md)
+
+[community-membership]: https://git.k8s.io/community/community-membership.md#member
+[handbook-branch-mgmt]: https://git.k8s.io/sig-release/release-engineering/role-handbooks/branch-manager.md
+[handbook-packaging]: https://git.k8s.io/sig-release/release-engineering/packaging.md
+[handbook-patch-release]: https://git.k8s.io/sig-release/release-engineering/role-handbooks/patch-release-team.md
+[k-sig-release-releases]: https://git.k8s.io/sig-release/releases
+[patches]: /patch-releases.md
+[psc]: https://git.k8s.io/community/committee-product-security/README.md
+[release-team]: https://git.k8s.io/sig-release/release-team/README.md
+[security-release-process]: https://git.k8s.io/security/security-release-process.md
@@ -0,0 +1,364 @@
+---
+title: Kubernetes Release Cycle
+type: docs
+auto_generated: true
+---
+<!-- THIS CONTENT IS AUTO-GENERATED via ./scripts/releng/update-release-info.sh in k/website -->
+
+{{< warning >}}
+This content is auto-generated and links may not function. The source of the document is located [here](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/release.md).
+{{< /warning >}}
+
+# Targeting enhancements, Issues and PRs to Release Milestones
+
+This document is focused on Kubernetes developers and contributors who need to
+create an enhancement, issue, or pull request which targets a specific release
+milestone.
+
+- [TL;DR](#tldr)
+  - [Normal Dev (Weeks 1-8)](#normal-dev-weeks-1-8)
+  - [Code Freeze (Weeks 9-11)](#code-freeze-weeks-9-11)
+  - [Post-Release (Weeks 11+)](#post-release-weeks-11)
+- [Definitions](#definitions)
+- [The Release Cycle](#the-release-cycle)
+- [Removal Of Items From The Milestone](#removal-of-items-from-the-milestone)
+- [Adding An Item To The Milestone](#adding-an-item-to-the-milestone)
+  - [Milestone Maintainers](#milestone-maintainers)
+  - [Feature additions](#feature-additions)
+  - [Issue additions](#issue-additions)
+  - [PR Additions](#pr-additions)
+- [Other Required Labels](#other-required-labels)
+  - [SIG Owner Label](#sig-owner-label)
+  - [Priority Label](#priority-label)
+  - [Issue/PR Kind Label](#issuepr-kind-label)
+
+The process for shepherding enhancements, issues, and pull requests into a
+Kubernetes release spans multiple stakeholders:
+
+- the enhancement, issue, and pull request owner(s)
+- SIG leadership
+- the [Release Team][release-team]
+
+Information on workflows and interactions are described below.
+
+As the owner of an enhancement, issue, or pull request (PR), it is your
+responsibility to ensure release milestone requirements are met. Automation and
+the Release Team will be in contact with you if updates are required, but
+inaction can result in your work being removed from the milestone. Additional
+requirements exist when the target milestone is a prior release (see
+[cherry pick process][cherry-picks] for more information).
+
+## TL;DR
+
+If you want your PR to get merged, it needs the following required labels and
+milestones, represented here by the Prow /commands it would take to add them:
+
+### Normal Dev (Weeks 1-8)
+
+- /sig {name}
+- /kind {type}
+- /lgtm
+- /approved
+
+### [Code Freeze][code-freeze] (Weeks 9-11)
+
+- /milestone {v1.y}
+- /sig {name}
+- /kind {bug, failing-test}
+- /lgtm
+- /approved
+
+### Post-Release (Weeks 11+)
+
+Return to 'Normal Dev' phase requirements:
+
+- /sig {name}
+- /kind {type}
+- /lgtm
+- /approved
+
+Merges into the 1.y branch are now [via cherry picks][cherry-picks], approved
+by [Release Managers][release-managers].
+
+In the past, there was a requirement for a milestone-targeted pull requests to
+have an associated GitHub issue opened, but this is no longer the case.
+Features or enhancements are effectively GitHub issues or [KEPs][keps] which
+lead to subsequent PRs.
+
+The general labeling process should be consistent across artifact types.
+
+## Definitions
+
+- *issue owners*: Creator, assignees, and user who moved the issue into a
+  release milestone
+
+- *Release Team*: Each Kubernetes release has a team doing project management
+  tasks described [here][release-team].
+
+  The contact info for the team associated with any given release can be found
+  [here](https://git.k8s.io/sig-release/releases/).
+
+- *Y days*: Refers to business days
+
+- *enhancement*: see "[Is My Thing an Enhancement?](https://git.k8s.io/enhancements/README.md#is-my-thing-an-enhancement)"
+
+- *[Enhancements Freeze][enhancements-freeze]*:
+  the deadline by which [KEPs][keps] have to be completed in order for
+  enhancements to be part of the current release
+
+- *[Exception Request][exceptions]*:
+  The process of requesting an extension on the deadline for a particular
+  Enhancement
+
+- *[Code Freeze][code-freeze]*:
+  The period of ~4 weeks before the final release date, during which only
+  critical bug fixes are merged into the release.
+
+- *[Pruning](https://git.k8s.io/sig-release/releases/release_phases.md#pruning)*:
+  The process of removing an Enhancement from a release milestone if it is not
+  fully implemented or is otherwise considered not stable.
+
+- *release milestone*: semantic version string or
+  [GitHub milestone](https://help.github.com/en/github/managing-your-work-on-github/associating-milestones-with-issues-and-pull-requests)
+  referring to a release MAJOR.MINOR `vX.Y` version.
+
+  See also
+  [release versioning](/contributors/design-proposals/release/versioning.md).
+
+- *release branch*: Git branch `release-X.Y` created for the `vX.Y` milestone.
+
+  Created at the time of the `vX.Y-rc.0` release and maintained after the
+  release for approximately 12 months with `vX.Y.Z` patch releases.
+
+  Note: releases 1.19 and newer receive 1 year of patch release support, and
+  releases 1.18 and earlier received 9 months of patch release support.
+
+## The Release Cycle
+
+![Image of one Kubernetes release cycle](release-cycle.png)
+
+Kubernetes releases currently happen approximately four times per year.
+
+The release process can be thought of as having three main phases:
+
+- Enhancement Definition
+- Implementation
+- Stabilization
+
+But in reality, this is an open source and agile project, with feature planning
+and implementation happening at all times. Given the project scale and globally
+distributed developer base, it is critical to project velocity to not rely on a
+trailing stabilization phase and rather have continuous integration testing
+which ensures the project is always stable so that individual commits can be
+flagged as having broken something.
+
+With ongoing feature definition through the year, some set of items will bubble
+up as targeting a given release. **[Enhancements Freeze][enhancements-freeze]**
+starts ~4 weeks into release cycle. By this point all intended feature work for
+the given release has been defined in suitable planning artifacts in
+conjunction with the Release Team's [Enhancements Lead](https://git.k8s.io/sig-release/release-team/role-handbooks/enhancements/README.md).
+
+After Enhancements Freeze, tracking milestones on PRs and issues is important.
+Items within the milestone are used as a punchdown list to complete the
+release. *On issues*, milestones must be applied correctly, via triage by the
+SIG, so that [Release Team][release-team] can track bugs and enhancements (any
+enhancement-related issue needs a milestone).
+
+There is some automation in place to help automatically assign milestones to
+PRs.
+
+This automation currently applies to the following repos:
+
+- `kubernetes/enhancements`
+- `kubernetes/kubernetes`
+- `kubernetes/release`
+- `kubernetes/sig-release`
+- `kubernetes/test-infra`
+
+At creation time, PRs against the `master` branch need humans to hint at which
+milestone they might want the PR to target. Once merged, PRs against the
+`master` branch have milestones auto-applied so from that time onward human
+management of that PR's milestone is less necessary. On PRs against release
+branches, milestones are auto-applied when the PR is created so no human
+management of the milestone is ever necessary.
+
+Any other effort that should be tracked by the Release Team that doesn't fall
+under that automation umbrella should be have a milestone applied.
+
+Implementation and bug fixing is ongoing across the cycle, but culminates in a
+code freeze period.
+
+**[Code Freeze][code-freeze]** starts in week ~10 and continues for ~2 weeks.
+Only critical bug fixes are accepted into the release codebase during this
+time.
+
+There are approximately two weeks following Code Freeze, and preceding release,
+during which all remaining critical issues must be resolved before release.
+This also gives time for documentation finalization.
+
+When the code base is sufficiently stable, the master branch re-opens for
+general development and work begins there for the next release milestone. Any
+remaining modifications for the current release are cherry picked from master
+back to the release branch. The release is built from the release branch.
+
+Each release is part of a broader Kubernetes lifecycle:
+
+![Image of Kubernetes release lifecycle spanning three releases](release-lifecycle.png)
+
+## Removal Of Items From The Milestone
+
+Before getting too far into the process for adding an item to the milestone,
+please note:
+
+Members of the [Release Team][release-team] may remove issues from the
+milestone if they or the responsible SIG determine that the issue is not
+actually blocking the release and is unlikely to be resolved in a timely
+fashion.
+
+Members of the Release Team may remove PRs from the milestone for any of the
+following, or similar, reasons:
+
+- PR is potentially de-stabilizing and is not needed to resolve a blocking
+  issue
+- PR is a new, late feature PR and has not gone through the enhancements
+  process or the [exception process][exceptions]
+- There is no responsible SIG willing to take ownership of the PR and resolve
+  any follow-up issues with it
+- PR is not correctly labelled
+- Work has visibly halted on the PR and delivery dates are uncertain or late
+
+While members of the Release Team will help with labelling and contacting
+SIG(s), it is the responsibility of the submitter to categorize PRs, and to
+secure support from the relevant SIG to guarantee that any breakage caused by
+the PR will be rapidly resolved.
+
+Where additional action is required, an attempt at human to human escalation
+will be made by the Release Team through the following channels:
+
+- Comment in GitHub mentioning the SIG team and SIG members as appropriate for
+  the issue type
+- Emailing the SIG mailing list
+  - bootstrapped with group email addresses from the
+    [community sig list][sig-list]
+  - optionally also directly addressing SIG leadership or other SIG members
+- Messaging the SIG's Slack channel
+  - bootstrapped with the slackchannel and SIG leadership from the
+    [community sig list][sig-list]
+  - optionally directly "@" mentioning SIG leadership or others by handle
+
+## Adding An Item To The Milestone
+
+### Milestone Maintainers
+
+The members of the [`milestone-maintainers`](https://github.com/orgs/kubernetes/teams/milestone-maintainers/members)
+GitHub team are entrusted with the responsibility of specifying the release
+milestone on GitHub artifacts.
+
+This group is [maintained](https://git.k8s.io/sig-release/release-team/README.md#milestone-maintainers)
+by SIG Release and has representation from the various SIGs' leadership.
+
+### Feature additions
+
+Feature planning and definition takes many forms today, but a typical example
+might be a large piece of work described in a [KEP][keps], with associated task
+issues in GitHub. When the plan has reached an implementable state and work is
+underway, the enhancement or parts thereof are targeted for an upcoming milestone
+by creating GitHub issues and marking them with the Prow "/milestone" command.
+
+For the first ~4 weeks into the release cycle, the Release Team's Enhancements
+Lead will interact with SIGs and feature owners via GitHub, Slack, and SIG
+meetings to capture all required planning artifacts.
+
+If you have an enhancement to target for an upcoming release milestone, begin a
+conversation with your SIG leadership and with that release's Enhancements
+Lead.
+
+### Issue additions
+
+Issues are marked as targeting a milestone via the Prow "/milestone" command.
+
+The Release Team's [Bug Triage Lead](https://git.k8s.io/sig-release/release-team/role-handbooks/bug-triage/README.md)
+and overall community watch incoming issues and triage them, as described in
+the contributor guide section on
+[issue triage](/contributors/guide/issue-triage.md).
+
+Marking issues with the milestone provides the community better visibility
+regarding when an issue was observed and by when the community feels it must be
+resolved. During [Code Freeze][code-freeze], a milestone must be set to merge
+a PR.
+
+An open issue is no longer required for a PR, but open issues and associated
+PRs should have synchronized labels. For example a high priority bug issue
+might not have its associated PR merged if the PR is only marked as lower
+priority.
+
+### PR Additions
+
+PRs are marked as targeting a milestone via the Prow "/milestone" command.
+
+This is a blocking requirement during Code Freeze as described above.
+
+## Other Required Labels
+
+[Here is the list of labels and their use and purpose.](https://git.k8s.io/test-infra/label_sync/labels.md#labels-that-apply-to-all-repos-for-both-issues-and-prs)
+
+### SIG Owner Label
+
+The SIG owner label defines the SIG to which we escalate if a milestone issue
+is languishing or needs additional attention. If there are no updates after
+escalation, the issue may be automatically removed from the milestone.
+
+These are added with the Prow "/sig" command. For example to add the label
+indicating SIG Storage is responsible, comment with `/sig storage`.
+
+### Priority Label
+
+Priority labels are used to determine an escalation path before moving issues
+out of the release milestone. They are also used to determine whether or not a
+release should be blocked on the resolution of the issue.
+
+- `priority/critical-urgent`: Never automatically move out of a release
+  milestone; continually escalate to contributor and SIG through all available
+  channels.
+  - considered a release blocking issue
+  - requires daily updates from issue owners during [Code Freeze][code-freeze]
+  - would require a patch release if left undiscovered until after the minor
+    release
+- `priority/important-soon`: Escalate to the issue owners and SIG owner; move
+  out of milestone after several unsuccessful escalation attempts.
+  - not considered a release blocking issue
+  - would not require a patch release
+  - will automatically be moved out of the release milestone at Code Freeze
+    after a 4 day grace period
+- `priority/important-longterm`: Escalate to the issue owners; move out of the
+  milestone after 1 attempt.
+  - even less urgent / critical than `priority/important-soon`
+  - moved out of milestone more aggressively than `priority/important-soon`
+
+### Issue/PR Kind Label
+
+The issue kind is used to help identify the types of changes going into the
+release over time. This may allow the Release Team to develop a better
+understanding of what sorts of issues we would miss with a faster release
+cadence.
+
+For release targeted issues, including pull requests, one of the following
+issue kind labels must be set:
+
+- `kind/api-change`: Adds, removes, or changes an API
+- `kind/bug`: Fixes a newly discovered bug.
+- `kind/cleanup`: Adding tests, refactoring, fixing old bugs.
+- `kind/design`: Related to design
+- `kind/documentation`: Adds documentation
+- `kind/failing-test`: CI test case is failing consistently.
+- `kind/feature`: New functionality.
+- `kind/flake`: CI test case is showing intermittent failures.
+
+[cherry-picks]: /contributors/devel/sig-release/cherry-picks.md
+[code-freeze]: https://git.k8s.io/sig-release/releases/release_phases.md#code-freeze
+[enhancements-freeze]: https://git.k8s.io/sig-release/releases/release_phases.md#enhancements-freeze
+[exceptions]: https://git.k8s.io/sig-release/releases/release_phases.md#exceptions
+[keps]: https://git.k8s.io/enhancements/keps
+[release-managers]: https://git.k8s.io/sig-release/release-managers.md
+[release-team]: https://git.k8s.io/sig-release/release-team
+[sig-list]: /sig-list.md
@@ -6,22 +6,21 @@ reviewers:
 - sig-cluster-lifecycle
 - sig-node
 - sig-release
-title: Kubernetes version and version skew support policy
-content_type: concept
-weight: 30
+title: Version Skew Policy
+type: docs
+description: >
+  The maximum version skew supported between various Kubernetes components.
 ---

 <!-- overview -->
 This document describes the maximum version skew supported between various Kubernetes components.
 Specific cluster deployment tools may place additional restrictions on version skew.

-
 <!-- body -->

 ## Supported versions

-Kubernetes versions are expressed as **x.y.z**,
-where **x** is the major version, **y** is the minor version, and **z** is the patch version, following [Semantic Versioning](https://semver.org/) terminology.
+Kubernetes versions are expressed as **x.y.z**, where **x** is the major version, **y** is the minor version, and **z** is the patch version, following [Semantic Versioning](https://semver.org/) terminology.
 For more information, see [Kubernetes Release Versioning](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#kubernetes-release-versioning).

 The Kubernetes project maintains release branches for the most recent three minor releases ({{< skew latestVersion >}}, {{< skew prevMinorVersion >}}, {{< skew oldestMinorVersion >}}).  Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.