Merge branch 'dev-1.21' into new-annotations

2021-03-19 13:43:05 +08:00
parent 352e776eed bb2f640858
commit 4c5130519c
133 changed files with 4024 additions and 1353 deletions
@@ -31,7 +31,7 @@ The [components](/docs/concepts/overview/components/#node-components) on a node
 There are two main ways to have Nodes added to the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}}:

 1. The kubelet on a node self-registers to the control plane
-2. You, or another human user, manually add a Node object
+2. You (or another human user) manually add a Node object

 After you create a Node object, or the kubelet on a node self-registers, the
 control plane checks whether the new Node object is valid. For example, if you
@@ -52,8 +52,8 @@ try to create a Node from the following JSON manifest:

 Kubernetes creates a Node object internally (the representation). Kubernetes checks
 that a kubelet has registered to the API server that matches the `metadata.name`
-field of the Node. If the node is healthy (if all necessary services are running),
-it is eligible to run a Pod. Otherwise, that node is ignored for any cluster activity
+field of the Node. If the node is healthy (i.e. all necessary services are running),
+then it is eligible to run a Pod. Otherwise, that node is ignored for any cluster activity
 until it becomes healthy.

 {{< note >}}
@@ -96,14 +96,14 @@ You can create and modify Node objects using
 When you want to create Node objects manually, set the kubelet flag `--register-node=false`.

 You can modify Node objects regardless of the setting of `--register-node`.
-For example, you can set labels on an existing Node, or mark it unschedulable.
+For example, you can set labels on an existing Node or mark it unschedulable.

 You can use labels on Nodes in conjunction with node selectors on Pods to control
 scheduling. For example, you can constrain a Pod to only be eligible to run on
 a subset of the available nodes.

 Marking a node as unschedulable prevents the scheduler from placing new pods onto
-that Node, but does not affect existing Pods on the Node. This is useful as a
+that Node but does not affect existing Pods on the Node. This is useful as a
 preparatory step before a node reboot or other maintenance.

 To mark a Node unschedulable, run:
@@ -179,14 +179,14 @@ The node condition is represented as a JSON object. For example, the following s
 ]
 ```

-If the Status of the Ready condition remains `Unknown` or `False` for longer than the `pod-eviction-timeout` (an argument passed to the {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}), all the Pods on the node are scheduled for deletion by the node controller. The default eviction timeout duration is **five minutes**. In some cases when the node is unreachable, the API server is unable to communicate with the kubelet on the node. The decision to delete the pods cannot be communicated to the kubelet until communication with the API server is re-established. In the meantime, the pods that are scheduled for deletion may continue to run on the partitioned node.
+If the Status of the Ready condition remains `Unknown` or `False` for longer than the `pod-eviction-timeout` (an argument passed to the {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}), then all the Pods on the node are scheduled for deletion by the node controller. The default eviction timeout duration is **five minutes**. In some cases when the node is unreachable, the API server is unable to communicate with the kubelet on the node. The decision to delete the pods cannot be communicated to the kubelet until communication with the API server is re-established. In the meantime, the pods that are scheduled for deletion may continue to run on the partitioned node.

 The node controller does not force delete pods until it is confirmed that they have stopped
 running in the cluster. You can see the pods that might be running on an unreachable node as
 being in the `Terminating` or `Unknown` state. In cases where Kubernetes cannot deduce from the
 underlying infrastructure if a node has permanently left a cluster, the cluster administrator
-may need to delete the node object by hand.  Deleting the node object from Kubernetes causes
-all the Pod objects running on the node to be deleted from the API server, and frees up their
+may need to delete the node object by hand. Deleting the node object from Kubernetes causes
+all the Pod objects running on the node to be deleted from the API server and frees up their
 names.

 The node lifecycle controller automatically creates
@@ -199,7 +199,7 @@ for more details.

 ### Capacity and Allocatable {#capacity}

-Describes the resources available on the node: CPU, memory and the maximum
+Describes the resources available on the node: CPU, memory, and the maximum
 number of pods that can be scheduled onto the node.

 The fields in the capacity block indicate the total amount of resources that a
@@ -225,18 +225,19 @@ CIDR block to the node when it is registered (if CIDR assignment is turned on).

 The second is keeping the node controller's internal list of nodes up to date with
 the cloud provider's list of available machines. When running in a cloud
-environment, whenever a node is unhealthy, the node controller asks the cloud
+environment and whenever a node is unhealthy, the node controller asks the cloud
 provider if the VM for that node is still available. If not, the node
 controller deletes the node from its list of nodes.

 The third is monitoring the nodes' health. The node controller is
-responsible for updating the NodeReady condition of NodeStatus to
-ConditionUnknown when a node becomes unreachable (i.e. the node controller stops
-receiving heartbeats for some reason, for example due to the node being down), and then later evicting
-all the pods from the node (using graceful termination) if the node continues
-to be unreachable. (The default timeouts are 40s to start reporting
-ConditionUnknown and 5m after that to start evicting pods.) The node controller
-checks the state of each node every `--node-monitor-period` seconds.
+responsible for:
+- Updating the NodeReady condition of NodeStatus to ConditionUnknown when a node
+  becomes unreachable, as the node controller stops receiving heartbeats for some
+  reason such as the node being down.
+- Evicting all the pods from the node using graceful termination if
+  the node continues to be unreachable. The default timeouts are 40s to start
+  reporting ConditionUnknown and 5m after that to start evicting pods.
+The node controller checks the state of each node every `--node-monitor-period` seconds.

 #### Heartbeats

@@ -252,13 +253,14 @@ of the node heartbeats as the cluster scales.
 The kubelet is responsible for creating and updating the `NodeStatus` and
 a Lease object.

- The kubelet updates the `NodeStatus` either when there is change in status,
+- The kubelet updates the `NodeStatus` either when there is change in status
  or if there has been no update for a configured interval. The default interval
-  for `NodeStatus` updates is 5 minutes (much longer than the 40 second default
-  timeout for unreachable nodes).
+  for `NodeStatus` updates is 5 minutes, which is much longer than the 40 second default
+  timeout for unreachable nodes.
 - The kubelet creates and then updates its Lease object every 10 seconds
  (the default update interval). Lease updates occur independently from the
-  `NodeStatus` updates. If the Lease update fails, the kubelet retries with exponential backoff starting at 200 milliseconds and capped at 7 seconds.
+  `NodeStatus` updates. If the Lease update fails, the kubelet retries with
+  exponential backoff starting at 200 milliseconds and capped at 7 seconds.

 #### Reliability

@@ -269,23 +271,24 @@ from more than 1 node per 10 seconds.
 The node eviction behavior changes when a node in a given availability zone
 becomes unhealthy. The node controller checks what percentage of nodes in the zone
 are unhealthy (NodeReady condition is ConditionUnknown or ConditionFalse) at
-the same time. If the fraction of unhealthy nodes is at least
-`--unhealthy-zone-threshold` (default 0.55) then the eviction rate is reduced:
-if the cluster is small (i.e. has less than or equal to
-`--large-cluster-size-threshold` nodes - default 50) then evictions are
-stopped, otherwise the eviction rate is reduced to
-`--secondary-node-eviction-rate` (default 0.01) per second. The reason these
-policies are implemented per availability zone is because one availability zone
-might become partitioned from the master while the others remain connected. If
-your cluster does not span multiple cloud provider availability zones, then
-there is only one availability zone (the whole cluster).
+the same time:
+- If the fraction of unhealthy nodes is at least `--unhealthy-zone-threshold` 
+  (default 0.55), then the eviction rate is reduced.
+- If the cluster is small (i.e. has less than or equal to
+  `--large-cluster-size-threshold` nodes - default 50), then evictions are stopped.
+- Otherwise, the eviction rate is reduced to `--secondary-node-eviction-rate`
+  (default 0.01) per second.
+The reason these policies are implemented per availability zone is because one
+availability zone might become partitioned from the master while the others remain
+connected. If your cluster does not span multiple cloud provider availability zones,
+then there is only one availability zone (i.e. the whole cluster).

 A key reason for spreading your nodes across availability zones is so that the
 workload can be shifted to healthy zones when one entire zone goes down.
-Therefore, if all nodes in a zone are unhealthy then the node controller evicts at
+Therefore, if all nodes in a zone are unhealthy, then the node controller evicts at
 the normal rate of `--node-eviction-rate`.  The corner case is when all zones are
 completely unhealthy (i.e. there are no healthy nodes in the cluster). In such a
-case, the node controller assumes that there's some problem with master
+case, the node controller assumes that there is some problem with master
 connectivity and stops all evictions until some connectivity is restored.

 The node controller is also responsible for evicting pods running on nodes with
@@ -303,8 +306,8 @@ eligible for, effectively removing incoming load balancer traffic from the cordo

 ### Node capacity

-Node objects track information about the Node's resource capacity (for example: the amount
-of memory available, and the number of CPUs).
+Node objects track information about the Node's resource capacity: for example, the amount
+of memory available and the number of CPUs.
 Nodes that [self register](#self-registration-of-nodes) report their capacity during
 registration. If you [manually](#manual-node-administration) add a Node, then
 you need to set the node's capacity information when you add it.
@@ -338,7 +341,7 @@ for more information.
 If you have enabled the `GracefulNodeShutdown` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), then the kubelet attempts to detect the node system shutdown and terminates pods running on the node.
 Kubelet ensures that pods follow the normal [pod termination process](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) during the node shutdown.

-When the `GracefulNodeShutdown` feature gate is enabled, kubelet uses [systemd inhibitor locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to delay the node shutdown with a given duration. During a shutdown kubelet terminates pods in two phases:
+When the `GracefulNodeShutdown` feature gate is enabled, kubelet uses [systemd inhibitor locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to delay the node shutdown with a given duration. During a shutdown, kubelet terminates pods in two phases:

 1. Terminate regular pods running on the node.
 2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node.
@@ -83,12 +83,15 @@ As an example, you can find detailed information about how `kube-up.sh` sets
 up logging for COS image on GCP in the corresponding
 [`configure-helper` script](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh).

+When using a **CRI container runtime**, the kubelet is responsible for rotating the logs and managing the logging directory structure. The kubelet
+sends this information to the CRI container runtime and the runtime writes the container logs to the given location. The two kubelet flags `container-log-max-size` and `container-log-max-files` can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
+
 When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
 the basic logging example, the kubelet on the node handles the request and
 reads directly from the log file. The kubelet returns the content of the log file.

 {{< note >}}
-If an external system has performed the rotation,
+If an external system has performed the rotation or a CRI container runtime is used,
 only the contents of the latest log file will be available through
 `kubectl logs`. For example, if there's a 10MB file, `logrotate` performs
 the rotation and there are two files: one file that is 10MB in size and a second file that is empty.
@@ -134,7 +134,7 @@ cloudprovider_gce_api_request_duration_seconds { request = "list_disk"}

 ### kube-scheduler metrics

-{{< feature-state for_k8s_version="v1.20" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.21" state="beta" >}}

 The scheduler exposes optional metrics that reports the requested resources and the desired limits of all running pods. These metrics can be used to build capacity planning dashboards, assess current or historical scheduling limits, quickly identify workloads that cannot schedule due to lack of resources, and compare actual usage to the pod's request.

@@ -43,7 +43,7 @@ Kubernetes objects that have a `spec`, a ConfigMap has `data` and `binaryData`
 fields. These fields accept key-value pairs as their values.  Both the `data`
 field and the `binaryData` are optional. The `data` field is designed to
 contain UTF-8 byte sequences while the `binaryData` field is designed to
-contain binary data.
+contain binary data as base64-encoded strings.

 The name of a ConfigMap must be a valid
 [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names).
@@ -81,9 +81,9 @@ The [imagePullPolicy](/docs/concepts/containers/images/#updating-images) and the

 - `imagePullPolicy: Always`: every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet downloads (pulls) the image with the resolved digest, and uses that image to launch the container.

- `imagePullPolicy` is omitted and either the image tag is `:latest` or it is omitted: `Always` is applied.
+- `imagePullPolicy` is omitted and either the image tag is `:latest` or it is omitted: `imagePullPolicy` is automatically set to `Always`. Note that this will _not_ be updated to `IfNotPresent` if the tag changes value.

- `imagePullPolicy` is omitted and the image tag is present but not `:latest`: `IfNotPresent` is applied.
+- `imagePullPolicy` is omitted and the image tag is present but not `:latest`: `imagePullPolicy` is automatically set to `IfNotPresent`. Note that this will _not_ be updated to `Always` if the tag is later removed or changed to `:latest`.

 - `imagePullPolicy: Never`: the image is assumed to exist locally. No attempt is made to pull the image.

@@ -96,7 +96,7 @@ You should avoid using the `:latest` tag when deploying containers in production
 {{< /note >}}

 {{< note >}}
-The caching semantics of the underlying image provider make even `imagePullPolicy: Always` efficient. With Docker, for example, if the image already exists, the pull attempt is fast because all image layers are cached and no image download is needed.
+The caching semantics of the underlying image provider make even `imagePullPolicy: Always` efficient, as long as the registry is reliably accessible. With Docker, for example, if the image already exists, the pull attempt is fast because all image layers are cached and no image download is needed.
 {{< /note >}}

 ## Using kubectl
@@ -49,16 +49,32 @@ Instead, specify a meaningful tag such as `v1.42.0`.

 ## Updating images

-The default pull policy is `IfNotPresent` which causes the
-{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} to skip
-pulling an image if it already exists. If you would like to always force a pull,
-you can do one of the following:
+When you first create a {{< glossary_tooltip text="Deployment" term_id="deployment" >}},
+{{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}}, Pod, or other
+object that includes a Pod template, then by default the pull policy of all
+containers in that pod will be set to `IfNotPresent` if it is not explicitly
+specified. This policy causes the
+{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} to skip pulling an
+image if it already exists.
+
+If you would like to always force a pull, you can do one of the following:

 - set the `imagePullPolicy` of the container to `Always`.
- omit the `imagePullPolicy` and use `:latest` as the tag for the image to use.
+- omit the `imagePullPolicy` and use `:latest` as the tag for the image to use;
+  Kubernetes will set the policy to `Always`.
 - omit the `imagePullPolicy` and the tag for the image to use.
 - enable the [AlwaysPullImages](/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages) admission controller.

+{{< note >}}
+The value of `imagePullPolicy` of the container is always set when the object is
+first _created_, and is not updated if the image's tag later changes.
+
+For example, if you create a Deployment with an image whose tag is _not_
+`:latest`, and later update that Deployment's image to a `:latest` tag, the
+`imagePullPolicy` field will _not_ change to `Always`. You must manually change
+the pull policy of any object after its initial creation.
+{{< /note >}}
+
 When `imagePullPolicy` is defined without a specific value, it is also set to `Always`.

 ## Multi-architecture images with image indexes
@@ -103,26 +103,27 @@ as well as keeping the existing service in good shape.
 ## Writing your own Operator {#writing-operator}

 If there isn't an Operator in the ecosystem that implements the behavior you
-want, you can code your own. In [What's next](#what-s-next) you'll find a few
-links to libraries and tools you can use to write your own cloud native
-Operator.
+want, you can code your own. 

 You also implement an Operator (that is, a Controller) using any language / runtime
 that can act as a [client for the Kubernetes API](/docs/reference/using-api/client-libraries/).

+Following are a few libraries and tools you can use to write your own cloud native
+Operator.

+{{% thirdparty-content %}}
+
+* [kubebuilder](https://book.kubebuilder.io/)
+* [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator)
+* [Metacontroller](https://metacontroller.app/) along with WebHooks that
+  you implement yourself
+* [Operator Framework](https://operatorframework.io)

 ## {{% heading "whatsnext" %}}


 * Learn more about [Custom Resources](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
 * Find ready-made operators on [OperatorHub.io](https://operatorhub.io/) to suit your use case
-* Use existing tools to write your own operator, eg:
-  * using [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator)
-  * using [kubebuilder](https://book.kubebuilder.io/)
-  * using [Metacontroller](https://metacontroller.app/) along with WebHooks that
-    you implement yourself
-  * using the [Operator Framework](https://operatorframework.io)
 * [Publish](https://operatorhub.io/) your operator for other people to use
 * Read [CoreOS' original article](https://web.archive.org/web/20170129131616/https://coreos.com/blog/introducing-operators.html) that introduced the Operator pattern (this is an archived version of the original article).
 * Read an [article](https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps) from Google Cloud about best practices for building Operators
@@ -124,6 +124,10 @@ In release 1.8, quota support for local ephemeral storage is added as an alpha f
 | `limits.ephemeral-storage` | Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value. |
 | `ephemeral-storage` | Same as `requests.ephemeral-storage`. |

+{{< note >}}
+When using a CRI container runtime, container logs will count against the ephemeral storage quota. This can result in the unexpected eviction of pods that have exhausted their storage quotas. Refer to [Logging Architecture](/docs/concepts/cluster-administration/logging/) for details.
+{{< /note >}}
+
 ## Object Count Quota

 You can set quota for the total number of certain resources of all standard,
@@ -5,7 +5,7 @@ reviewers:
 - bsalamat
 title: Assigning Pods to Nodes
 content_type: concept
-weight: 50
+weight: 20
 ---


@@ -5,7 +5,7 @@ reviewers:
 - ahg-g
 title: Resource Bin Packing for Extended Resources
 content_type: concept
-weight: 50
+weight: 30
 ---

 <!-- overview -->
@@ -80,7 +80,7 @@ parameters:
 Users request dynamically provisioned storage by including a storage class in
 their `PersistentVolumeClaim`. Before Kubernetes v1.6, this was done via the
 `volume.beta.kubernetes.io/storage-class` annotation. However, this annotation
-is deprecated since v1.6. Users now can and should instead use the
+is deprecated since v1.9. Users now can and should instead use the
 `storageClassName` field of the `PersistentVolumeClaim` object. The value of
 this field must match the name of a `StorageClass` configured by the
 administrator (see [below](#enabling-dynamic-provisioning)).
@@ -17,6 +17,7 @@ which a pod runs: network-attached storage might not be accessible by
 all nodes, or storage is local to a node to begin with.

 {{< feature-state for_k8s_version="v1.19" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.21" state="beta" >}}

 This page describes how Kubernetes keeps track of storage capacity and
 how the scheduler uses that information to schedule Pods onto nodes
@@ -103,34 +104,10 @@ to handle this automatically.

 ## Enabling storage capacity tracking

-Storage capacity tracking is an *alpha feature* and only enabled when
-the `CSIStorageCapacity` [feature
-gate](/docs/reference/command-line-tools-reference/feature-gates/) and
-the `storage.k8s.io/v1alpha1` {{< glossary_tooltip text="API group" term_id="api-group" >}} are enabled. For details on
-that, see the `--feature-gates` and `--runtime-config` [kube-apiserver
-parameters](/docs/reference/command-line-tools-reference/kube-apiserver/).
-
-A quick check
-whether a Kubernetes cluster supports the feature is to list
-CSIStorageCapacity objects with:
-```shell
-kubectl get csistoragecapacities --all-namespaces
-```
-
-If your cluster supports CSIStorageCapacity, the response is either a list of CSIStorageCapacity objects or:
-```
-No resources found
-```
-
-If not supported, this error is printed instead:
-```
-error: the server doesn't have a resource type "csistoragecapacities"
-```
-
-In addition to enabling the feature in the cluster, a CSI
-driver also has to
-support it. Please refer to the driver's documentation for
-details.
+Storage capacity tracking is a beta feature and enabled by default in
+a Kubernetes cluster since Kubernetes 1.21. In addition to having the
+feature enabled in the cluster, a CSI driver also has to support
+it. Please refer to the driver's documentation for details.

 ## {{% heading "whatsnext" %}}

@@ -34,8 +34,9 @@ Kubernetes supports many types of volumes. A {{< glossary_tooltip term_id="pod"
 can use any number of volume types simultaneously.
 Ephemeral volume types have a lifetime of a pod, but persistent volumes exist beyond
 the lifetime of a pod. Consequently, a volume outlives any containers
-that run within the pod, and data is preserved across container restarts. When a
-pod ceases to exist, the volume is destroyed.
+that run within the pod, and data is preserved across container restarts. When a pod
+ceases to exist, Kubernetes destroys ephemeral volumes; however, Kubernetes does not
+destroy persistent volumes.

 At its core, a volume is just a directory, possibly with some data in it, which
 is accessible to the containers in a pod. How that directory comes to be, the
@@ -152,14 +153,16 @@ For more details, see the [`azureFile` volume plugin](https://github.com/kuberne

 #### azureFile CSI migration

-{{< feature-state for_k8s_version="v1.15" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.21" state="beta" >}}

 The `CSIMigration` feature for `azureFile`, when enabled, redirects all plugin operations
 from the existing in-tree plugin to the `file.csi.azure.com` Container
 Storage Interface (CSI) Driver. In order to use this feature, the [Azure File CSI
 Driver](https://github.com/kubernetes-sigs/azurefile-csi-driver)
 must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureFile`
-alpha features must be enabled.
+[feature gates](/docs/reference/command-line-tools-reference/feature-gates/) must be enabled.
+
+Azure File CSI driver does not support using same volume with different fsgroups, if Azurefile CSI migration is enabled, using same volume with different fsgroups won't be supported at all.

 ### cephfs

@@ -54,7 +54,7 @@ In this example:
  {{< note >}}
  The `.spec.selector.matchLabels` field is a map of {key,value} pairs.
  A single {key,value} in the `matchLabels` map is equivalent to an element of `matchExpressions`,
-  whose key field is "key" the operator is "In", and the values array contains only "value".
+  whose `key` field is "key", the `operator` is "In", and the `values` array contains only "value".
  All of the requirements, from both `matchLabels` and `matchExpressions`, must be satisfied in order to match.
  {{< /note >}}

@@ -145,8 +145,8 @@ There are three main types of task suitable to run as a Job:
   - the Job is complete as soon as its Pod terminates successfully.
 1. Parallel Jobs with a *fixed completion count*:
   - specify a non-zero positive value for `.spec.completions`.
-   - the Job represents the overall task, and is complete when there is one successful Pod for each value in the range 1 to `.spec.completions`.
-   - **not implemented yet:** Each Pod is passed a different index in the range 1 to `.spec.completions`.
+   - the Job represents the overall task, and is complete when there are `.spec.completions` successful Pods.
+   - when using `.spec.completionMode="Indexed"`, each Pod gets a different index in the range 0 to `.spec.completions-1`.
 1. Parallel Jobs with a *work queue*:
   - do not specify `.spec.completions`, default to `.spec.parallelism`.
   - the Pods must coordinate amongst themselves or an external service to determine what each should work on. For example, a Pod might fetch a batch of up to N items from the work queue.
@@ -166,7 +166,6 @@ a non-negative integer.

 For more information about how to make use of the different types of job, see the [job patterns](#job-patterns) section.

-
 #### Controlling parallelism

 The requested parallelism (`.spec.parallelism`) can be set to any non-negative value.
@@ -185,6 +184,33 @@ parallelism, for a variety of reasons:
 - The Job controller may throttle new Pod creation due to excessive previous pod failures in the same Job.
 - When a Pod is gracefully shut down, it takes time to stop.

+### Completion mode
+
+{{< feature-state for_k8s_version="v1.21" state="alpha" >}}
+
+{{< note >}}
+To be able to create Indexed Jobs, make sure to enable the `IndexedJob`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+on the [API server](docs/reference/command-line-tools-reference/kube-apiserver/)
+and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/).
+{{< /note >}}
+
+Jobs with _fixed completion count_ - that is, jobs that have non null
+`.spec.completions` - can have a completion mode that is specified in `.spec.completionMode`:
+
+- `NonIndexed` (default): the Job is considered complete when there have been
+  `.spec.completions` successfully completed Pods. In other words, each Pod
+  completion is homologous to each other. Note that Jobs that have null
+  `.spec.completions` are implicitly `NonIndexed`.
+- `Indexed`: the Pods of a Job get an associated completion index from 0 to
+  `.spec.completions-1`, available in the annotation `batch.kubernetes.io/job-completion-index`.
+  The Job is considered complete when there is one successfully completed Pod
+  for each index. For more information about how to use this mode, see
+  [Indexed Job for Parallel Processing with Static Work Assignment](/docs/tasks/job/indexed-parallel-processing-static/).
+  Note that, although rare, more than one Pod could be started for the same
+  index, but only one of them will count towards the completion count.
+
+
 ## Handling Pod and container failures

 A container in a Pod may fail for a number of reasons, such as because the process in it exited with
@@ -348,12 +374,12 @@ The tradeoffs are:
 The tradeoffs are summarized here, with columns 2 to 4 corresponding to the above tradeoffs.
 The pattern names are also links to examples and more detailed description.

-|                            Pattern                                   | Single Job object | Fewer pods than work items? | Use app unmodified? |  Works in Kube 1.1? |
-| -------------------------------------------------------------------- |:-----------------:|:---------------------------:|:-------------------:|:-------------------:|
-| [Job Template Expansion](/docs/tasks/job/parallel-processing-expansion/)            |                   |                             |          ✓          |          ✓          |
-| [Queue with Pod Per Work Item](/docs/tasks/job/coarse-parallel-processing-work-queue/)   |         ✓         |                             |      sometimes      |          ✓          |
-| [Queue with Variable Pod Count](/docs/tasks/job/fine-parallel-processing-work-queue/)  |         ✓         |             ✓               |                     |          ✓          |
-| Single Job with Static Work Assignment                               |         ✓         |                             |          ✓          |                     |
+|                  Pattern                  | Single Job object | Fewer pods than work items? | Use app unmodified? |
+| ----------------------------------------- |:-----------------:|:---------------------------:|:-------------------:|
+| [Queue with Pod Per Work Item]            |         ✓         |                             |      sometimes      |
+| [Queue with Variable Pod Count]           |         ✓         |             ✓               |                     |
+| [Indexed Job with Static Work Assignment] |         ✓         |                             |          ✓          | 
+| [Job Template Expansion]                  |                   |                             |          ✓          |

 When you specify completions with `.spec.completions`, each Pod created by the Job controller
 has an identical [`spec`](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status).  This means that
@@ -364,13 +390,17 @@ are different ways to arrange for pods to work on different things.
 This table shows the required settings for `.spec.parallelism` and `.spec.completions` for each of the patterns.
 Here, `W` is the number of work items.

-|                             Pattern                                  | `.spec.completions` |  `.spec.parallelism` |
-| -------------------------------------------------------------------- |:-------------------:|:--------------------:|
-| [Job Template Expansion](/docs/tasks/job/parallel-processing-expansion/)           |          1          |     should be 1      |
-| [Queue with Pod Per Work Item](/docs/tasks/job/coarse-parallel-processing-work-queue/)   |          W          |        any           |
-| [Queue with Variable Pod Count](/docs/tasks/job/fine-parallel-processing-work-queue/)  |          1          |        any           |
-| Single Job with Static Work Assignment                               |          W          |        any           |
+|             Pattern                       | `.spec.completions` |  `.spec.parallelism` |
+| ----------------------------------------- |:-------------------:|:--------------------:|
+| [Queue with Pod Per Work Item]            |          W          |        any           |
+| [Queue with Variable Pod Count]           |         null        |        any           |
+| [Indexed Job with Static Work Assignment] |          W          |        any           |
+| [Job Template Expansion]                  |          1          |     should be 1      |

+[Queue with Pod Per Work Item]: /docs/tasks/job/coarse-parallel-processing-work-queue/
+[Queue with Variable Pod Count]: /docs/tasks/job/fine-parallel-processing-work-queue/
+[Indexed Job with Static Work Assignment]: /docs/tasks/job/indexed-parallel-processing-static/
+[Job Template Expansion]: /docs/tasks/job/parallel-processing-expansion/

 ## Advanced usage

@@ -62,8 +62,6 @@ different Kubernetes components.
 | `BoundServiceAccountTokenVolume` | `false` | Alpha | 1.13 | |
 | `CPUManager` | `false` | Alpha | 1.8 | 1.9 |
 | `CPUManager` | `true` | Beta | 1.10 | |
-| `CRIContainerLogRotation` | `false` | Alpha | 1.10 | 1.10 |
-| `CRIContainerLogRotation` | `true` | Beta| 1.11 | |
 | `CSIInlineVolume` | `false` | Alpha | 1.15 | 1.15 |
 | `CSIInlineVolume` | `true` | Beta | 1.16 | - |
 | `CSIMigration` | `false` | Alpha | 1.14 | 1.16 |
@@ -74,7 +72,8 @@ different Kubernetes components.
 | `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | 1.18 |
 | `CSIMigrationAzureDisk` | `false` | Beta | 1.19 | |
 | `CSIMigrationAzureDiskComplete` | `false` | Alpha | 1.17 | |
-| `CSIMigrationAzureFile` | `false` | Alpha | 1.15 | |
+| `CSIMigrationAzureFile` | `false` | Alpha | 1.15 | 1.19 |
+| `CSIMigrationAzureFile` | `false` | Beta | 1.21 | |
 | `CSIMigrationAzureFileComplete` | `false` | Alpha | 1.17 | |
 | `CSIMigrationGCE` | `false` | Alpha | 1.14 | 1.16 |
 | `CSIMigrationGCE` | `false` | Beta | 1.17 | |
@@ -85,7 +84,8 @@ different Kubernetes components.
 | `CSIMigrationvSphere` | `false` | Beta | 1.19 | |
 | `CSIMigrationvSphereComplete` | `false` | Beta | 1.19 | |
 | `CSIServiceAccountToken` | `false` | Alpha | 1.20 | |
-| `CSIStorageCapacity` | `false` | Alpha | 1.19 | |
+| `CSIStorageCapacity` | `false` | Alpha | 1.19 | 1.20 |
+| `CSIStorageCapacity` | `true` | Beta | 1.21 | |
 | `CSIVolumeFSGroupPolicy` | `false` | Alpha | 1.19 | 1.19 |
 | `CSIVolumeFSGroupPolicy` | `true` | Beta | 1.20 | |
 | `ConfigurableFSGroupPolicy` | `false` | Alpha | 1.18 | 1.19 |
@@ -199,6 +199,9 @@ different Kubernetes components.
 | `BlockVolume` | `false` | Alpha | 1.9 | 1.12 |
 | `BlockVolume` | `true` | Beta | 1.13 | 1.17 |
 | `BlockVolume` | `true` | GA | 1.18 | - |
+| `CRIContainerLogRotation` | `false` | Alpha | 1.10 | 1.10 |
+| `CRIContainerLogRotation` | `true` | Beta | 1.11 | 1.20 |
+| `CRIContainerLogRotation` | `true` | GA | 1.21 | - |
 | `CSIBlockVolume` | `false` | Alpha | 1.11 | 1.13 |
 | `CSIBlockVolume` | `true` | Beta | 1.14 | 1.17 |
 | `CSIBlockVolume` | `true` | GA | 1.18 | - |
@@ -260,6 +263,7 @@ different Kubernetes components.
 | `ImmutableEphemeralVolumes` | `false` | Alpha | 1.18 | 1.18 |
 | `ImmutableEphemeralVolumes` | `true` | Beta | 1.19 | 1.20 |
 | `ImmutableEphemeralVolumes` | `true` | GA | 1.21 | |
+| `IndexedJob` | `false` | Alpha | 1.21 | |
 | `Initializers` | `false` | Alpha | 1.7 | 1.13 |
 | `Initializers` | - | Deprecated | 1.14 | - |
 | `KubeletConfigFile` | `false` | Alpha | 1.8 | 1.9 |
@@ -449,7 +453,9 @@ Each feature gate is designed for enabling/disabling a specific feature:
   for more details.
 - `CPUManager`: Enable container level CPU affinity support, see
  [CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/).
- `CRIContainerLogRotation`: Enable container log rotation for cri container runtime.
+- `CRIContainerLogRotation`: Enable container log rotation for CRI container runtime. The default max size of a log file is 10MB and the
+  default max number of log files allowed for a container is 5. These values can be configured in the kubelet config.
+  See the [logging at node level](/docs/concepts/cluster-administration/logging/#logging-at-the-node-level) documentation for more details.
 - `CSIBlockVolume`: Enable external CSI volume drivers to support block storage.
  See the [`csi` raw block volume support](/docs/concepts/storage/volumes/#csi-raw-block-volume-support)
  documentation for more details.
@@ -629,10 +635,12 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `HyperVContainer`: Enable
  [Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container)
  for Windows containers.
- `IPv6DualStack`: Enable [dual stack](/docs/concepts/services-networking/dual-stack/)
-  support for IPv6.
 - `ImmutableEphemeralVolumes`: Allows for marking individual Secrets and ConfigMaps as
  immutable for better safety and performance.
+- `IndexedJob`: Allows the [Job](/docs/concepts/workloads/controllers/job/)
+  controller to manage Pod completions per completion index.
+- `IPv6DualStack`: Enable [dual stack](/docs/concepts/services-networking/dual-stack/)
+  support for IPv6.
 - `KubeletConfigFile` (*deprecated*): Enable loading kubelet configuration from
  a file specified using a config file.
  See [setting kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/)
@@ -737,7 +745,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
  [ServiceTopology](/docs/concepts/services-networking/service-topology/)
  for more details.
 - `SizeMemoryBackedVolumes`: Enables kubelet support to size memory backed volumes.
-  See [volumes](docs/concepts/storage/volumes) for more details.
+  See [volumes](/docs/concepts/storage/volumes) for more details.
 - `SetHostnameAsFQDN`: Enable the ability of setting Fully Qualified Domain
  Name(FQDN) as the hostname of a pod. See
  [Pod's `setHostnameAsFQDN` field](/docs/concepts/services-networking/dns-pod-service/#pod-sethostnameasfqdn-field).
@@ -224,14 +224,14 @@ kubelet [flags]
 <td colspan="2">--container-log-max-files int32&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 5</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;">&lt;Warning: Beta feature&gt; Set the maximum number of container log files that can be present for a container. The number must be &ge; 2. This flag can only be used with `--container-runtime=remote`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;">Set the maximum number of container log files that can be present for a container. The number must be &ge; 2. This flag can only be used with `--container-runtime=remote`. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
 </tr>

 <tr>
 <td colspan="2">--container-log-max-size string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: `10Mi`</td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;">&lt;Warning: Beta feature&gt; Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with `--container-runtime=remote`.  (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;">Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with `--container-runtime=remote`.  (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
 </tr>

 <tr>
@@ -298,13 +298,6 @@ kubelet [flags]
 <td></td><td style="line-height: 130%; word-wrap: break-word;">The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. The `DynamicKubeletConfig` feature gate must be enabled to pass this flag; this gate currently defaults to `true` because the feature is beta.</td>
 </tr>

-<tr>
-<td colspan="2">--enable-cadvisor-json-endpoints&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: `false`</td>
-</tr>
-<tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;">Enable cAdvisor json `/spec` and `/stats/*` endpoints. This flag has no effect on the /stats/summary endpoint. (DEPRECATED: will be removed in a future version)</td>
-</tr>
-
 <tr>
 <td colspan="2">--enable-controller-attach-detach&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: `true`</td>
 </tr>
@@ -462,7 +455,6 @@ AppArmor=true|false (BETA - default=true)<br/>
 BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>
 BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>
 CPUManager=true|false (BETA - default=true)<br/>
-CRIContainerLogRotation=true|false (BETA - default=true)<br/>
 CSIInlineVolume=true|false (BETA - default=true)<br/>
 CSIMigration=true|false (BETA - default=true)<br/>
 CSIMigrationAWS=true|false (BETA - default=false)<br/>
@@ -19,7 +19,7 @@ files by setting the KUBECONFIG environment variable or by setting the
 This overview covers `kubectl` syntax, describes the command operations, and provides common examples.
 For details about each command, including all the supported flags and subcommands, see the
 [kubectl](/docs/reference/generated/kubectl/kubectl-commands/) reference documentation.
-For installation instructions see [installing kubectl](/docs/tasks/tools/install-kubectl/).
+For installation instructions see [installing kubectl](/docs/tasks/tools/).

 <!-- body -->

@@ -198,6 +198,15 @@ The kubelet can set this annotation on a Node to denote its configured IPv4 addr

 When kubelet is started with the "external" cloud provider, it sets this annotation on the Node to denote an IP address set from the command line flag (`--node-ip`). This IP is verified with the cloud provider as valid by the cloud-controller-manager.

+## batch.kubernetes.io/job-completion-index
+
+Example: `batch.kubernetes.io/job-completion-index: "3"`
+
+Used on: Pod
+
+The Job controller in the kube-controller-manager sets this annotation for Pods
+created with Indexed [completion mode](/docs/concepts/workloads/controllers/job/#completion-mode).
+
 ## kubectl.kubernetes.io/default-container

 Example: `kubectl.kubernetes.io/default-container: "front-end-app"`
@@ -181,8 +181,6 @@ that are not enabled by default:
 - `RequestedToCapacityRatio`: Favor nodes according to a configured function of
  the allocated resources.
  Extension points: `Score`.
- `NodeResourceLimits`: Favors nodes that satisfy the Pod resource limits.
-  Extension points: `PreScore`, `Score`.
 - `CinderVolume`: Checks that OpenStack Cinder volume limits can be satisfied
  for the node.
  Extension points: `Filter`.
@@ -9,7 +9,7 @@ weight: 40
 <!-- overview -->

 Kubernetes requires PKI certificates for authentication over TLS.
-If you install Kubernetes with [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/), the certificates that your cluster requires are automatically generated.
+If you install Kubernetes with [kubeadm](/docs/reference/setup-tools/kubeadm/), the certificates that your cluster requires are automatically generated.
 You can also generate your own certificates -- for example, to keep your private keys more secure by not storing them on the API server.
 This page explains the certificates that your cluster requires.

@@ -74,7 +74,7 @@ Required certificates:
 | kube-apiserver-kubelet-client | kubernetes-ca             | system:masters | client                                 |                                             |
 | front-proxy-client            | kubernetes-front-proxy-ca |                | client                                 |                                             |

-[1]: any other IP or DNS name you contact your cluster on (as used by [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/)
+[1]: any other IP or DNS name you contact your cluster on (as used by [kubeadm](/docs/reference/setup-tools/kubeadm/)
 the load balancer stable IP and/or DNS name, `kubernetes`, `kubernetes.default`, `kubernetes.default.svc`,
 `kubernetes.default.svc.cluster`, `kubernetes.default.svc.cluster.local`)

@@ -100,7 +100,7 @@ For kubeadm users only:

 ### Certificate paths

-Certificates should be placed in a recommended path (as used by [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/)).
+Certificates should be placed in a recommended path (as used by [kubeadm](/docs/reference/setup-tools/kubeadm/)).
 Paths should be specified using the given argument regardless of location.

 | Default CN                   | recommended key path         | recommended cert path       | command        | key argument                 | cert argument                             |
@@ -59,7 +59,7 @@ When nodes start up, the kubelet on each node automatically adds
 {{< glossary_tooltip text="labels" term_id="label" >}} to the Node object
 that represents that specific kubelet in the Kubernetes API.
 These labels can include
-[zone information](/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone).
+[zone information](/docs/reference/labels-annotations-taints/#topologykubernetesiozone).

 If your cluster spans multiple zones or regions, you can use node labels
 in conjunction with
@@ -63,7 +63,7 @@ configuration, or reinstall it using automation.

 ### containerd

-This section contains the necessary steps to use `containerd` as CRI runtime.
+This section contains the necessary steps to use containerd as CRI runtime.

 Use the following commands to install Containerd on your system:

@@ -92,170 +92,62 @@ sudo sysctl --system
 Install containerd:

 {{< tabs name="tab-cri-containerd-installation" >}}
-{{% tab name="Ubuntu 16.04" %}}
+{{% tab name="Linux" %}}

-```shell
-# (Install containerd)
-## Set up the repository
-### Install packages to allow apt to use a repository over HTTPS
-sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
-```
+1. Install the `containerd.io` package from the official Docker repositories. Instructions for setting up the Docker repository for your respective Linux distribution and installing the `containerd.io` package can be found at [Install Docker Engine](https://docs.docker.com/engine/install/#server).

-```shell
-## Add Docker's official GPG key
-curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
-```
+2. Configure containerd:

-```shell
-## Add Docker apt repository.
-sudo add-apt-repository \
-    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
-    $(lsb_release -cs) \
-    stable"
-```
+   ```shell
+   sudo mkdir -p /etc/containerd
+   containerd config default | sudo tee /etc/containerd/config.toml
+   ```

-```shell
-## Install containerd
-sudo apt-get update && sudo apt-get install -y containerd.io
-```
+3. Restart containerd:

-```shell
-# Configure containerd
-sudo mkdir -p /etc/containerd
-containerd config default | sudo tee /etc/containerd/config.toml
-```
+   ```shell
+   sudo systemctl restart containerd
+   ```

-```shell
-# Restart containerd
-sudo systemctl restart containerd
-```
-{{% /tab %}}
-{{% tab name="Ubuntu 18.04/20.04" %}}
-
-```shell
-# (Install containerd)
-sudo apt-get update && sudo apt-get install -y containerd
-```
-
-```shell
-# Configure containerd
-sudo mkdir -p /etc/containerd
-containerd config default | sudo tee /etc/containerd/config.toml
-```
-
-```shell
-# Restart containerd
-sudo systemctl restart containerd
-```
-{{% /tab %}}
-{{% tab name="Debian 9+" %}}
-
-```shell
-# (Install containerd)
-## Set up the repository
-### Install packages to allow apt to use a repository over HTTPS
-sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
-```
-
-```shell
-## Add Docker's official GPG key
-curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
-```
-
-```shell
-## Add Docker apt repository.
-sudo add-apt-repository \
-   "deb [arch=amd64] https://download.docker.com/linux/debian \
-   $(lsb_release -cs) \
-   stable"
-```
-
-```shell
-## Install containerd
-sudo apt-get update && sudo apt-get install -y containerd.io
-```
-
-```shell
-# Set default containerd configuration
-sudo mkdir -p /etc/containerd
-containerd config default | sudo tee /etc/containerd/config.toml
-```
-
-```shell
-# Restart containerd
-sudo systemctl restart containerd
-```
-{{% /tab %}}
-{{% tab name="CentOS/RHEL 7.4+" %}}
-
-```shell
-# (Install containerd)
-## Set up the repository
-### Install required packages
-sudo yum install -y yum-utils device-mapper-persistent-data lvm2
-```
-
-```shell
-## Add docker repository
-sudo yum-config-manager \
-    --add-repo \
-    https://download.docker.com/linux/centos/docker-ce.repo
-```
-
-```shell
-## Install containerd
-sudo yum update -y && sudo yum install -y containerd.io
-```
-
-```shell
-## Configure containerd
-sudo mkdir -p /etc/containerd
-containerd config default | sudo tee /etc/containerd/config.toml
-```
-
-```shell
-# Restart containerd
-sudo systemctl restart containerd
-```
 {{% /tab %}}
 {{% tab name="Windows (PowerShell)" %}}

-<br />
 Start a Powershell session, set `$Version` to the desired version (ex: `$Version=1.4.3`), and then run the following commands:
-<br />

-```powershell
-# (Install containerd)
-# Download containerd
-curl.exe -L https://github.com/containerd/containerd/releases/download/v$Version/containerd-$Version-windows-amd64.tar.gz -o containerd-windows-amd64.tar.gz
-tar.exe xvf .\containerd-windows-amd64.tar.gz
-```
+1. Download containerd:

-```powershell
-# Extract and configure
-Copy-Item -Path ".\bin\" -Destination "$Env:ProgramFiles\containerd" -Recurse -Force
-cd $Env:ProgramFiles\containerd\
-.\containerd.exe config default | Out-File config.toml -Encoding ascii
+   ```powershell
+   curl.exe -L https://github.com/containerd/containerd/releases/download/v$Version/containerd-$Version-windows-amd64.tar.gz -o containerd-windows-amd64.tar.gz
+   tar.exe xvf .\containerd-windows-amd64.tar.gz
+   ```

-# Review the configuration. Depending on setup you may want to adjust:
-# - the sandbox_image (Kubernetes pause image)
-# - cni bin_dir and conf_dir locations
-Get-Content config.toml
+2. Extract and configure:

-# (Optional - but highly recommended) Exclude containerd form Windows Defender Scans
-Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe" 
-```
+   ```powershell
+   Copy-Item -Path ".\bin\" -Destination "$Env:ProgramFiles\containerd" -Recurse -Force
+   cd $Env:ProgramFiles\containerd\
+   .\containerd.exe config default | Out-File config.toml -Encoding ascii

-```powershell
-# Start containerd
-.\containerd.exe --register-service
-Start-Service containerd
-```
+   # Review the configuration. Depending on setup you may want to adjust:
+   # - the sandbox_image (Kubernetes pause image)
+   # - cni bin_dir and conf_dir locations
+   Get-Content config.toml
+
+   # (Optional - but highly recommended) Exclude containerd from Windows Defender Scans
+   Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
+   ```
+
+3. Start containerd:
+
+   ```powershell
+   .\containerd.exe --register-service
+   Start-Service containerd
+   ```

 {{% /tab %}}
 {{< /tabs >}}

-#### systemd {#containerd-systemd}
+#### Using the `systemd` cgroup driver {#containerd-systemd}

 To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc`, set

@@ -266,6 +158,12 @@ To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc`,
    SystemdCgroup = true
 ```

+If you apply this change make sure to restart containerd again:
+
+```shell
+sudo systemctl restart containerd
+```
+
 When using kubeadm, manually configure the
 [cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node).

@@ -455,138 +353,38 @@ in sync.

 ### Docker

-On each of your nodes, install Docker CE.
+1. On each of your nodes, install the Docker for your Linux distribution as per [Install Docker Engine](https://docs.docker.com/engine/install/#server). You can find the latest validated version of Docker in this [dependencies](https://git.k8s.io/kubernetes/build/dependencies.yaml) file.

-The Kubernetes release notes list which versions of Docker are compatible
-with that version of Kubernetes.
+2. Configure the Docker daemon, in particular to use systemd for the management of the container’s cgroups.

-Use the following commands to install Docker on your system:
+   ```shell
+   sudo mkdir /etc/docker
+   cat <<EOF | sudo tee /etc/docker/daemon.json
+   {
+     "exec-opts": ["native.cgroupdriver=systemd"],
+     "log-driver": "json-file",
+     "log-opts": {
+       "max-size": "100m"
+     },
+     "storage-driver": "overlay2"
+   }
+   EOF
+   ```

-{{< tabs name="tab-cri-docker-installation" >}}
-{{% tab name="Ubuntu 16.04+" %}}
+   {{< note >}}
+   `overlay2` is the preferred storage driver for systems running Linux kernel version 4.0 or higher, or RHEL or CentOS using version 3.10.0-514 and above.
+   {{< /note >}}

-```shell
-# (Install Docker CE)
-## Set up the repository:
-### Install packages to allow apt to use a repository over HTTPS
-sudo apt-get update && sudo apt-get install -y \
-  apt-transport-https ca-certificates curl software-properties-common gnupg2
-```
+3. Restart Docker and enable on boot:

-```shell
-# Add Docker's official GPG key:
-curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
-```
+   ```shell
+   sudo systemctl enable docker
+   sudo systemctl daemon-reload
+   sudo systemctl restart docker
+   ```

-```shell
-# Add the Docker apt repository:
-sudo add-apt-repository \
-  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
-  $(lsb_release -cs) \
-  stable"
-```
-
-```shell
-# Install Docker CE
-sudo apt-get update && sudo apt-get install -y \
-  containerd.io=1.2.13-2 \
-  docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \
-  docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs)
-```
-
-```shell
-## Create /etc/docker
-sudo mkdir /etc/docker
-```
-
-```shell
-# Set up the Docker daemon
-cat <<EOF | sudo tee /etc/docker/daemon.json
-{
-  "exec-opts": ["native.cgroupdriver=systemd"],
-  "log-driver": "json-file",
-  "log-opts": {
-    "max-size": "100m"
-  },
-  "storage-driver": "overlay2"
-}
-EOF
-```
-
-```shell
-# Create /etc/systemd/system/docker.service.d
-sudo mkdir -p /etc/systemd/system/docker.service.d
-```
-
-```shell
-# Restart Docker
-sudo systemctl daemon-reload
-sudo systemctl restart docker
-```
-{{% /tab %}}
-{{% tab name="CentOS/RHEL 7.4+" %}}
-
-```shell
-# (Install Docker CE)
-## Set up the repository
-### Install required packages
-sudo yum install -y yum-utils device-mapper-persistent-data lvm2
-```
-
-```shell
-## Add the Docker repository
-sudo yum-config-manager --add-repo \
-  https://download.docker.com/linux/centos/docker-ce.repo
-```
-
-```shell
-# Install Docker CE
-sudo yum update -y && sudo yum install -y \
-  containerd.io-1.2.13 \
-  docker-ce-19.03.11 \
-  docker-ce-cli-19.03.11
-```
-
-```shell
-## Create /etc/docker
-sudo mkdir /etc/docker
-```
-
-```shell
-# Set up the Docker daemon
-cat <<EOF | sudo tee /etc/docker/daemon.json
-{
-  "exec-opts": ["native.cgroupdriver=systemd"],
-  "log-driver": "json-file",
-  "log-opts": {
-    "max-size": "100m"
-  },
-  "storage-driver": "overlay2",
-  "storage-opts": [
-    "overlay2.override_kernel_check=true"
-  ]
-}
-EOF
-```
-
-```shell
-# Create /etc/systemd/system/docker.service.d
-sudo mkdir -p /etc/systemd/system/docker.service.d
-```
-
-```shell
-# Restart Docker
-sudo systemctl daemon-reload
-sudo systemctl restart docker
-```
-{{% /tab %}}
-{{< /tabs >}}
-
-If you want the `docker` service to start on boot, run the following command:
-
-```shell
-sudo systemctl enable docker
-```
-
-Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/)
-for more information.
+{{< note >}}
+For more information refer to
+  - [Configure the Docker daemon](https://docs.docker.com/config/daemon/)
+  - [Control Docker with systemd](https://docs.docker.com/config/daemon/systemd/)
+{{< /note >}}
@@ -23,7 +23,7 @@ kops is an automated provisioning system:
 ## {{% heading "prerequisites" %}}


-* You must have [kubectl](/docs/tasks/tools/install-kubectl/) installed.
+* You must have [kubectl](/docs/tasks/tools/) installed.

 * You must [install](https://github.com/kubernetes/kops#installing) `kops` on a 64-bit (AMD64 and Intel 64) device architecture.

@@ -137,7 +137,7 @@ is not supported by kubeadm.

 ### More information

-For more information about `kubeadm init` arguments, see the [kubeadm reference guide](/docs/reference/setup-tools/kubeadm/kubeadm/).
+For more information about `kubeadm init` arguments, see the [kubeadm reference guide](/docs/reference/setup-tools/kubeadm/).

 To configure `kubeadm init` with a configuration file see [Using kubeadm init with a configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file).

@@ -160,7 +160,7 @@ kubelet and the control plane is supported, but the kubelet version may never ex
 server version. For example, the kubelet running 1.7.0 should be fully compatible with a 1.8.0 API server,
 but not vice versa.

-For information about installing `kubectl`, see [Install and set up kubectl](/docs/tasks/tools/install-kubectl/).
+For information about installing `kubectl`, see [Install and set up kubectl](/docs/tasks/tools/).

 {{< warning >}}
 These instructions exclude all Kubernetes packages from any system upgrades.
@@ -175,16 +175,34 @@ For more information on version skews, see:

 {{< tabs name="k8s_install" >}}
 {{% tab name="Debian-based distributions" %}}
-```bash
-sudo apt-get update && sudo apt-get install -y apt-transport-https curl
-curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
-cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
-deb https://apt.kubernetes.io/ kubernetes-xenial main
-EOF
-sudo apt-get update
-sudo apt-get install -y kubelet kubeadm kubectl
-sudo apt-mark hold kubelet kubeadm kubectl
-```
+
+1. Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:
+
+   ```shell
+   sudo apt-get update
+   sudo apt-get install -y apt-transport-https ca-certificates curl
+   ```
+
+2. Download the Google Cloud public signing key:
+
+   ```shell
+   sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
+   ```
+
+3. Add the Kubernetes `apt` repository:
+
+   ```shell
+   echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
+   ```
+
+4. Update `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:
+
+   ```shell
+   sudo apt-get update
+   sudo apt-get install -y kubelet kubeadm kubectl
+   sudo apt-mark hold kubelet kubeadm kubectl
+   ```
+
 {{% /tab %}}
 {{% tab name="Red Hat-based distributions" %}}
 ```bash
@@ -23,7 +23,7 @@ Kubespray is a composition of [Ansible](https://docs.ansible.com/) playbooks, [i
 * continuous integration tests

 To choose a tool which best fits your use case, read [this comparison](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/comparisons.md) to
-[kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) and [kops](/docs/setup/production-environment/tools/kops/).
+[kubeadm](/docs/reference/setup-tools/kubeadm/) and [kops](/docs/setup/production-environment/tools/kops/).

 <!-- body -->

@@ -221,7 +221,7 @@ On Windows, you can use the following settings to configure Services and load ba

 #### IPv4/IPv6 dual-stack

-You can enable IPv4/IPv6 dual-stack networking for `l2bridge` networks using the `IPv6DualStack` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/). See [enable IPv4/IPv6 dual stack](/docs/concepts/services-networking/dual-stack#enable-ipv4ipv6-dual-stack) for more details.
+You can enable IPv4/IPv6 dual-stack networking for `l2bridge` networks using the `IPv6DualStack` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). See [enable IPv4/IPv6 dual stack](/docs/concepts/services-networking/dual-stack#enable-ipv4ipv6-dual-stack) for more details.

 {{< note >}}
 On Windows, using IPv6 with Kubernetes require Windows Server, version 2004 (kernel version 10.0.19041.610) or later.
@@ -237,7 +237,7 @@ Overlay (VXLAN) networks on Windows do not support dual-stack networking today.

 Windows is only supported as a worker node in the Kubernetes architecture and component matrix. This means that a Kubernetes cluster must always include Linux master nodes, zero or more Linux worker nodes, and zero or more Windows worker nodes.

-#### Compute {compute-limitations}
+#### Compute {#compute-limitations}

 ##### Resource management and process isolation

@@ -297,7 +297,7 @@ As a result, the following storage functionality is not supported on Windows nod
 * NFS based storage/volume support
 * Expanding the mounted volume (resizefs)

-#### Networking {networking-limitations}
+#### Networking {#networking-limitations}

 Windows Container Networking differs in some important ways from Linux networking. The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) contains additional details and background.

@@ -1906,7 +1906,7 @@ filename | sha512 hash
 - Promote SupportNodePidsLimit to GA to provide node to pod pid isolation
  Promote SupportPodPidsLimit to GA to provide ability to limit pids per pod ([#94140](https://github.com/kubernetes/kubernetes/pull/94140), [@derekwaynecarr](https://github.com/derekwaynecarr)) [SIG Node and Testing]
 - Rename pod_preemption_metrics to preemption_metrics. ([#93256](https://github.com/kubernetes/kubernetes/pull/93256), [@ahg-g](https://github.com/ahg-g)) [SIG Instrumentation and Scheduling]
- Server-side apply behavior has been regularized in the case where a field is removed from the applied configuration. Removed fields which have no other owners are deleted from the live object, or reset to their default value if they have one. Safe ownership transfers, such as the transfer of a `replicas` field from a user to an HPA without resetting to the default value are documented in [Transferring Ownership](https://kubernetes.io/docs/reference/using-api/api-concepts/&#35;transferring-ownership) ([#92661](https://github.com/kubernetes/kubernetes/pull/92661), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Testing]
+- Server-side apply behavior has been regularized in the case where a field is removed from the applied configuration. Removed fields which have no other owners are deleted from the live object, or reset to their default value if they have one. Safe ownership transfers, such as the transfer of a `replicas` field from a user to an HPA without resetting to the default value are documented in [Transferring Ownership](/docs/reference/using-api/server-side-apply/#transferring-ownership) ([#92661](https://github.com/kubernetes/kubernetes/pull/92661), [@jpbetz](https://github.com/jpbetz)) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Testing]
 - Set CSIMigrationvSphere feature gates to beta.
  Users should enable CSIMigration + CSIMigrationvSphere features and install the vSphere CSI Driver (https://github.com/kubernetes-sigs/vsphere-csi-driver) to move workload from the in-tree vSphere plugin "kubernetes.io/vsphere-volume" to vSphere CSI Driver.
  
@@ -192,7 +192,7 @@ func main() {
 }
 ```

-If the application is deployed as a Pod in the cluster, see [Accessing the API from within a Pod](#accessing-the-api-from-within-a-pod).
+If the application is deployed as a Pod in the cluster, see [Accessing the API from within a Pod](/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod).

 #### Python client

@@ -34,7 +34,7 @@ If your cluster was deployed using the `kubeadm` tool, refer to
 for detailed information on how to upgrade the cluster.

 Once you have upgraded the cluster, remember to
-[install the latest version of `kubectl`](/docs/tasks/tools/install-kubectl/).
+[install the latest version of `kubectl`](/docs/tasks/tools/).

 ### Manual deployments

@@ -52,7 +52,7 @@ You should manually update the control plane following this sequence:
 - cloud controller manager, if you use one

 At this point you should
-[install the latest version of `kubectl`](/docs/tasks/tools/install-kubectl/).
+[install the latest version of `kubectl`](/docs/tasks/tools/).

 For each node in your cluster, [drain](/docs/tasks/administer-cluster/safely-drain-node/)
 that node and then either replace it with a new node that uses the {{< skew latestVersion >}}
@@ -170,36 +170,7 @@ controllerManager:

 ### Create certificate signing requests (CSR)

-You can create the certificate signing requests for the Kubernetes certificates API with `kubeadm certs renew --use-api`.
-
-If you set up an external signer such as [cert-manager](https://github.com/jetstack/cert-manager), certificate signing requests (CSRs) are automatically approved.
-Otherwise, you must manually approve certificates with the [`kubectl certificate`](/docs/setup/best-practices/certificates/) command.
-The following kubeadm command outputs the name of the certificate to approve, then blocks and waits for approval to occur:
-
-```shell
-sudo kubeadm certs renew apiserver --use-api &
-```
-The output is similar to this:
-```
-[1] 2890
-[certs] certificate request "kubeadm-cert-kube-apiserver-ld526" created
-```
-
-### Approve certificate signing requests (CSR)
-
-If you set up an external signer, certificate signing requests (CSRs) are automatically approved.
-
-Otherwise, you must manually approve certificates with the [`kubectl certificate`](/docs/setup/best-practices/certificates/) command. e.g.
-
-```shell
-kubectl certificate approve kubeadm-cert-kube-apiserver-ld526
-```
-The output is similar to this:
-```shell
-certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kube-apiserver-ld526 approved
-```
-
-You can view a list of pending certificates with `kubectl get csr`.
+See [Create CertificateSigningRequest](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#create-certificatesigningrequest) for creating CSRs with the Kubernetes API.

 ## Renew certificates with external CA

@@ -202,4 +202,7 @@ verify that the pods were scheduled by the desired schedulers.
 ```shell
 kubectl get events
 ```
+You can also use a [custom scheduler configuration](/docs/reference/scheduling/config/#multiple-profiles)
+or a custom container image for the cluster's main scheduler by modifying its static pod manifest
+on the relevant control plane nodes.

@@ -1129,8 +1129,6 @@ resources that have the scale subresource enabled.

 ### Categories

-{{< feature-state state="beta" for_k8s_version="v1.10" >}}
-
 Categories is a list of grouped resources the custom resource belongs to (eg. `all`).
 You can use `kubectl get <category-name>` to list the resources belonging to the category.

@@ -2,7 +2,7 @@
 title: Coarse Parallel Processing Using a Work Queue
 min-kubernetes-server-version: v1.8
 content_type: task
-weight: 30
+weight: 20
 ---


@@ -2,7 +2,7 @@
 title: Fine Parallel Processing Using a Work Queue
 content_type: task
 min-kubernetes-server-version: v1.8
-weight: 40
+weight: 30
 ---

 <!-- overview -->
@@ -0,0 +1,176 @@
+---
+title: Indexed Job for Parallel Processing with Static Work Assignment
+content_type: task
+min-kubernetes-server-version: v1.21
+weight: 30
+---
+
+{{< feature-state for_k8s_version="v1.21" state="alpha" >}}
+
+<!-- overview -->
+
+
+In this example, you will run a Kubernetes Job that uses multiple parallel
+worker processes.
+Each worker is a different container running in its own Pod. The Pods have an
+_index number_ that the control plane sets automatically, which allows each Pod
+to identify which part of the overall task to work on.
+
+The pod index is available in the {{< glossary_tooltip text="annotation" term_id="annotation" >}}
+`batch.kubernetes.io/job-completion-index` as string representing its
+decimal value. In order for the containerized task process to obtain this index,
+you can publish the value of the annotation using the [downward API](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api)
+mechanism.
+For convenience, the control plane automatically sets the downward API to
+expose the index in the `JOB_COMPLETION_INDEX` environment variable.
+
+Here is an overview of the steps in this example:
+
+1. **Create an image that can read the pod index**. You might modify the worker
+   program or add a script wrapper.
+2. **Start an Indexed Job**. The downward API allows you to pass the annotation
+   as an environment variable or file to the container.
+
+## {{% heading "prerequisites" %}}
+
+Be familiar with the basic,
+non-parallel, use of [Job](/docs/concepts/workloads/controllers/job/).
+
+{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
+
+To be able to create Indexed Jobs, make sure to enable the `IndexedJob`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+on the [API server](docs/reference/command-line-tools-reference/kube-apiserver/)
+and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/).
+
+<!-- steps -->
+
+## Choose an approach
+
+To access the work item from the worker program, you have a few options:
+
+1. Read the `JOB_COMPLETION_INDEX` environment variable. The Job
+   {{< glossary_tooltip text="controller" term_id="controller" >}}
+   automatically links this variable to the annotation containing the completion
+   index.
+1. Read a file that contains the completion index.
+1. Assuming that you can't modify the program, you can wrap it with a script
+   that reads the index using any of the methods above and converts it into
+   something that the program can use as input.
+  
+For this example, imagine that you chose option 3 and you want to run the
+[rev](https://man7.org/linux/man-pages/man1/rev.1.html) utility. This
+program accepts a file as an argument and prints its content reversed.
+
+```shell
+rev data.txt
+```
+
+For this example, you'll use the `rev` tool from the
+[`busybox`](https://hub.docker.com/_/busybox) container image.
+
+## Define an Indexed Job
+
+Here is a job definition. You'll need to edit the container image to match your
+preferred registry.
+
+{{< codenew language="yaml" file="application/job/indexed-job.yaml" >}}
+
+In the example above, you use the builtin `JOB_COMPLETION_INDEX` environment
+variable set by the Job controller for all containers. An [init container](/docs/concepts/workloads/pods/init-containers/)
+maps the index to a static value and writes it to a file that is shared with the
+container running the worker through an [emptyDir volume](/docs/concepts/storage/volumes/#emptydir).
+Optionally, you can [define your own environment variable through the downward
+API](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/)
+to publish the index to containers. You can also choose to load a list of values
+from a [ConfigMap as an environment variable or file](/docs/tasks/configure-pod-container/configure-pod-configmap/).
+
+Alternatively, you can directly [use the downward API to pass the annotation
+value as a volume file](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#store-pod-fields),
+like shown in the following example:
+
+{{< codenew language="yaml" file="application/job/indexed-job-vol.yaml" >}}
+
+## Running the Job
+
+Now run the Job:
+
+```shell
+kubectl apply -f ./indexed-job.yaml
+```
+
+Wait a bit, then check on the job:
+
+```shell
+kubectl describe jobs/indexed-job
+```
+
+The output is similar to:
+
+```
+Name:              indexed-job
+Namespace:         default
+Selector:          controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
+Labels:            controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
+                   job-name=indexed-job
+Annotations:       <none>
+Parallelism:       3
+Completions:       5
+Start Time:        Thu, 11 Mar 2021 15:47:34 +0000
+Pods Statuses:     2 Running / 3 Succeeded / 0 Failed
+Completed Indexes: 0-2
+Pod Template:
+  Labels:  controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
+           job-name=indexed-job
+  Init Containers:
+   input:
+    Image:      docker.io/library/bash
+    Port:       <none>
+    Host Port:  <none>
+    Command:
+      bash
+      -c
+      items=(foo bar baz qux xyz)
+      echo ${items[$JOB_COMPLETION_INDEX]} > /input/data.txt
+
+    Environment:  <none>
+    Mounts:
+      /input from input (rw)
+  Containers:
+   worker:
+    Image:      docker.io/library/busybox
+    Port:       <none>
+    Host Port:  <none>
+    Command:
+      rev
+      /input/data.txt
+    Environment:  <none>
+    Mounts:
+      /input from input (rw)
+  Volumes:
+   input:
+    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
+    Medium:
+    SizeLimit:  <unset>
+Events:
+  Type    Reason            Age   From            Message
+  ----    ------            ----  ----            -------
+  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-njkjj
+  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-9kd4h
+  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-qjwsz
+  Normal  SuccessfulCreate  1s    job-controller  Created pod: indexed-job-fdhq5
+  Normal  SuccessfulCreate  1s    job-controller  Created pod: indexed-job-ncslj
+```
+
+In this example, we run the job with custom values for each index. You can
+inspect the output of the pods:
+
+```shell
+kubectl logs indexed-job-fdhq5 # Change this to match the name of a Pod in your cluster.
+```
+
+The output is similar to:
+
+```
+xuq
+```
@@ -2,7 +2,7 @@
 title: Parallel Processing using Expansions
 content_type: task
 min-kubernetes-server-version: v1.8
-weight: 20
+weight: 50
 ---

 <!-- overview -->
@@ -16,7 +16,7 @@ preview of what changes `apply` will make.
 ## {{% heading "prerequisites" %}}


-Install [`kubectl`](/docs/tasks/tools/install-kubectl/).
+Install [`kubectl`](/docs/tasks/tools/).

 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

@@ -12,7 +12,7 @@ explains how those commands are organized and how to use them to manage live obj

 ## {{% heading "prerequisites" %}}

-Install [`kubectl`](/docs/tasks/tools/install-kubectl/).
+Install [`kubectl`](/docs/tasks/tools/).

 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

@@ -13,7 +13,7 @@ This document explains how to define and manage objects using configuration file
 ## {{% heading "prerequisites" %}}


-Install [`kubectl`](/docs/tasks/tools/install-kubectl/).
+Install [`kubectl`](/docs/tasks/tools/).

 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

@@ -29,7 +29,7 @@ kubectl apply -k <kustomization_directory>
 ## {{% heading "prerequisites" %}}


-Install [`kubectl`](/docs/tasks/tools/install-kubectl/).
+Install [`kubectl`](/docs/tasks/tools/).

 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}

@@ -19,7 +19,7 @@ Up to date information on this process can be found at the
 * You must have a Kubernetes cluster with cluster DNS enabled.
    * If you are using a cloud-based Kubernetes cluster or {{< glossary_tooltip text="Minikube" term_id="minikube" >}}, you may already have cluster DNS enabled.
    * If you are using `hack/local-up-cluster.sh`, ensure that the `KUBE_ENABLE_CLUSTER_DNS` environment variable is set, then run the install script.
-* [Install and setup kubectl](/docs/tasks/tools/install-kubectl/) v1.7 or higher. Make sure it is configured to connect to the Kubernetes cluster.
+* [Install and setup kubectl](/docs/tasks/tools/) v1.7 or higher. Make sure it is configured to connect to the Kubernetes cluster.
 * Install [Helm](https://helm.sh/) v2.7.0 or newer.
    * Follow the [Helm install instructions](https://helm.sh/docs/intro/install/).
    * If you already have an appropriate version of Helm installed, execute `helm init` to install Tiller, the server-side component of Helm.
@@ -23,7 +23,7 @@ Service Catalog itself can work with any kind of managed service, not just Googl
 * Install [Go 1.6+](https://golang.org/dl/) and set the `GOPATH`.
 * Install the [cfssl](https://github.com/cloudflare/cfssl) tool needed for generating SSL artifacts.
 * Service Catalog requires Kubernetes version 1.7+.
-* [Install and setup kubectl](/docs/tasks/tools/install-kubectl/) so that it is configured to connect to a Kubernetes v1.7+ cluster.
+* [Install and setup kubectl](/docs/tasks/tools/) so that it is configured to connect to a Kubernetes v1.7+ cluster.
 * The kubectl user must be bound to the *cluster-admin* role for it to install Service Catalog. To ensure that this is true, run the following command:

        kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<user-name>
@@ -80,10 +80,10 @@ You now have to ensure that the kubectl completion script gets sourced in all yo
    echo 'complete -F __start_kubectl k' >>~/.bash_profile
    ```

- If you installed kubectl with Homebrew (as explained [above](#install-with-homebrew-on-macos)), then the kubectl completion script should already be in `/usr/local/etc/bash_completion.d/kubectl`. In that case, you don't need to do anything.
+- If you installed kubectl with Homebrew (as explained [here](/docs/tasks/tools/install-kubectl-macos/#install-with-homebrew-on-macos)), then the kubectl completion script should already be in `/usr/local/etc/bash_completion.d/kubectl`. In that case, you don't need to do anything.

   {{< note >}}
   The Homebrew installation of bash-completion v2 sources all the files in the `BASH_COMPLETION_COMPAT_DIR` directory, that's why the latter two methods work.
   {{< /note >}}

-In any case, after reloading your shell, kubectl completion should be working.
+In any case, after reloading your shell, kubectl completion should be working.
@@ -100,15 +100,38 @@ For example, to download version {{< param "fullversion" >}} on Linux, type:
 ### Install using native package management

 {{< tabs name="kubectl_install" >}}
-{{< tab name="Ubuntu, Debian or HypriotOS" codelang="bash" >}}
-sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2 curl
-curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
-echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
-sudo apt-get update
-sudo apt-get install -y kubectl
-{{< /tab >}}
+{{% tab name="Debian-based distributions" %}}

-{{< tab name="CentOS, RHEL or Fedora" codelang="bash" >}}cat <<EOF > /etc/yum.repos.d/kubernetes.repo
+1. Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:
+
+   ```shell
+   sudo apt-get update
+   sudo apt-get install -y apt-transport-https ca-certificates curl
+   ```
+
+2. Download the Google Cloud public signing key:
+
+   ```shell
+   sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
+   ```
+
+3. Add the Kubernetes `apt` repository:
+
+   ```shell
+   echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
+   ```
+
+4. Update `apt` package index with the new repository and install kubectl:
+
+   ```shell
+   sudo apt-get update
+   sudo apt-get install -y kubectl
+   ```
+
+{{% /tab %}}
+
+{{< tab name="Red Hat-based distributions" codelang="bash" >}}
+cat <<EOF > /etc/yum.repos.d/kubernetes.repo
 [kubernetes]
 name=Kubernetes
 baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
@@ -23,7 +23,7 @@ The following methods exist for installing kubectl on macOS:
 - [Install kubectl binary with curl on macOS](#install-kubectl-binary-with-curl-on-macos)
 - [Install with Homebrew on macOS](#install-with-homebrew-on-macos)
 - [Install with Macports on macOS](#install-with-macports-on-macos)
- [Install on Linux as part of the Google Cloud SDK](#install-on-linux-as-part-of-the-google-cloud-sdk)
+- [Install on macOS as part of the Google Cloud SDK](#install-on-macos-as-part-of-the-google-cloud-sdk)

 ### Install kubectl binary with curl on macOS

@@ -157,4 +157,4 @@ Below are the procedures to set up autocompletion for Bash and Zsh.

 ## {{% heading "whatsnext" %}}

-{{< include "included/kubectl-whats-next.md" >}}
+{{< include "included/kubectl-whats-next.md" >}}
@@ -37,7 +37,7 @@ profiles that give only the necessary privileges to your container processes.

 In order to complete all steps in this tutorial, you must install
 [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and
-[kubectl](/docs/tasks/tools/install-kubectl/). This tutorial will show examples
+[kubectl](/docs/tasks/tools/). This tutorial will show examples
 with both alpha (pre-v1.19) and generally available seccomp functionality, so
 make sure that your cluster is [configured
 correctly](https://kind.sigs.k8s.io/docs/user/quick-start/#setting-kubernetes-version)
@@ -15,10 +15,8 @@ This page provides a real world example of how to configure Redis using a Config
 ## {{% heading "objectives" %}}


-* Create a `kustomization.yaml` file containing:
-  * a ConfigMap generator
-  * a Pod resource config using the ConfigMap
-* Apply the directory by running `kubectl apply -k ./`
+* Create a ConfigMap with Redis configuration values
+* Create a Redis Pod that mounts and uses the created ConfigMap
 * Verify that the configuration was correctly applied.


@@ -38,82 +36,218 @@ This page provides a real world example of how to configure Redis using a Config

 ## Real World Example: Configuring Redis using a ConfigMap

-You can follow the steps below to configure a Redis cache using data stored in a ConfigMap.
+Follow the steps below to configure a Redis cache using data stored in a ConfigMap.

-First create a `kustomization.yaml` containing a ConfigMap from the `redis-config` file:
-
-{{< codenew file="pods/config/redis-config" >}}
+First create a ConfigMap with an empty configuration block:

 ```shell
-curl -OL https://k8s.io/examples/pods/config/redis-config
-
-cat <<EOF >./kustomization.yaml
-configMapGenerator:
- name: example-redis-config
-  files:
-  - redis-config
+cat <<EOF >./example-redis-config.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: example-redis-config
+data:
+  redis-config: ""
 EOF
 ```

-Add the pod resource config to the `kustomization.yaml`:
+Apply the ConfigMap created above, along with a Redis pod manifest:
+
+```shell
+kubectl apply -f example-redis-config.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/pods/config/redis-pod.yaml
+```
+
+Examine the contents of the Redis pod manifest and note the following:
+
+* A volume named `config` is created by `spec.volumes[1]`
+* The `key` and `path` under `spec.volumes[1].items[0]` exposes the `redis-config` key from the 
+  `example-redis-config` ConfigMap as a file named `redis.conf` on the `config` volume.
+* The `config` volume is then mounted at `/redis-master` by `spec.containers[0].volumeMounts[1]`.
+
+This has the net effect of exposing the data in `data.redis-config` from the `example-redis-config`
+ConfigMap above as `/redis-master/redis.conf` inside the Pod.

 {{< codenew file="pods/config/redis-pod.yaml" >}}

-```shell
-curl -OL https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/pods/config/redis-pod.yaml
+Examine the created objects:

-cat <<EOF >>./kustomization.yaml
-resources:
- redis-pod.yaml
-EOF
+```shell
+kubectl get pod/redis configmap/example-redis-config 
 ```

-Apply the kustomization directory to create both the ConfigMap and Pod objects:
+You should see the following output:

 ```shell
-kubectl apply -k .
-```
-
-Examine the created objects by
-```shell
-> kubectl get -k .
-NAME                                        DATA   AGE
-configmap/example-redis-config-dgh9dg555m   1      52s
-
 NAME        READY   STATUS    RESTARTS   AGE
-pod/redis   1/1     Running   0          52s
+pod/redis   1/1     Running   0          8s
+
+NAME                             DATA   AGE
+configmap/example-redis-config   1      14s
 ```

-In the example, the config volume is mounted at `/redis-master`.
-It uses `path` to add the `redis-config` key to a file named `redis.conf`.
-The file path for the redis config, therefore, is `/redis-master/redis.conf`.
-This is where the image will look for the config file for the redis master.
+Recall that we left `redis-config` key in the `example-redis-config` ConfigMap blank:

-Use `kubectl exec` to enter the pod and run the `redis-cli` tool to verify that
-the configuration was correctly applied:
+```shell
+kubectl describe configmap/example-redis-config
+```
+
+You should see an empty `redis-config` key:
+
+```shell
+Name:         example-redis-config
+Namespace:    default
+Labels:       <none>
+Annotations:  <none>
+
+Data
+====
+redis-config:
+```
+
+Use `kubectl exec` to enter the pod and run the `redis-cli` tool to check the current configuration:

 ```shell
 kubectl exec -it redis -- redis-cli
+```
+
+Check `maxmemory`:
+
+```shell
 127.0.0.1:6379> CONFIG GET maxmemory
+```
+
+It should show the default value of 0:
+
+```shell
+1) "maxmemory"
+2) "0"
+```
+
+Similarly, check `maxmemory-policy`:
+
+```shell
+127.0.0.1:6379> CONFIG GET maxmemory-policy
+```
+
+Which should also yield its default value of `noeviction`:
+
+```shell
+1) "maxmemory-policy"
+2) "noeviction"
+```
+
+Now let's add some configuration values to the `example-redis-config` ConfigMap:
+
+{{< codenew file="pods/config/example-redis-config.yaml" >}}
+
+Apply the updated ConfigMap:
+
+```shell
+kubectl apply -f example-redis-config.yaml
+```
+
+Confirm that the ConfigMap was updated:
+
+```shell
+kubectl describe configmap/example-redis-config
+```
+
+You should see the configuration values we just added:
+
+```shell
+Name:         example-redis-config
+Namespace:    default
+Labels:       <none>
+Annotations:  <none>
+
+Data
+====
+redis-config:
+----
+maxmemory 2mb
+maxmemory-policy allkeys-lru
+```
+
+Check the Redis Pod again using `redis-cli` via `kubectl exec` to see if the configuration was applied:
+
+```shell
+kubectl exec -it redis -- redis-cli
+```
+
+Check `maxmemory`:
+
+```shell
+127.0.0.1:6379> CONFIG GET maxmemory
+```
+
+It remains at the default value of 0:
+
+```shell
+1) "maxmemory"
+2) "0"
+```
+
+Similarly, `maxmemory-policy` remains at the `noeviction` default setting:
+
+```shell
+127.0.0.1:6379> CONFIG GET maxmemory-policy
+```
+
+Returns:
+
+```shell
+1) "maxmemory-policy"
+2) "noeviction"
+```
+
+The configuration values have not changed because the Pod needs to be restarted to grab updated
+values from associated ConfigMaps. Let's delete and recreate the Pod:
+
+```shell
+kubectl delete pod redis
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/pods/config/redis-pod.yaml
+```
+
+Now re-check the configuration values one last time:
+
+```shell
+kubectl exec -it redis -- redis-cli
+```
+
+Check `maxmemory`:
+
+```shell
+127.0.0.1:6379> CONFIG GET maxmemory
+```
+
+It should now return the updated value of 2097152:
+
+```shell
 1) "maxmemory"
 2) "2097152"
+```
+
+Similarly, `maxmemory-policy` has also been updated:
+
+```shell
 127.0.0.1:6379> CONFIG GET maxmemory-policy
+```
+
+It now reflects the desired value of `allkeys-lru`:
+
+```shell
 1) "maxmemory-policy"
 2) "allkeys-lru"
 ```

-Delete the created pod:
+Clean up your work by deleting the created resources:
+
 ```shell
-kubectl delete pod redis
+kubectl delete pod/redis configmap/example-redis-config
 ```

-
-
 ## {{% heading "whatsnext" %}}


 * Learn more about [ConfigMaps](/docs/tasks/configure-pod-container/configure-pod-configmap/).
-
-
-
-
@@ -37,7 +37,7 @@ weight: 10
 				<li><i>ClusterIP</i> (default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.</li>
 				<li><i>NodePort</i> - Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using <code>&lt;NodeIP&gt;:&lt;NodePort&gt;</code>. Superset of ClusterIP.</li>
 				<li><i>LoadBalancer</i> - Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.</li>
-				<li><i>ExternalName</i> - Exposes the Service using an arbitrary name (specified by <code>externalName</code> in the spec) by returning a CNAME record with the name. No proxy is used. This type requires v1.7 or higher of <code>kube-dns</code>.</li>
+				<li><i>ExternalName</i> - Maps the Service to the contents of the <code>externalName</code> field (e.g. `foo.bar.example.com`), by returning a <code>CNAME</code> record with its value. No proxying of any kind is set up. This type requires v1.7 or higher of <code>kube-dns</code>, or CoreDNS version 0.0.8 or higher.</li>
 			</ul>
 			<p>More information about the different types of Services can be found in the <a href="/docs/tutorials/services/source-ip/">Using Source IP</a> tutorial. Also see <a href="/docs/concepts/services-networking/connect-applications-service">Connecting Applications with Services</a>.</p>
 			<p>Additionally, note that there are some use cases with Services that involve not defining <code>selector</code> in the spec. A Service created without <code>selector</code> will also not create the corresponding Endpoints object. This allows users to manually map a Service to specific endpoints. Another possibility why there may be no selector is you are strictly using <code>type: ExternalName</code>.</p>
@@ -11,7 +11,7 @@ external IP address.

 ## {{% heading "prerequisites" %}}

-* Install [kubectl](/docs/tasks/tools/install-kubectl/).
+* Install [kubectl](/docs/tasks/tools/).
 * Use a cloud provider like Google Kubernetes Engine or Amazon Web Services to
  create a Kubernetes cluster. This tutorial creates an
  [external load balancer](/docs/tasks/access-application-cluster/create-external-load-balancer/),
@@ -104,7 +104,7 @@ kubectl apply -f ./content/en/examples/application/guestbook/mongo-service.yaml
      ```shell
      NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
      kubernetes     ClusterIP   10.0.0.1     <none>        443/TCP    1m
-      mongo          ClusterIP   10.0.0.151   <none>        6379/TCP   8s
+      mongo          ClusterIP   10.0.0.151   <none>        27017/TCP   8s
      ```

 {{< note >}}