Revert v1.17 release changes on v1.16 branch (#18123)

This reverts commits: 1131f8603e. cf5b8b200c. 75de70a431. dff2f7b960. cf925bb688. ffd1363c0e. 87a2cafb20. ffe259d6f2.
2019-12-13 19:51:36 +00:00
parent 5733771d62
commit 065ccd94be
74 changed files with 647 additions and 63699 deletions
@@ -68,11 +68,11 @@ showedit = true
 latest = "v1.17"
-fullversion = "v1.17.0"
+fullversion = "v1.16.3"
-version = "v1.17"
+version = "v1.16"
-githubbranch = "master"
+githubbranch = "v1.16.3"
-docsbranch = "master"
+docsbranch = "release-1.16"
-deprecated = false
+deprecated = true
 currentUrl = "https://kubernetes.io/docs/home/"
 nextUrl = "https://kubernetes-io-vnext-staging.netlify.com/"
 githubWebsiteRepo = "github.com/kubernetes/website"
@@ -1,162 +0,0 @@
 ---
 layout: blog
 title: "Kubernetes 1.17 Feature: Kubernetes In-Tree to CSI Volume Migration Moves to Beta"
 date: 2019-12-09T09:00:00+08:00
 slug: kubernetes-1-17-feature-csi-migration-beta
 ---
 **Authors:** David Zhu, Software Engineer, Google
 The Kubernetes in-tree storage plugin to [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) migration infrastructure is now beta in Kubernetes v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.
 Kubernetes features are generally introduced as alpha and moved to beta (and eventually to stable/GA) over subsequent Kubernetes releases. This process allows Kubernetes developers to get feedback, discover and fix issues, iterate on the designs, and deliver high quality, production grade features.
 ## Why are we migrating in-tree plugins to CSI?
 Prior to CSI, Kubernetes provided a powerful volume plugin system. These volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries. However, adding support for new volume plugins to Kubernetes was challenging. Vendors that wanted to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain. Using the Container Storage Interface in Kubernetes resolves these major issues.
 As more CSI Drivers were created and became production ready, we wanted all Kubernetes users to reap the benefits of the CSI model. However, we did not want to force users into making workload/configuration changes by breaking the existing generally available storage APIs. The way forward was clear - we would have to replace the backend of the “in-tree plugin” APIs with CSI.
 ## What is CSI migration?
 The CSI migration effort enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding [CSI driver](https://kubernetes-csi.github.io/docs/introduction.html). If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.
 When a Kubernetes cluster administrator updates a cluster to enable CSI migration, existing stateful deployments and workloads continue to function as they always have; however, behind the scenes Kubernetes hands control of all storage management operations (previously targeting in-tree drivers) to CSI drivers.
 The Kubernetes team has worked hard to ensure the stability of storage APIs and for the promise of a smooth upgrade experience. This involves meticulous accounting of all existing features and behaviors to ensure backwards compatibility and API stability. You can think of it like changing the wheels on a racecar while it’s speeding down the straightaway.
 ## How to try out CSI migration for existing plugins?
 If you are Kubernetes distributor that deploys in one of the environments listed below, now would be a good time to start testing the CSI migration and figuring out how to deploy/manage the appropriate CSI driver.
 To try out CSI migration in beta for an existing plugin you must be using Kubernetes v1.17 or higher. First, you must update/create a Kubernetes cluster with the feature flags `CSIMigration` (on by default in 1.17) and `CSIMigration{provider}` (off by default) enabled on the `kube-apiserver`, `kube-controller-manager`, as well as the `kubelet`’s. Where {provider} is the in-tree cloud provider storage type that is used in your cluster.
 You must also install the requisite CSI driver on your cluster - instructions for this can generally be found from you provider of choice. CSI migration is available for GCE Persistent Disk and AWS Elastic Block Store in beta as well as for Azure Ffile/Ddisk and Openstack/Cinder in alpha. Kubernetes distributors should look at automating the deployment and management (upgrade, downgrade, etc.) of the CSI Drivers they will depend on.
 To verify the feature flag is enabled and driver installed on a particular node you can get the CSINode object. You should see the in-tree plugin name of the migrated plugin as well as your driver in the drivers list.
 ```shell
 kubectl get csinodes -o yaml
 ```
 ```yaml
 - apiVersion: storage.k8s.io/v1
  kind: CSINode
  metadata:
    annotations:
      storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/gce-pd
    name: test-node
    ...
  spec:
    drivers:
      name: pd.csi.storage.gke.io
      ...
 ```
 After the above set up is complete you can confirm that your cluster has functioning CSI migration by deploying a stateful workload using the legacy APIs.
 ```yaml
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: test-disk
 spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
 ---
 apiVersion: v1
 kind: Pod
 metadata:
  name: web-server
 spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - mountPath: /var/lib/www/html
         name: mypvc
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: test-disk
 ```
 Verify that the pod is RUNNING after some time
 ```shell
 kubectl get pods web-server
 ```
 ```shell
 NAME         READY   STATUS    RESTARTS   AGE
 web-server   1/1     Running   0          39s
 ```
 To confirm that the CSI driver is actually serving your requests it may be prudent to check the container logs of the CSI Driver after exercising the storage management operations. Note that your container logs may look different depending on the provider used.
 ```shell
 kubectl logs {CSIdriverPodName} --container={CSIdriverContainerName}
 ```
 ```shell
 /csi.v1.Controller/ControllerPublishVolume called with request: ...
 Attaching disk ... to ...
 ControllerPublishVolume succeeded for disk ... to instance ...
 ```
 ## Current limitations
 Although CSI migration is now beta there is one major limitation that prevents us from turning it on by default. Turning on migration still requires a cluster administrator to install a CSI driver before storage functionality is seamlessly handed over. We are currently working with SIGsig-cloudprovider to provide a frictionless experience of bundling the required CSI Drivers with cloud distributions.
 ## What is the timeline/status?
 The timeline for CSI migration is actually set by the cloud provider extraction project. It is part of the effort to remove all cloud provider code from Kubernetes. By migrating cloud storage plugins to external CSI drivers we are able to extract out all the cloud provider dependencies.
 Although the overall feature is beta and not on by default, there is still work to be done on a per-plugin basis. Currently only GCE PD and AWS EBS have gone beta with Migration and yet both are still off by default since they depend on a manual installation of their respective CSI Drivers. Azure File/Disk, OpenStack, and VMWare plugins are currently in less mature states and non-cloud plugins such as NFS, Portworx, RBD etc are still in the planning stages.
 The current and targeted releases for each individual cloud driver is shown in the table below:
 | Driver            | Alpha         | Beta (in-tree deprecated) | GA            | Target "in-tree plugin" removal |
 | ----------------- | ------------- | ------------------------- | ------------- | ------------------------------- |
 | AWS EBS           | 1.14          | 1.17                      | 1.19 (Target) | 1.21                            |
 | GCE PD            | 1.14          | 1.17                      | 1.19 (Target) | 1.21                            |
 | OpenStack Cinder  | 1.14          | 1.18 (Target)             | 1.19 (Target) | 1.21                            |
 | Azure Disk + File | 1.15          | 1.18 (Target)             | 1.19 (Target) | 1.21                            |
 | VSphere           | 1.18 (Target) | 1.19 (Target)             | 1.20 (Target) | 1.22                            |
 ## What's next?
 Major upcoming work includes implementing and hardening CSI migration for the remaining in-tree plugins, installing CSI Drivers by default in distributions, turning on CSI migration by default, and finally removing all in-tree plugin code as a part of cloud provider extraction. We expect to complete this project including the full switch to “on-by-default” migration by Kubernetes v1.21.
 ## What should I do as a user?
 Note that all new features for the Kubernetes storage system (like volume snapshotting) will only be added to the CSI interface. Therefore, if you are starting up a new cluster, creating stateful applications for the first time, or require these new features we recommend using CSI drivers natively (instead of the in-tree volume plugin API). Follow the [updated user guides for CSI drivers](https://kubernetes-csi.github.io/docs/drivers.html) and use the new CSI APIs.
 However, if you choose to roll a cluster forward or continue using specifications with the legacy volume APIs, CSI Migration will ensure we continue to support those deployments with the new CSI drivers.
 ## How do I get involved?
 The Kubernetes Slack channel csi-migration along with any of the standard [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams.
 This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. We offer a huge thank you to the contributors who stepped up these last quarters to help the project reach Beta:
 - David Zhu
 - Deep Debroy
 - Cheng Pan
 - Jan Šafránek
 With special thanks to:
 - Michelle Au
 - Saad Ali
 - Jonathan Basseri
 - Fabio Bertinatto
 - Ben Elder
 - Andrew Sy Kim
 - Hemant Kumar
 For fruitful dialogues, insightful reviews, and thorough consideration of CSI migration in other features.
@@ -1,129 +0,0 @@
 ---
 layout: blog
 title: "Kubernetes 1.17: Stability"
 date: 2019-12-09T13:00:00-08:00
 slug: kubernetes-1-17-release-announcement
 ---
 **Authors:** [Kubernetes 1.17 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.17/release_team.md)
 We’re pleased to announce the delivery of Kubernetes 1.17, our fourth and final release of 2019! Kubernetes v1.17 consists of 22 enhancements: 14 enhancements have graduated to stable, 4 enhancements are moving to beta, and 4 enhancements are entering alpha.
 ## Major Themes
 ### Cloud Provider Labels reach General Availability
 Added as a beta feature way back in v1.2, v1.17 sees the general availability of cloud provider labels.
 ### Volume Snapshot Moves to Beta
 The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced as alpha in Kubernetes v1.12, with a second alpha with breaking changes in Kubernetes v1.13.
 ### CSI Migration Beta
 The Kubernetes in-tree storage plugin to Container Storage Interface (CSI) migration infrastructure is now beta in Kubernetes v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.
 ## Cloud Provider Labels reach General Availability
 When nodes and volumes are created, a set of standard labels are applied based on the underlying cloud provider of the Kubernetes cluster. Nodes get a label for the instance type. Both nodes and volumes get two labels describing the location of the resource in the cloud provider topology, usually organized in zones and regions.
 Standard labels are used by Kubernetes components to support some features. For example, the scheduler would ensure that pods are placed on the same zone as the volumes they claim; and when scheduling pods belonging to a deployment, the scheduler would prioritize spreading them across zones. You can also use the labels in your pod specs to configure things as such node affinity. Standard labels allow you to write pod specs that are portable among different cloud providers.
 The labels are reaching general availability in this release. Kubernetes components have been updated to populate the GA and beta labels and to react to both. However, if you are using the beta labels in your pod specs for features such as node affinity, or in your custom controllers, we recommend that you start migrating them to the new GA labels. You can find the documentation for the new labels here:
 - [node.kubernetes.io/instance-type](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#nodekubernetesioinstance-type)
 - [topology.kubernetes.io/region](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesioregion)
 - [topology.kubernetes.io/zone](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone)
 ## Volume Snapshot Moves to Beta
 The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced as alpha in Kubernetes v1.12, with a second alpha with breaking changes in Kubernetes v1.13.  This post summarizes the changes in the beta release.
 ### What is a Volume Snapshot?
 Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).
 ### Why add Volume Snapshots to Kubernetes?
 The Kubernetes volume plugin system already provides a powerful abstraction that automates the provisioning, attaching, and mounting of block and file storage.
 Underpinning all these features is the Kubernetes goal of workload portability: Kubernetes aims to create an abstraction layer between distributed systems applications and underlying clusters so that applications can be agnostic to the specifics of the cluster they run on and application deployment requires no “cluster specific” knowledge.
 The Kubernetes Storage SIG identified snapshot operations as critical functionality for many stateful workloads. For example, a database administrator may want to snapshot a database volume before starting a database operation.
 By providing a standard way to trigger snapshot operations in the Kubernetes API, Kubernetes users can now handle use cases like this without having to go around the Kubernetes API (and manually executing storage system specific operations).
 Instead, Kubernetes users are now empowered to incorporate snapshot operations in a cluster agnostic way into their tooling and policy with the comfort of knowing that it will work against arbitrary Kubernetes clusters regardless of the underlying storage.
 Additionally these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise grade, storage administration features for Kubernetes: including application or cluster level backup solutions.
 You can read more in the blog entry about [releasing CSI volume snapshots to beta](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta/).
 ## CSI Migration Beta
 ### Why are we migrating in-tree plugins to CSI?
 Prior to CSI, Kubernetes provided a powerful volume plugin system. These volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries. However, adding support for new volume plugins to Kubernetes was challenging. Vendors that wanted to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain. Using the Container Storage Interface in Kubernetes resolves these major issues.
 As more CSI Drivers were created and became production ready, we wanted all Kubernetes users to reap the benefits of the CSI model. However, we did not want to force users into making workload/configuration changes by breaking the existing generally available storage APIs. The way forward was clear - we would have to replace the backend of the “in-tree plugin” APIs with CSI.
 What is CSI migration?
 The CSI migration effort enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding CSI driver. If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.
 When a Kubernetes cluster administrator updates a cluster to enable CSI migration, existing stateful deployments and workloads continue to function as they always have; however, behind the scenes Kubernetes hands control of all storage management operations (previously targeting in-tree drivers) to CSI drivers.
 The Kubernetes team has worked hard to ensure the stability of storage APIs and for the promise of a smooth upgrade experience. This involves meticulous accounting of all existing features and behaviors to ensure backwards compatibility and API stability. You can think of it like changing the wheels on a racecar while it’s speeding down the straightaway.
 You can read more in the blog entry about [CSI migration going to beta](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/).
 ## Other Updates
 ### Graduated to Stable 💯
 - [Taint Node by Condition](https://github.com/kubernetes/enhancements/issues/382)
 - [Configurable Pod Process Namespace Sharing](https://github.com/kubernetes/enhancements/issues/495)
 - [Schedule DaemonSet Pods by kube-scheduler](https://github.com/kubernetes/enhancements/issues/548)
 - [Dynamic Maximum Volume Count](https://github.com/kubernetes/enhancements/issues/554)
 - [Kubernetes CSI Topology Support](https://github.com/kubernetes/enhancements/issues/557)
 - [Provide Environment Variables Expansion in SubPath Mount](https://github.com/kubernetes/enhancements/issues/559)
 - [Defaulting of Custom Resources](https://github.com/kubernetes/enhancements/issues/575)
 - [Move Frequent Kubelet Heartbeats To Lease Api](https://github.com/kubernetes/enhancements/issues/589)
 - [Break Apart The Kubernetes Test Tarball](https://github.com/kubernetes/enhancements/issues/714)
 - [Add Watch Bookmarks Support](https://github.com/kubernetes/enhancements/issues/956)
 - [Behavior-Driven Conformance Testing](https://github.com/kubernetes/enhancements/issues/960)
 - [Finalizer Protection For Service Loadbalancers](https://github.com/kubernetes/enhancements/issues/980)
 - [Avoid Serializing The Same Object Independently For Every Watcher](https://github.com/kubernetes/enhancements/issues/1152)
 ### Major Changes
 - [Add IPv4/IPv6 Dual Stack Support](https://github.com/kubernetes/enhancements/issues/563)
 ### Other Notable Features
 - [Topology Aware Routing of Services (Alpha)](https://github.com/kubernetes/enhancements/issues/536)
 - [RunAsUserName for Windows](https://github.com/kubernetes/enhancements/issues/1043)
 ### Availability
 Kubernetes 1.17 is available for [download on GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.17.0). To get started with Kubernetes, check out these [interactive tutorials](https://kubernetes.io/docs/tutorials/). You can also easily install 1.17 using [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/).
 ### Release Team
 This release is made possible through the efforts of hundreds of individuals who contributed both technical and non-technical content. Special thanks to the [release team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.17/release_team.md) led by Guinevere Saenger. The 35 individuals on the release team coordinated many aspects of the release, from documentation to testing, validation, and feature completeness.
 As the Kubernetes community has grown, our release process represents an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid pace. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. Kubernetes has had over [39,000 individual contributors](https://k8s.devstats.cncf.io/d/24/overall-project-statistics?orgId=1) to date and an active community of more than 66,000 people.
 ### Webinar
 Join members of the Kubernetes 1.17 release team on Jan 7th, 2020 to learn about the major features in this release. Register [here](https://zoom.us/webinar/register/9315759188139/WN_kPOZA_6RTjeGdXTG7YFO3A).
 ### Get Involved
 The simplest way to get involved with Kubernetes is by joining one of the many [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly [community meeting](https://github.com/kubernetes/community/tree/master/communication), and through the channels below. Thank you for your continued feedback and support.
 - Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for latest updates
 - Join the community discussion on [Discuss](https://discuss.kubernetes.io/)
 - Join the community on [Slack](http://slack.k8s.io/)
 - Post questions (or answer questions) on [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes)
 - Share your Kubernetes [story](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform)
@@ -1,452 +0,0 @@
 ---
 layout: blog
 title: "Kubernetes 1.17 Feature: Kubernetes Volume Snapshot Moves to Beta"
 date: 2019-12-09T10:00:00-08:00
 slug: kubernetes-1-17-feature-cis-volume-snapshot-beta
 ---
 **Authors:** Xing Yang, VMware & Xiangqian Yu, Google
 The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced [as alpha](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) in Kubernetes v1.12, with a [second alpha](https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/) with breaking changes in Kubernetes v1.13.  This post summarizes the changes in the beta release.
 ## What is a Volume Snapshot?
 Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).
 ## Why add Volume Snapshots to Kubernetes?
 The Kubernetes volume plugin system already provides a powerful abstraction that automates the provisioning, attaching, and mounting of block and file storage.
 Underpinning all these features is the Kubernetes goal of workload portability: Kubernetes aims to create an abstraction layer between distributed systems applications and underlying clusters so that applications can be agnostic to the specifics of the cluster they run on and application deployment requires no “cluster specific” knowledge.
 The Kubernetes Storage SIG identified snapshot operations as critical functionality for many stateful workloads. For example, a database administrator may want to snapshot a database volume before starting a database operation.
 By providing a standard way to trigger snapshot operations in the Kubernetes API, Kubernetes users can now handle use cases like this without having to go around the Kubernetes API (and manually executing storage system specific operations).
 Instead, Kubernetes users are now empowered to incorporate snapshot operations in a cluster agnostic way into their tooling and policy with the comfort of knowing that it will work against arbitrary Kubernetes clusters regardless of the underlying storage.
 Additionally these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise grade, storage administration features for Kubernetes: including application or cluster level backup solutions.
 ## What’s new in Beta?
 With the promotion of Volume Snapshot to beta, the feature is now enabled by default on standard Kubernetes deployments instead of being opt-in.
 The move of the Kubernetes Volume Snapshot feature to beta also means:
 - A revamp of volume snapshot APIs.
 - The CSI external-snapshotter sidecar is split into two controllers, a common snapshot controller and a CSI external-snapshotter sidecar.
 - Deletion secret is added as an annotation to the volume snapshot content.
 - A new finalizer is added to the volume snapshot API object to prevent it from being deleted when it is bound to a volume snapshot content API object.
 ## Kubernetes Volume Snapshots Requirements
 As mentioned above, with the promotion of Volume Snapshot to beta, the feature is now enabled by default on standard Kubernetes deployments instead of being opt-in.
 In order to use the Kubernetes Volume Snapshot feature, you must ensure the following components have been deployed on your Kubernetes cluster:
 - [Kubernetes Volume Snapshot CRDs](https://github.com/kubernetes-csi/external-snapshotter/tree/master/config/crd)
 - [Volume snapshot controller](https://github.com/kubernetes-csi/external-snapshotter/tree/master/pkg/common-controller)
 - CSI Driver supporting Kubernetes volume snapshot beta
 See the deployment section below for details.
 ## Which drivers support Kubernetes Volume Snapshots?
 Kubernetes supports three types of volume plugins: in-tree, Flex, and CSI. See [Kubernetes Volume Plugin FAQ](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md) for details.
 Snapshots are only supported for CSI drivers (not for in-tree or Flex). To use the Kubernetes snapshots feature, ensure that a CSI Driver that implements snapshots is deployed on your cluster.
 Read the “[Container Storage Interface (CSI) for Kubernetes GA](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/)” blog post to learn more about CSI and how to deploy CSI drivers.
 As of the publishing of this blog, the following CSI drivers have been updated to support volume snapshots beta:
 - [GCE Persistent Disk CSI Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver)
 - [Portworx CSI Driver](https://github.com/libopenstorage/openstorage/tree/master/csi)
 - [NetApp Trident CSI Driver](https://github.com/NetApp/trident)
 Beta level Volume Snapshot support for other [CSI drivers](https://kubernetes-csi.github.io/docs/drivers.html) is pending, and should be available soon.
 ## Kubernetes Volume Snapshot Beta API
 A number of changes were made to the Kubernetes volume snapshot API between alpha to beta. These changes are not backward compatible. The purpose of these changes was to make API definitions clear and easier to use.
 The following changes are made:
 - `DeletionPolicy` is now a required field rather than optional in both `VolumeSnapshotClass` and `VolumeSnapshotContent`. This way the user has to explicitly specify it, leaving no room for confusion.
 - `VolumeSnapshotSpec` has a new required `Source` field. `Source` may be either a `PersistentVolumeClaimName` (if dynamically provisioning a snapshot) or `VolumeSnapshotContentName` (if pre-provisioning a snapshot).
 - `VolumeSnapshotContentSpec` also has a new required `Source` field. This `Source` may be either a `VolumeHandle` (if dynamically provisioning a snapshot) or a `SnapshotHandle` (if pre-provisioning volume snapshots).
 - `VolumeSnapshotStatus` now contains a `BoundVolumeSnapshotContentName` to indicate the `VolumeSnapshot` object is bound to a `VolumeSnapshotContent`.
 - `VolumeSnapshotContent`now contains a `Status` to indicate the current state of the content. It has a field `SnapshotHandle` to indicate that the `VolumeSnapshotContent` represents a snapshot on the storage system.
 The beta Kubernetes VolumeSnapshot API object:
 ```go
 type VolumeSnapshot struct {
        metav1.TypeMeta
        metav1.ObjectMeta
        Spec VolumeSnapshotSpec
        Status *VolumeSnapshotStatus
 }
 ```
 ```go
 type VolumeSnapshotSpec struct {
 	Source VolumeSnapshotSource
 	VolumeSnapshotClassName *string
 }
 // Exactly one of its members MUST be specified
 type VolumeSnapshotSource struct {
 	// +optional
 	PersistentVolumeClaimName *string
 	// +optional
 	VolumeSnapshotContentName *string
 }
 ```
 ```go
 type VolumeSnapshotStatus struct {
 	BoundVolumeSnapshotContentName *string
 	CreationTime *metav1.Time
 	ReadyToUse *bool
 	RestoreSize *resource.Quantity
 	Error *VolumeSnapshotError
 }
 ```
 The beta Kubernetes VolumeSnapshotContent API object:
 ```go
 type VolumeSnapshotContent struct {
        metav1.TypeMeta
        metav1.ObjectMeta
        Spec VolumeSnapshotContentSpec
        Status *VolumeSnapshotContentStatus
 }
 ```
 ```go
 type VolumeSnapshotContentSpec struct {
         VolumeSnapshotRef core_v1.ObjectReference
         Source VolumeSnapshotContentSource
         DeletionPolicy DeletionPolicy
         Driver string
         VolumeSnapshotClassName *string
 }
 ```
 ```go
 type VolumeSnapshotContentSource struct {
 	// +optional
 	VolumeHandle *string
 	// +optional
 	SnapshotHandle *string
 }
 ```
 ```go
 type VolumeSnapshotContentStatus struct {
 	CreationTime *int64
 	ReadyToUse *bool
 	RestoreSize *int64
 	Error *VolumeSnapshotError
           SnapshotHandle *string
 }
 ```
 The beta Kubernetes VolumeSnapshotClass API object:
 ```go
 type VolumeSnapshotClass struct {
        metav1.TypeMeta
        metav1.ObjectMeta
        Driver string
        Parameters map[string]string
        DeletionPolicy DeletionPolicy
 }
 ```
 ### How do I deploy support for Volume Snapshots on my Kubernetes Cluster?
 Please note that the Volume Snapshot feature now depends on a new, common [volume snapshot controller](https://github.com/kubernetes-csi/external-snapshotter/tree/master/pkg/common-controller) in addition to the volume snapshot CRDs. Both the volume snapshot controller and the CRDs are independent of any CSI driver. Regardless of the number CSI drivers deployed on the cluster, there must be only one instance of the volume snapshot controller running and one set of volume snapshot CRDs installed per cluster.
 Therefore, it is strongly recommended that Kubernetes distributors bundle and deploy the controller and CRDs as part of their Kubernetes cluster management process (independent of any CSI Driver).
 If your cluster does not come pre-installed with the correct components, you may manually install these components by executing the following steps.
 #### Install Snapshot Beta CRDs
 - `kubectl create -f config/crd`
 - [https://github.com/kubernetes-csi/external-snapshotter/tree/master/config/crd](https://github.com/kubernetes-csi/external-snapshotter/tree/master/config/crd)
 - Do this once per cluster
 #### Install Common Snapshot Controller
 - `kubectl create -f deploy/kubernetes/snapshot-controller`
 - [https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller](https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller)
 - Do this once per cluster
 #### Install CSI Driver
 Follow instructions provided by your CSI Driver vendor.
 ### How do I use Kubernetes Volume Snapshots?
 Assuming all the required components (including CSI driver) are already deployed and running on your cluster, you can create volume snapshots using the VolumeSnapshot API object, and restore them by specifying a VolumeSnapshot data source on a PVC.
 #### Creating a New Volume Snapshot with Kubernetes
 You can enable creation/deletion of volume snapshots in a Kubernetes cluster, by creating a VolumeSnapshotClass API object pointing to a CSI Driver that support volume snapshots.
 The following VolumeSnapshotClass, for example, tells the Kubernetes cluster that a CSI driver, `testdriver.csi.k8s.io`, can handle volume snapshots, and that when these snapshots are created, their deletion policy should be to delete.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshotClass
 metadata:
  name: test-snapclass
 driver: testdriver.csi.k8s.io
 deletionPolicy: Delete
 parameters:
  csi.storage.k8s.io/snapshotter-secret-name: mysecret
  csi.storage.k8s.io/snapshotter-secret-namespace: mysecretnamespace
 ```
 The common snapshot controller reserves the parameter keys `csi.storage.k8s.io/snapshotter-secret-name` and `csi.storage.k8s.io/snapshotter-secret-namespace`. If specified, it fetches the referenced Kubernetes secret and sets it as an annotation on the volume snapshot content object.  The CSI external-snapshotter sidecar retrieves it from the content annotation and passes it to the CSI driver during snapshot creation.
 #### Creation of a volume snapshot is triggered by the creation of a VolumeSnapshot API object.
 The VolumeSnapshot object must specify the following source type:
 `persistentVolumeClaimName` - The name of the PVC to snapshot. Please note that the source PVC, PV, and VolumeSnapshotClass for a VolumeSnapshot object must point to the same CSI driver.
 The following VolumeSnapshot, for example, triggers the creation of a snapshot for a PVC called `test-pvc` using the VolumeSnapshotClass above.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshot
 metadata:
  name: test-snapshot
 spec:
  volumeSnapshotClassName: test-snapclass
  source:
    persistentVolumeClaimName: test-pvc
 ```
 When volume snapshot creation is invoked, the common snapshot controller first creates a VolumeSnapshotContent object with the `volumeSnapshotRef`, source `volumeHandle`, `volumeSnapshotClassName` if specified, `driver`, and `deletionPolicy`.
 The CSI external-snapshotter sidecar then passes the VolumeSnapshotClass parameters, the source volume ID, and any referenced secret(s) to the CSI driver (in this case `testdriver.csi.k8s.io`) via a CSI `CreateSnapshot` call. In response, the CSI driver creates a new snapshot for the specified volume, and returns the ID for that snapshot. The CSI external-snapshotter sidecar then updates the `snapshotHandle`, `creationTime`, `restoreSize`, and `readyToUse` in the status field of the VolumeSnapshotContent object that represents the new snapshot. For a storage system that needs to upload the snapshot after it is being cut, the CSI external-snapshotter sidecar will keep calling the CSI `CreateSnapshot` to check the status until upload is complete and `readyToUse` is true.
 The common snapshot controller binds the VolumeSnapshotContent object to the VolumeSnapshot (sets `BoundVolumeSnapshotContentName`), updating the `creationTime`, `restoreSize`, and `readyToUse` in the status field of the VolumeSnapshot object based on the status field of the VolumeSnapshotContent object.
 If no `volumeSnapshotClassName` is specified, one is automatically selected as follows:
 The `StorageClass` from PVC or PV of the source volume is fetched.
 The default VolumeSnapshotClass is fetched, if available. A default VolumeSnapshotClass is a snapshot class created by the admin with the `snapshot.storage.kubernetes.io/is-default-class` annotation. If the `Driver` field of the default VolumeSnapshotClass is the same as the `Provisioner` field in the StorageClass, the default VolumeSnapshotClass is used. If there is no default VolumeSnapshotClass or more than one default VolumeSnapshotClass for a snapshot, an error will be returned.
 Please note that the Kubernetes Snapshot API does not provide any consistency guarantees. You have to prepare your application (pause application, freeze filesystem etc.) before taking the snapshot for data consistency either manually or using some other higher level APIs/controllers.
 You can verify that the VolumeSnapshot object is created and bound with VolumeSnapshotContent by running `kubectl describe volumesnapshot`:
 `Bound Volume Snapshot Content Name` - field in the `Status` indicates indicates the volume is bound to the specified VolumeSnapshotContent.
 `Ready To Use` - field in the `Status` indicates this volume snapshot is ready for use.
 `Creation Time` - field in the `Status` indicates when the snapshot was actually created (cut).
 `Restore Size` - field in the `Status` indicates the minimum volume size required when restoring a volume from this snapshot.
 ```
 Name:         test-snapshot
 Namespace:    default
 Labels:       <none>
 Annotations:  <none>
 API Version:  snapshot.storage.k8s.io/v1beta1
 Kind:         VolumeSnapshot
 Metadata:
  Creation Timestamp:  2019-11-16T00:36:04Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:        1
  Resource Version:  1294
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/new-snapshot-demo
  UID:               32ceaa2a-3802-4edd-a808-58c4f1bd7869
 Spec:
  Source:
    Persistent Volume Claim Name:  test-pvc
  Volume Snapshot Class Name:      test-snapclass
 Status:
  Bound Volume Snapshot Content Name:  snapcontent-32ceaa2a-3802-4edd-a808-58c4f1bd7869
  Creation Time:                       2019-11-16T00:36:04Z
  Ready To Use:                        true
  Restore Size:                        1Gi
 ```
 As a reminder to any developers building controllers using volume snapshot APIs: before using a VolumeSnapshot API object, validate the bi-directional binding between the VolumeSnpashot and the VolumeSnapshotContent it is bound to, to ensure the binding is complete and correct (not doing so may result in security issues).
 ```shell
 kubectl describe volumesnapshotcontent
 ```
 ```
 Name:         snapcontent-32ceaa2a-3802-4edd-a808-58c4f1bd7869
 Namespace:
 Labels:       <none>
 Annotations:  <none>
 API Version:  snapshot.storage.k8s.io/v1beta1
 Kind:         VolumeSnapshotContent
 Metadata:
  Creation Timestamp:  2019-11-16T00:36:04Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  Generation:        1
  Resource Version:  1292
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-32ceaa2a-3802-4edd-a808-58c4f1bd7869
  UID:               7dfdf22e-0b0c-4b71-9ddf-2f1612ca2aed
 Spec:
  Deletion Policy:  Delete
  Driver:           testdriver.csi.k8s.io
  Source:
    Volume Handle:             d1b34a5f-0808-11ea-808a-0242ac110003
  Volume Snapshot Class Name:  test-snapclass
  Volume Snapshot Ref:
    API Version:       snapshot.storage.k8s.io/v1beta1
    Kind:              VolumeSnapshot
    Name:              test-snapshot
    Namespace:         default
    Resource Version:  1286
    UID:               32ceaa2a-3802-4edd-a808-58c4f1bd7869
 Status:
  Creation Time:    1573864564608810101
  Ready To Use:     true
  Restore Size:     1073741824
  Snapshot Handle:  127c5798-0809-11ea-808a-0242ac110003
 Events:             <none>
 ```
 #### Importing an existing volume snapshot with Kubernetes
 You can always expose a pre-existing volume snapshot in Kubernetes by manually creating a VolumeSnapshotContent object to represent the existing volume snapshot. Because VolumeSnapshotContent is a non-namespace API object, only a cluster admin may have the permission to create it. By specifying the `volumeSnapshotRef` the cluster admin specifies exactly which user can use the snapshot.
 The following VolumeSnapshotContent, for example exposes a volume snapshot with the name `7bdd0de3-aaeb-11e8-9aae-0242ac110002` belonging to a CSI driver called `testdriver.csi.k8s.io`.
 A VolumeSnapshotContent object should be created by a cluster admin with the following fields to represent an existing snapshot:
 - `driver` - CSI driver used to handle this volume. This field is required.
 - `source` - Snapshot identifying information
 - `snapshotHandle` - name/identifier of the snapshot. This field is required.
 - `volumeSnapshotRef` - Pointer to the VolumeSnapshot object this content should bind to.
 - `name` and `namespace` - Specifies the name and namespace of the VolumeSnapshot object which the content is bound to.
 - `deletionPolicy` - Valid values are `Delete` and `Retain`. If the `deletionPolicy` is `Delete`, then the underlying storage snapshot will be deleted along with the VolumeSnapshotContent object. If the - `deletionPolicy` is `Retain`, then both the underlying snapshot and VolumeSnapshotContent remain.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshotContent
 metadata:
  name: manually-created-snapshot-content
 spec:
  deletionPolicy: Delete
  driver: testdriver.csi.k8s.io
  source:
    snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002
  volumeSnapshotRef:
    name: test-snapshot
    namespace: default
 ```
 Once a VolumeSnapshotContent object is created, a user can create a VolumeSnapshot object pointing to the VolumeSnapshotContent object. The name and namespace of the VolumeSnapshot object must match the name/namespace specified in the volumeSnapshotRef of the VolumeSnapshotContent. It specifies the following fields:
 `volumeSnapshotContentName` - name of the volume snapshot content specified above. This field is required.
 `volumeSnapshotClassName` - name of the volume snapshot class. This field is optional.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshot
 metadata:
  name: manually-created-snapshot
 spec:
  source:
        volumeSnapshotContentName: test-content
 ```
 Once both objects are created, the common snapshot controller verifies the binding between VolumeSnapshot and VolumeSnapshotContent objects is correct and marks the VolumeSnapshot as ready (if the CSI driver supports the `ListSnapshots` call, the controller also validates that the referenced snapshot exists). The CSI external-snapshotter sidecar checks if the snapshot exists if ListSnapshots CSI method is implemented, otherwise it assumes the snapshot exists. The external-snapshotter sidecar sets `readyToUse` to true in the status field of VolumeSnapshotContent. The common snapshot controller marks the snapshot as ready accordingly.
 ## Create Volume From Snapshot
 Once you have a bound and ready VolumeSnapshot object, you can use that object to provision a new volume that is pre-populated with data from the snapshot.
 To provision a new volume pre-populated with data from a snapshot, use the `dataSource` field in the `PersistentVolumeClaim`. It has three parameters:
 `name` - name of the VolumeSnapshot object representing the snapshot to use as source
 `kind` - must be VolumeSnapshot
 `apiGroup` - must be snapshot.storage.k8s.io
 The namespace of the source VolumeSnapshot object is assumed to be the same as the namespace of the `PersistentVolumeClaim` object.
 ```yaml
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: pvc-restore
  namespace: demo-namespace
 spec:
  storageClassName: testdriver.csi.k8s.io
  dataSource:
    name: manually-created-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
 ```
 When the `PersistentVolumeClaim` object is created, it will trigger provisioning of a new volume that is pre-populated with data from the specified snapshot.
 As a storage vendor, how do I add support for snapshots to my CSI driver?
 To implement the snapshot feature, a CSI driver MUST add support for additional controller capabilities `CREATE_DELETE_SNAPSHOT` and `LIST_SNAPSHOTS`, and implement additional controller RPCs: `CreateSnapshot`, `DeleteSnapshot`, and `ListSnapshots`. For details, see the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md) and the [Kubernetes-CSI Driver Developer Guide](https://kubernetes-csi.github.io/docs/snapshot-restore-feature.html).
 Although Kubernetes is as minimally prescriptive on the packaging and deployment of a CSI Volume Driver as possible, it provides a suggested mechanism for deploying an arbitrary containerized CSI driver on Kubernetes to simplify deployment of containerized CSI compatible volume drivers.
 As part of this recommended deployment process, the Kubernetes team provides a number of sidecar (helper) containers, including the [external-snapshotter sidecar](https://kubernetes-csi.github.io/docs/external-snapshotter.html) container.
 The external-snapshotter watches the Kubernetes API server for VolumeSnapshotContent object and triggers `CreateSnapshot` and `DeleteSnapshot` operations against a CSI endpoint. The CSI [external-provisioner sidecar container](https://kubernetes-csi.github.io/docs/external-provisioner.html) has also been updated to support restoring volume from snapshot using the dataSource PVC field.
 In order to support snapshot feature, it is recommended that storage vendors deploy the external-snapshotter sidecar containers in addition to the external provisioner, along with their CSI driver.
 ## What are the limitations of beta?
 The beta implementation of volume snapshots for Kubernetes has the following limitations:
 - Does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).
 - No snapshot consistency guarantees beyond any guarantees provided by storage system (e.g. crash consistency). These are the responsibility of higher level APIs/controllers
 ##  What’s next?
 Depending on feedback and adoption, the Kubernetes team plans to push the CSI Snapshot implementation to GA in either 1.18 or 1.19. Some of the features we are interested in supporting include consistency groups, application consistent snapshots, workload quiescing, in-place restores, volume backups, and more.
 ## How can I learn more?
 You can also have a look at the [external-snapshotter source code repository](https://github.com/kubernetes-csi/external-snapshotter).
 Check out additional documentation on the snapshot feature [here](http://k8s.io/docs/concepts/storage/volume-snapshots) and [here](https://kubernetes-csi.github.io/docs/).
 ## How do I get involved?
 This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.
 We offer a huge thank you to the contributors who stepped up these last few quarters to help the project reach Beta:
 - Xing Yang (xing-yang)
 - Xiangqian Yu (yuxiangqian)
 - Jing Xu (jingxu97)
 - Grant Griffiths (ggriffiths)
 - Can Zhu (zhucan)
 With special thanks to the following people for their insightful reviews and thorough consideration with the design:
 - Michelle Au (msau42)
 - Saad Ali (saadali)
 - Patrick Ohly (pohly)
 - Tim Hockin (thockin)
 - Jordan Liggitt (liggitt).
 Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors.
 We also hold regular [SIG-Storage Snapshot Working Group meetings](https://docs.google.com/document/d/1qdfvAj5O-tTAZzqJyz3B-yczLLxOiQd-XKpJmTEMazs/edit?usp=sharing). New attendees are welcome to join for design and development discussions.
@@ -81,10 +81,19 @@ the `Terminating` or `Unknown` state. In cases where Kubernetes cannot deduce fr
 permanently left a cluster, the cluster administrator may need to delete the node object by hand.  Deleting the node object from
 Kubernetes causes all the Pod objects running on the node to be deleted from the apiserver, and frees up their names.
-The node lifecycle controller automatically creates
+In version 1.12, `TaintNodesByCondition` feature is promoted to beta, so node lifecycle controller automatically creates
 [taints](/docs/concepts/configuration/taint-and-toleration/) that represent conditions.
-When the scheduler is assigning a Pod to a Node, the scheduler takes the Node's taints
+Similarly the scheduler ignores conditions when considering a Node; instead
-into account, except for any taints that the Pod tolerates.
+it looks at the Node's taints and a Pod's tolerations.
 Now users can choose between the old scheduling model and a new, more flexible scheduling model.
 A Pod that does not have any tolerations gets scheduled according to the old model. But a Pod that
 tolerates the taints of a particular Node can be scheduled on that Node.
 {{< caution >}}
 Enabling this feature creates a small delay between the
 time when a condition is observed and when a taint is created. This delay is usually less than one second, but it can increase the number of Pods that are successfully scheduled but rejected by the kubelet.
 {{< /caution >}}
 ### Capacity and Allocatable {#capacity}
@@ -163,28 +172,18 @@ to be unreachable. (The default timeouts are 40s to start reporting
 ConditionUnknown and 5m after that to start evicting pods.) The node controller
 checks the state of each node every `--node-monitor-period` seconds.
-#### Heartbeats
+In versions of Kubernetes prior to 1.13, NodeStatus is the heartbeat from the
-
+node. Node lease feature is enabled by default since 1.14 as a beta feature
-Heartbeats, sent by Kubernetes nodes, help determine the availability of a node.
+(feature gate `NodeLease`, [KEP-0009](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/0009-node-heartbeat.md)).
-There are two forms of heartbeats: updates of `NodeStatus` and the
+When node lease feature is enabled, each node has an associated `Lease` object in
-[Lease object](/docs/reference/generated/kubernetes-api/{{< latest-version >}}/#lease-v1-coordination-k8s-io).
+`kube-node-lease` namespace that is renewed by the node periodically, and both
-Each Node has an associated Lease object in the `kube-node-lease`
+NodeStatus and node lease are treated as heartbeats from the node. Node leases
-{{< glossary_tooltip term_id="namespace" text="namespace">}}.
+are renewed frequently while NodeStatus is reported from node to master only
-Lease is a lightweight resource, which improves the performance
+when there is some change or enough time has passed (default is 1 minute, which
-of the node heartbeats as the cluster scales.
+is longer than the default timeout of 40 seconds for unreachable nodes). Since
-
+node lease is much more lightweight than NodeStatus, this feature makes node
-The kubelet is responsible for creating and updating the `NodeStatus` and
+heartbeat significantly cheaper from both scalability and performance
-a Lease object.
+perspectives.
 - The kubelet updates the `NodeStatus` either when there is change in status,
  or if there has been no update for a configured interval. The default interval
  for `NodeStatus` updates is 5 minutes (much longer than the 40 second default
  timeout for unreachable nodes).
 - The kubelet creates and then updates its Lease object every 10 seconds
  (the default update interval). Lease updates occur independently from the
  `NodeStatus` updates.
 #### Reliability
 In Kubernetes 1.4, we updated the logic of the node controller to better handle
 cases when a large number of nodes have problems with reaching the master
@@ -76,12 +76,9 @@ In addition to labels you [attach](#step-one-attach-label-to-the-node), nodes co
 with a standard set of labels. These labels are
 * [`kubernetes.io/hostname`](/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-hostname)
-* [`failure-domain.beta.kubernetes.io/zone`](/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone)
+* [`failure-domain.beta.kubernetes.io/zone`](/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domain-beta-kubernetes-io-zone)
-* [`failure-domain.beta.kubernetes.io/region`](/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesioregion)
+* [`failure-domain.beta.kubernetes.io/region`](/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domain-beta-kubernetes-io-region)
 * [`topology.kubernetes.io/zone`](/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone)
 * [`topology.kubernetes.io/region`](/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone)
 * [`beta.kubernetes.io/instance-type`](/docs/reference/kubernetes-api/labels-annotations-taints/#beta-kubernetes-io-instance-type)
 * [`node.kubernetes.io/instance-type`](/docs/reference/kubernetes-api/labels-annotations-taints/#nodekubernetesioinstance-type)
 * [`kubernetes.io/os`](/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-os)
 * [`kubernetes.io/arch`](/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-arch)
@@ -280,7 +280,7 @@ which matches the behavior when this feature is disabled.
 ## Taint Nodes by Condition
-The node lifecycle controller automatically creates taints corresponding to
+In version 1.12, `TaintNodesByCondition` feature is promoted to beta, so node lifecycle controller automatically creates taints corresponding to
 Node conditions.
 Similarly the scheduler does not check Node conditions; instead the scheduler checks taints. This assures that Node conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as Node conditions) by adding appropriate Pod tolerations.
 Note that `TaintNodesByCondition` only taints nodes with `NoSchedule` effect. `NoExecute` effect is controlled by `TaintBasedEviction` which is a beta feature and enabled by default since version 1.13.
@@ -174,7 +174,7 @@ Aggregated APIs offer more advanced API features and customization of other feat
 | Feature | Description | CRDs | Aggregated API |
 | ------- | ----------- | ---- | -------------- |
 | Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Yes.  Most validation can be specified in the CRD using [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation).  Any other validations supported by addition of a [Validating Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook-alpha-in-1-8-beta-in-1-9). | Yes, arbitrary validation checks |
-| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) `default` keyword (GA in 1.17), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) (though this will not be run when reading from etcd for old objects) | Yes |
+| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) `default` keyword (beta in 1.16), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9) | Yes |
 | Multi-versioning | Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions. | [Yes](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning) | Yes |
 | Custom Storage | If you need storage with a different performance mode (for example, time-series database instead of key-value store) or isolation for security (for example, encryption secrets or different | No | Yes |
 | Custom Business Logic | Perform arbitrary checks or actions when creating, reading, updating or deleting an object | Yes, using [Webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks). | Yes |
@@ -184,32 +184,6 @@ DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a
 Support for the "PodResources service" requires `KubeletPodResources` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled. It is enabled by default starting with Kubernetes 1.15.
 ## Device Plugin integration with the Topology Manager
 {{< feature-state for_k8s_version="v1.17" state="alpha" >}}
 The Topology Manager is a Kubelet component that allows resources to be co-ordintated in a Topology aligned manner. In order to do this, the Device Plugin API was extended to include a `TopologyInfo` struct. 
 ```gRPC
 message TopologyInfo {
 	repeated NUMANode nodes = 1;
 }
 message NUMANode {
    int64 ID = 1;
 }
 ```
 Device Plugins that wish to leverage the Topology Manager can send back a populated TopologyInfo struct as part of the device registration, along with the device IDs and the health of the device. The device manager will then use this information to consult with the Topology Manager and make resource assingment decisions.
 `TopologyInfo` supports a `nodes` field that is either `nil` (the default) or a list of NUMA nodes. This lets the Device Plugin publish that can span NUMA nodes.
 An example `TopologyInfo` struct populated for a device by a Device Plugin:
 ```
 pluginapi.Device{ID: "25102017", Health: pluginapi.Healthy, Topology:&pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{&pluginapi.NUMANode{ID: 0,},}}}
 ```
 ## Device plugin examples {#examples}
 Here are some examples of device plugin implementations:
@@ -231,6 +205,5 @@ Here are some examples of device plugin implementations:
 * Learn about [scheduling GPU resources](/docs/tasks/manage-gpus/scheduling-gpus/) using device plugins
 * Learn about [advertising extended resources](/docs/tasks/administer-cluster/extended-resource-node/) on a node
 * Read about using [hardware acceleration for TLS ingress](https://kubernetes.io/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes
 * Learn about the [Topology Manager] (/docs/tasks/adminster-cluster/topology-manager/)
 {{% /capture %}}
@@ -537,33 +537,12 @@ With this mechanism, operators will be able to restrict usage of certain high pr
 To enforce this, kube-apiserver flag `--admission-control-config-file` should be used to pass path to the following configuration file:
 {{< tabs name="example1" >}}
 {{% tab name="apiserver.config.k8s.io/v1" %}}
 ```yaml
 apiVersion: apiserver.config.k8s.io/v1
 kind: AdmissionConfiguration
 plugins:
 - name: "ResourceQuota"
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: ResourceQuotaConfiguration
    limitedResources:
    - resource: pods
      matchScopes:
      - scopeName: PriorityClass 
        operator: In
        values: ["cluster-services"]
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
 - name: "ResourceQuota"
  configuration:
    # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, ResourceQuotaConfiguration
    apiVersion: resourcequota.admission.k8s.io/v1beta1
    kind: Configuration
    limitedResources:
@@ -573,8 +552,6 @@ plugins:
        operator: In
        values: ["cluster-services"]
 ```
 {{% /tab %}}
 {{< /tabs >}}
 Now, "cluster-services" pods will be allowed in only those namespaces where a quota object with a matching `scopeSelector` is present.
 For example:
@@ -51,7 +51,6 @@ To enable IPv4/IPv6 dual-stack, enable the `IPv6DualStack` [feature gate](/docs/
      * `--feature-gates="IPv6DualStack=true"`
      * `--cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR>` eg. `--cluster-cidr=10.244.0.0/16,fc00::/24`
      * `--service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>`
      * `--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6
   * kubelet:
      * `--feature-gates="IPv6DualStack=true"`
   * kube-proxy:
@@ -99,7 +98,9 @@ The use of publicly routable and non-publicly routable IPv6 address blocks is ac
 ## Known Issues
   * IPv6 network block assignment uses the default IPv4 CIDR block size (/24)
   * Kubenet forces IPv4,IPv6 positional reporting of IPs (--cluster-cidr)
   * Dual-stack networking does not function if the `EndpointSlice` feature gate is enabled.
 {{% /capture %}}
@@ -14,7 +14,7 @@ weight: 10
 {{% capture overview %}}
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
 _Endpoint Slices_ provide a simple way to track network endpoints within a
 Kubernetes cluster. They offer a more scalable and extensible alternative to
@@ -26,7 +26,7 @@ Endpoints.
 ## Endpoint Slice resources {#endpointslice-resource}
-In Kubernetes, an EndpointSlice contains references to a set of network
+In Kubernetes, an Endpoint Slice contains references to a set of network
 endpoints. The EndpointSlice controller automatically creates Endpoint Slices
 for a Kubernetes Service when a selector is specified. These Endpoint Slices
 will include references to any Pods that match the Service selector. Endpoint
@@ -36,13 +36,13 @@ As an example, here's a sample EndpointSlice resource for the `example`
 Kubernetes Service.
 ```yaml
-apiVersion: discovery.k8s.io/v1beta1
+apiVersion: discovery.k8s.io/v1alpha1
 kind: EndpointSlice
 metadata:
  name: example-abc
  labels:
    kubernetes.io/service-name: example
-addressType: IPv4
+addressType: IP
 ports:
  - name: http
    protocol: TCP
@@ -50,6 +50,7 @@ ports:
 endpoints:
  - addresses:
    - "10.1.2.3"
    - "2001:db8::1234:5678"
    conditions:
      ready: true
    hostname: pod-1
@@ -66,14 +67,6 @@ Endpoint Slices can act as the source of truth for kube-proxy when it comes to
 how to route internal traffic. When enabled, they should provide a performance
 improvement for services with large numbers of endpoints.
 ## Address Types
 EndpointSlices support three address types:
 * IPv4
 * IPv6
 * FQDN (Fully Qualified Domain Name)
 ## Motivation
 The Endpoints API has provided a simple and straightforward way of
@@ -1,127 +0,0 @@
 ---
 reviewers:
 - johnbelamaric
 - imroc
 title: Service Topology
 feature:
  title: Service Topology
  description: >
    Routing of service traffice based upon cluster topology.
 content_template: templates/concept
 weight: 10
 ---
 {{% capture overview %}}
 {{< feature-state for_k8s_version="v1.17" state="alpha" >}}
 _Service Topology_ enables a service to route traffic based upon the Node
 topology of the cluster. For example, a service can specify that traffic be
 preferentially routed to endpoints that are on the same Node as the client, or
 in the same availability zone.
 {{% /capture %}}
 {{% capture body %}}
 ## Introduction
 By default, traffic sent to a `ClusterIP` or `NodePort` Service may be routed to
 any backend address for the Service. Since Kubernetes 1.7 it has been possible
 to route "external" traffic to the Pods running on the Node that received the
 traffic, but this is not supported for `ClusterIP` Services, and more complex
 topologies &mdash; such as routing zonally &mdash; have not been possible. The
 _Service Topology_ feature resolves this by allowing the Service creator to
 define a policy for routing traffic based upon the Node labels for the
 originating and destination Nodes.
 By using Node label matching between the source and destination, the operator
 may designate groups of Nodes that are "closer" and "farther" from one another,
 using whatever metric makes sense for that operator's requirements. For many
 operators in public clouds, for example, there is a preference to keep service
 traffic withing the same zone, because interzonal traffic has a cost associated
 with it, while intrazonal traffic does not. Other common needs include being able
 to route traffic to a local Pod managed by a DaemonSet, or keeping traffic to
 Nodes connected to the same top-of-rack switch for the lowest latency.
 ## Prerequisites
 The following prerequisites are needed in order to enable topology aware service
 routing:
   * Kubernetes 1.17 or later
   * Kube-proxy running in iptables mode or IPVS mode
   * Enable [Endpoint Slices](/docs/concepts/services-networking/endpoint-slices/)
 ## Enable Service Topology
 To enable service topology, enable the `ServiceTopology` feature gate for
 kube-apiserver and kube-proxy:
 ```
 --feature-gates="ServiceTopology=true"
 ```
 ## Using Service Topology
 If your cluster has Service Topology enabled, you can control Service traffic
 routing by specifying the `topologyKeys` field on the Service spec. This field
 is a preference-order list of Node labels which will be used to sort endpoints
 when accessing this Service. Traffic will be directed to a Node whose value for
 the first label matches the originating Node's value for that label. If there is
 no backend for the Service on a matching Node, then the second label will be
 considered, and so forth, until no labels remain.
 If no match is found, the traffic will be rejected, just as if there were no
 backends for the Service at all. That is, endpoints are chosen based on the first
 topology key with available backends. If this field is specified and all entries
 have no backends that match the topology of the client, the service has no
 backends for that client and connections should fail. The special value `"*"` may
 be used to mean "any topology". This catch-all value, if used, only makes sense
 as the last value in the list.
 If `topologyKeys` is not specified or empty, no topology constraints will be applied.
 Consider a cluster with Nodes that are labeled with their hostname, zone name,
 and region name. Then you can set the `topologyKeys` values of a service to direct
 traffic as follows.
 * Only to endpoints on the same node, failing if no endpoint exists on the node:
  `["kubernetes.io/hostname"]`.
 * Preferentially to endpoints on the same node, falling back to endpoints in the
  same zone, followed by the same region, and failing otherwise: `["kubernetes.io/hostname",
  "topology.kubernetes.io/zone", "topology.kubernetes.io/region"]`.
  This may be useful, for example, in cases where data locality is critical.
 * Preferentially to the same zone, but fallback on any available endpoint if
  none are available within this zone:
  `["topology.kubernetes.io/zone", "*"]`.
 ## Constraints
 * Service topology is not compatible with `externalTrafficPolicy=Local`, and
  therefore a Service cannot use both of these features. It is possible to use
  both features in the same cluster on different Services, just not on the same
  Service.
 * Valid topology keys are currently limited to `kubernetes.io/hostname`,
  `topology.kubernetes.io/zone`, and `topology.kubernetes.io/region`, but will
  be generalized to other node labels in the future.
 * Topology keys must be valid label keys and at most 16 keys may be specified.
 * The catch-all value, `"*"`, must be the last value in the topology keys, if
  it is used.
 {{% /capture %}}
 {{% capture whatsnext %}}
 * Read about [enabling Service Topology](/docs/tasks/administer-cluster/enabling-service-topology)
 * Read [Connecting Applications with Services](/docs/concepts/services-networking/connect-applications-service/)
 {{% /capture %}}
@@ -185,7 +185,7 @@ selectors and uses DNS names instead. For more information, see the
 [ExternalName](#externalname) section later in this document.
 ### Endpoint Slices
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
 Endpoint Slices are an API resource that can provide a more scalable alternative
 to Endpoints. Although conceptually quite similar to Endpoints, Endpoint Slices
@@ -185,10 +185,10 @@ The following plugins support `WaitForFirstConsumer` with pre-created Persistent
 * All of the above
 * [Local](#local)
-{{< feature-state state="stable" for_k8s_version="1.17" >}}
+{{< feature-state state="beta" for_k8s_version="1.14" >}}
 [CSI volumes](/docs/concepts/storage/volumes/#csi) are also supported with dynamic provisioning
 and pre-created PVs, but you'll need to look at the documentation for a specific CSI driver
-to see its supported topology keys and examples.
+to see its supported topology keys and examples. The `CSINodeInfo` feature gate must be enabled.
 ### Allowed Topologies
@@ -38,8 +38,6 @@ that can be attached to a Node:
 You can change these limits by setting the value of the
 `KUBE_MAX_PD_VOLS` environment variable, and then starting the scheduler.
 CSI drivers might have a different procedure, see their documentation
 on how to customize their limits.
 Use caution if you set a limit that is higher than the default limit. Consult
 the cloud provider's documentation to make sure that Nodes can actually support
@@ -49,7 +47,10 @@ The limit applies to the entire cluster, so it affects all Nodes.
 ## Dynamic volume limits
-{{< feature-state state="stable" for_k8s_version="v1.17" >}}
+{{< feature-state state="beta" for_k8s_version="v1.12" >}}
 Kubernetes 1.11 introduced support for dynamic volume limits based on Node type as an Alpha feature.
 In Kubernetes 1.12 this feature is graduating to Beta and will be enabled by default.
 Dynamic volume limits are supported for following volume types.
@@ -58,12 +59,14 @@ Dynamic volume limits are supported for following volume types.
 - Azure Disk
 - CSI
-For volumes managed by in-tree volume plugins, Kubernetes automatically determines the Node
+
-type and enforces the appropriate maximum number of volumes for the node. For example:
+When the dynamic volume limits feature is enabled, Kubernetes automatically
 determines the Node type and enforces the appropriate number of attachable
 volumes for the node. For example:
 * On
 <a href="https://cloud.google.com/compute/">Google Compute Engine</a>,
-up to 127 volumes can be attached to a node, [depending on the node
+up to 128 volumes can be attached to a node, [depending on the node
 type](https://cloud.google.com/compute/docs/disks/#pdnumberlimits).
 * For Amazon EBS disks on M5,C5,R5,T3 and Z1D instance types, Kubernetes allows only 25
@@ -73,9 +76,7 @@ Kubernetes allows 39 volumes to be attached to a Node.
 * On Azure, up to 64 disks can be attached to a node, depending on the node type. For more details, refer to [Sizes for virtual machines in Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes).
-* If a CSI storage driver advertises a maximum number of volumes for a Node (using `NodeGetInfo`), the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}} honors that limit.
+* For CSI, any driver that advertises volume attach limits via CSI specs will have those limits available as the Node's allocatable property
-Refer to the [CSI specifications](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfo) for details.
+  and the Scheduler will not schedule Pods with volumes on any Node that is already at its capacity. Refer to the [CSI specs](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfo) for more details.
 * For volumes managed by in-tree plugins that have been migrated to a CSI driver, the maximum number of volumes will be the one reported by the CSI driver.
 {{% /capture %}}
@@ -1,11 +1,9 @@
 ---
 reviewers:
 - jsafrane
 - saad-ali
 - thockin
 - msau42
 - jingxu97
 - xing-yang
 - yuxiangqian
 title: Volume Snapshot Classes
 content_template: templates/concept
 weight: 30
@@ -30,7 +28,7 @@ way to describe the "classes" of storage when provisioning a volume snapshot.
 ## The VolumeSnapshotClass Resource
-Each `VolumeSnapshotClass` contains the fields `driver`, `deletionPolicy`, and `parameters`,
+Each `VolumeSnapshotClass` contains the fields `snapshotter` and `parameters`,
 which are used when a `VolumeSnapshot` belonging to the class needs to be
 dynamically provisioned.
@@ -43,30 +41,23 @@ Administrators can specify a default `VolumeSnapshotClass` just for VolumeSnapsh
 that don't request any particular class to bind to.
 ```yaml
-apiVersion: snapshot.storage.k8s.io/v1beta1
+apiVersion: snapshot.storage.k8s.io/v1alpha1
 kind: VolumeSnapshotClass
 metadata:
  name: csi-hostpath-snapclass
-driver: hostpath.csi.k8s.io 
+snapshotter: csi-hostpath
 deletionPolicy: Delete
 parameters:
 ```
-### Driver
+### Snapshotter
-Volume snapshot classes have a driver that determines what CSI volume plugin is
+Volume snapshot classes have a snapshotter that determines what CSI volume plugin is
 used for provisioning VolumeSnapshots. This field must be specified.
 ### DeletionPolicy
 Volume snapshot classes have a deletionPolicy. It enables you to configure what happens to a `VolumeSnapshotContent` when the `VolumeSnapshot` object it is bound to is to be deleted. The deletionPolicy of a volume snapshot can either be `Retain` or `Delete`. This field must be specified.
 If the deletionPolicy is `Delete`, then the underlying storage snapshot will be deleted along with the `VolumeSnapshotContent` object. If the deletionPolicy is `Retain`, then both the underlying snapshot and `VolumeSnapshotContent` remain.
 ## Parameters
 Volume snapshot classes have parameters that describe volume snapshots belonging to
 the volume snapshot class. Different parameters may be accepted depending on the
-`driver`.
+`snapshotter`.
 {{% /capture %}}
@@ -1,11 +1,9 @@
 ---
 reviewers:
 - jsafrane
 - saad-ali
 - thockin
 - msau42
 - jingxu97
 - xing-yang
 - yuxiangqian
 title: Volume Snapshots
 content_template: templates/concept
 weight: 20
@@ -13,8 +11,8 @@ weight: 20
 {{% capture overview %}}
-{{< feature-state for_k8s_version="1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
-In Kubernetes, a _VolumeSnapshot_ represents a snapshot of a volume on a storage system. This document assumes that you are already familiar with Kubernetes [persistent volumes](/docs/concepts/storage/persistent-volumes/).
+This document describes the current state of `VolumeSnapshots` in Kubernetes. Familiarity with [persistent volumes](/docs/concepts/storage/persistent-volumes/) is suggested.
 {{% /capture %}}
@@ -29,15 +27,18 @@ A `VolumeSnapshotContent` is a snapshot taken from a volume in the cluster that
 A `VolumeSnapshot` is a request for snapshot of a volume by a user. It is similar to a PersistentVolumeClaim.
-`VolumeSnapshotClass` allows you to specify different attributes belonging to a `VolumeSnapshot`. These attibutes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same `StorageClass` of a `PersistentVolumeClaim`.
+While `VolumeSnapshots` allow a user to consume abstract storage resources, cluster administrators
 need to be able to offer a variety of `VolumeSnapshotContents` without exposing
 users to the details of how those volume snapshots should be provisioned. For these needs
 there is the `VolumeSnapshotClass` resource.
 Users need to be aware of the following when using this feature:
-* API Objects `VolumeSnapshot`, `VolumeSnapshotContent`, and `VolumeSnapshotClass` are {{< glossary_tooltip term_id="CustomResourceDefinition" text="CRDs" >}}, not part of the core API.
+* API Objects `VolumeSnapshot`, `VolumeSnapshotContent`, and `VolumeSnapshotClass` are CRDs, not part of the core API.
 * `VolumeSnapshot` support is only available for CSI drivers.
-* As part of the deployment process in the beta version of `VolumeSnapshot`, the Kubernetes team provides a snapshot controller to be deployed into the control plane, and a sidecar helper container called csi-snapshotter to be deployed together with the CSI driver.  The snapshot controller watches `VolumeSnapshot` and `VolumeSnapshotContent` objects and is responsible for the creation and deletion of `VolumeSnapshotContent` object in dynamic provisioning.  The sidecar csi-snapshotter watches `VolumeSnapshotContent` objects and triggers `CreateSnapshot` and `DeleteSnapshot` operations against a CSI endpoint.
+* As part of the deployment process, the Kubernetes team provides a sidecar helper container for the snapshot controller called `external-snapshotter`. It watches `VolumeSnapshot` objects and triggers `CreateSnapshot` and `DeleteSnapshot` operations against a CSI endpoint.
-* CSI drivers may or may not have implemented the volume snapshot functionality. The CSI drivers that have provided support for volume snapshot will likely use the csi-snapshotter. See [CSI Driver documentation](https://kubernetes-csi.github.io/docs/) for details.
+* CSI drivers may or may not have implemented the volume snapshot functionality. The CSI drivers that have provided support for volume snapshot will likely use `external-snapshotter`.
-* The CRDs and snapshot controller installations are the responsibility of the Kubernetes distribution.
+* The CSI drivers that support volume snapshot will automatically install CRDs defined for the volume snapshots.
 ## Lifecycle of a volume snapshot and volume snapshot content
@@ -45,104 +46,89 @@ Users need to be aware of the following when using this feature:
 ### Provisioning Volume Snapshot
-There are two ways snapshots may be provisioned: pre-provisioned or dynamically provisioned.
+There are two ways snapshots may be provisioned: statically or dynamically.
-#### Pre-provisioned {#static}
+#### Static
-A cluster administrator creates a number of `VolumeSnapshotContents`. They carry the details of the real volume snapshot on the storage system which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
+A cluster administrator creates a number of `VolumeSnapshotContents`. They carry the details of the real storage which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
 #### Dynamic
-Instead of using a pre-existing snapshot, you can request that a snapshot to be dynamically taken from a PersistentVolumeClaim. The [VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/) specifies storage provider-specific parameters to use when taking a snapshot.
+When none of the static `VolumeSnapshotContents` the administrator created matches a user's `VolumeSnapshot`,
 the cluster may try to dynamically provision a volume snapshot specially for the `VolumeSnapshot` object.
 This provisioning is based on `VolumeSnapshotClasses`: the `VolumeSnapshot` must request a
 [volume snapshot class](/docs/concepts/storage/volume-snapshot-classes/) and
 the administrator must have created and configured that class in order for dynamic
 provisioning to occur.
 ### Binding
-The snapshot controller handles the binding of a `VolumeSnapshot` object with an appropriate `VolumeSnapshotContent` object, in both pre-provisioned and dynamically provisioned scenarios. The binding is a one-to-one mapping.
+A user creates, or has already created in the case of dynamic provisioning, a `VolumeSnapshot` with a specific amount of storage requested and with certain access modes. A control loop watches for new VolumeSnapshots, finds a matching VolumeSnapshotContent (if possible), and binds them together. If a VolumeSnapshotContent was dynamically provisioned for a new VolumeSnapshot, the loop will always bind that VolumeSnapshotContent to the VolumeSnapshot. Once bound, `VolumeSnapshot` binds are exclusive, regardless of how they were bound. A VolumeSnapshot to VolumeSnapshotContent binding is a one-to-one mapping.
-In the case of pre-provisioned binding, the VolumeSnapshot will remain unbound until the requested VolumeSnapshotContent object is created.
+VolumeSnapshots will remain unbound indefinitely if a matching VolumeSnapshotContent does not exist. VolumeSnapshots will be bound as matching VolumeSnapshotContents become available.
-### Persistent Volume Claim as Snapshot Source Protection
+### Persistent Volume Claim in Use Protection
-The purpose of this protection is to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss).
+The purpose of the Persistent Volume Claim Object in Use Protection feature is to ensure that in-use PVC API objects are not removed from the system (as this may result in data loss).
-While a snapshot is being taken of a PersistentVolumeClaim, that PersistentVolumeClaim is in-use. If you delete a PersistentVolumeClaim API object in active use as a snapshot source, the PersistentVolumeClaim object is not removed immediately. Instead, removal of the PersistentVolumeClaim object is postponed until the snapshot is readyToUse or aborted.
+If a PVC is in active use by a snapshot as a source to create the snapshot, the PVC is in-use. If a user deletes a PVC API object in active use as a snapshot source, the PVC object is not removed immediately. Instead, removal of the PVC object is postponed until the PVC is no longer actively used by any snapshots. A PVC is no longer used as a snapshot source when `ReadyToUse` of the snapshot `Status` becomes `true`.
 ### Delete
-Deletion is triggered by deleting the `VolumeSnapshot` object, and the `DeletionPolicy` will be followed. If the `DeletionPolicy` is `Delete`, then the underlying storage snapshot will be deleted along with the `VolumeSnapshotContent` object. If the `DeletionPolicy` is `Retain`, then both the underlying snapshot and `VolumeSnapshotContent` remain.
+Deletion removes both the `VolumeSnapshotContent` object from the Kubernetes API, as well as the associated storage asset in the external infrastructure.
 ## VolumeSnapshots
 Each VolumeSnapshot contains a spec and a status.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshot
 metadata:
  name: new-snapshot-test
 spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-test
 ```
 `persistentVolumeClaimName` is the name of the PersistentVolumeClaim data source for the snapshot. This field is required for dynamically provisioning a snapshot.
 A volume snapshot can request a particular class by specifying the name of a
 [VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/)
 using the attribute `volumeSnapshotClassName`. If nothing is set, then the default class is used if available.
 For pre-provisioned snapshots, you need to specify a `volumeSnapshotContentName` as the source for the snapshot as shown in the following example. The `volumeSnapshotContentName` source field is required for pre-provisioned snapshots.
 ```
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshot
 metadata:
  name: test-snapshot
 spec:
  source:
        volumeSnapshotContentName: test-content
 ```
 ## Volume Snapshot Contents
-Each VolumeSnapshotContent contains a spec and status. In dynamic provisioning, the snapshot common controller creates `VolumeSnapshotContent` objects. Here is an example:
+Each VolumeSnapshotContent contains a spec, which is the specification of the volume snapshot.
 ```yaml
-apiVersion: snapshot.storage.k8s.io/v1beta1
+apiVersion: snapshot.storage.k8s.io/v1alpha1
 kind: VolumeSnapshotContent
 metadata:
  name: snapcontent-72d9a349-aacd-42d2-a240-d775650d2455
 spec:
  deletionPolicy: Delete
  driver: hostpath.csi.k8s.io
  source:
    volumeHandle: ee0cfb94-f8d4-11e9-b2d8-0242ac110002
  volumeSnapshotClassName: csi-hostpath-snapclass
  volumeSnapshotRef:
    name: new-snapshot-test
    namespace: default
    uid: 72d9a349-aacd-42d2-a240-d775650d2455
 ```
 `volumeHandle` is the unique identifier of the volume created on the storage backend and returned by the CSI driver during the volume creation. This field is required for dynamically provisioning a snapshot. It specifies the volume source of the snapshot.
 For pre-provisioned snapshots, you (as cluster administrator) are responsible for creating the `VolumeSnapshotContent` object as follows.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshotContent
 metadata:
  name: new-snapshot-content-test
 spec:
-  deletionPolicy: Delete
+  snapshotClassName: csi-hostpath-snapclass
  driver: hostpath.csi.k8s.io
  source:
-    snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002
+    name: pvc-test
-  volumeSnapshotRef:
+    kind: PersistentVolumeClaim
-    name: new-snapshot-test
+  volumeSnapshotSource:
-    namespace: default
+    csiVolumeSnapshotSource:
      creationTime:    1535478900692119403
      driver:          csi-hostpath
      restoreSize:     10Gi
      snapshotHandle:  7bdd0de3-aaeb-11e8-9aae-0242ac110002
 ```
-`snapshotHandle` is the unique identifier of the volume snapshot created on the storage backend. This field is required for the pre-provisioned snapshots. It specifies the CSI snapshot id on the storage system that this `VolumeSnapshotContent` represents.
+### Class
 A VolumeSnapshotContent can have a class, which is specified by setting the
 `snapshotClassName` attribute to the name of a
 [VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/).
 A VolumeSnapshotContent of a particular class can only be bound to VolumeSnapshots requesting
 that class. A VolumeSnapshotContent with no `snapshotClassName` has no class and can only be bound
 to VolumeSnapshots that request no particular class.
 ## VolumeSnapshots
 Each VolumeSnapshot contains a spec and a status, which is the specification and status of the volume snapshot.
 ```yaml
 apiVersion: snapshot.storage.k8s.io/v1alpha1
 kind: VolumeSnapshot
 metadata:
  name: new-snapshot-test
 spec:
  snapshotClassName: csi-hostpath-snapclass
  source:
    name: pvc-test
    kind: PersistentVolumeClaim
 ```
 ### Class
 A volume snapshot can request a particular class by specifying the name of a
 [VolumeSnapshotClass](/docs/concepts/storage/volume-snapshot-classes/)
 using the attribute `snapshotClassName`.
 Only VolumeSnapshotContents of the requested class, ones with the same `snapshotClassName`
 as the VolumeSnapshot, can be bound to the VolumeSnapshot.
 ## Provisioning Volumes from Snapshots
@@ -151,19 +151,14 @@ spec:
 #### CSI Migration
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.14" state="alpha" >}}
 The CSI Migration feature for awsElasticBlockStore, when enabled, shims all plugin operations
 from the existing in-tree plugin to the `ebs.csi.aws.com` Container
 Storage Interface (CSI) Driver. In order to use this feature, the [AWS EBS CSI
 Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver)
 must be installed on the cluster and the `CSIMigration` and `CSIMigrationAWS`
-Beta features must be enabled.
+Alpha features must be enabled.
 #### CSI Migration Complete
 {{< feature-state for_k8s_version="v1.17" state="alpha" >}}
 To turn off the awsElasticBlockStore storage plugin from being loaded by controller manager and kubelet, you need to set this feature flag to true. This requires `ebs.csi.aws.com` Container Storage Interface (CSI) driver being installed on all worker nodes.
 ### azureDisk {#azuredisk}
@@ -484,14 +479,14 @@ spec:
 #### CSI Migration
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.14" state="alpha" >}}
 The CSI Migration feature for GCE PD, when enabled, shims all plugin operations
 from the existing in-tree plugin to the `pd.csi.storage.gke.io` Container
 Storage Interface (CSI) Driver. In order to use this feature, the [GCE PD CSI
 Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver)
 must be installed on the cluster and the `CSIMigration` and `CSIMigrationGCE`
-Beta features must be enabled.
+Alpha features must be enabled.
 ### gitRepo (deprecated) {#gitrepo}
@@ -1159,7 +1154,7 @@ spec:
 ### Using subPath with expanded environment variables
-{{< feature-state for_k8s_version="v1.17" state="stable" >}}
+{{< feature-state for_k8s_version="v1.15" state="beta" >}}
 Use the `subPathExpr` field to construct `subPath` directory names from Downward API environment variables.
@@ -101,9 +101,21 @@ If you do not specify either, then the DaemonSet controller will create Pods on
 ## How Daemon Pods are Scheduled
-### Scheduled by default scheduler
+### Scheduled by DaemonSet controller (disabled by default since 1.12)
-{{< feature-state state="stable" for-kubernetes-version="1.17" >}}
+Normally, the machine that a Pod runs on is selected by the Kubernetes scheduler. However, Pods
 created by the DaemonSet controller have the machine already selected (`.spec.nodeName` is specified
 when the Pod is created, so it is ignored by the scheduler).  Therefore:
 - The [`unschedulable`](/docs/admin/node/#manual-node-administration) field of a node is not respected
   by the DaemonSet controller.
 - The DaemonSet controller can make Pods even when the scheduler has not been started, which can help cluster
   bootstrap.
 ### Scheduled by default scheduler (enabled by default since 1.12)
 {{< feature-state state="beta" for-kubernetes-version="1.12" >}}
 A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the
 node that a Pod runs on is selected by the Kubernetes scheduler. However,
@@ -128,7 +128,7 @@ repos:
    cd $GOPATH
    git clone https://github.com/kubernetes/kubernetes.git src/k8s.io/kubernetes
    cd src/k8s.io/kubernetes
-    git checkout release-1.17
+    git checkout release-1.16
    make generated_files
    cp -L -R vendor $GOPATH/src
    rm -r vendor
@@ -20,11 +20,11 @@ This section of the Kubernetes documentation contains references.
 * [Kubernetes API Overview](/docs/reference/using-api/api-overview/) - Overview of the API for Kubernetes.
 * Kubernetes API Versions
  * [1.17](/docs/reference/generated/kubernetes-api/v1.17/)
  * [1.16](/docs/reference/generated/kubernetes-api/v1.16/)
  * [1.15](/docs/reference/generated/kubernetes-api/v1.15/)
  * [1.14](/docs/reference/generated/kubernetes-api/v1.14/)
  * [1.13](/docs/reference/generated/kubernetes-api/v1.13/)
  * [1.12](/docs/reference/generated/kubernetes-api/v1.12/)
 ## API Client Libraries
@@ -172,20 +172,7 @@ event requests. The cluster admin can specify event rate limits by:
 * Referencing an `EventRateLimit` configuration file from the file provided to the API
   server's command line flag `--admission-control-config-file`:
 {{< tabs name="eventratelimit_example" >}}
 {{% tab name="apiserver.config.k8s.io/v1" %}}
 ```yaml
 apiVersion: apiserver.config.k8s.io/v1
 kind: AdmissionConfiguration
 plugins:
 - name: EventRateLimit
  path: eventconfig.yaml
 ...
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
@@ -193,8 +180,6 @@ plugins:
  path: eventconfig.yaml
 ...
 ```
 {{% /tab %}}
 {{< /tabs >}}
 There are four types of limits that can be specified in the configuration:
@@ -255,20 +240,7 @@ imagePolicy:
 Reference the ImagePolicyWebhook configuration file from the file provided to the API server's command line flag `--admission-control-config-file`:
 {{< tabs name="imagepolicywebhook_example1" >}}
 {{% tab name="apiserver.config.k8s.io/v1" %}}
 ```yaml
 apiVersion: apiserver.config.k8s.io/v1
 kind: AdmissionConfiguration
 plugins:
 - name: ImagePolicyWebhook
  path: imagepolicyconfig.yaml
 ...
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
@@ -276,44 +248,22 @@ plugins:
  path: imagepolicyconfig.yaml
 ...
 ```
 {{% /tab %}}
 {{< /tabs >}}
 Alternatively, you can embed the configuration directly in the file:
 {{< tabs name="imagepolicywebhook_example2" >}}
 {{% tab name="apiserver.config.k8s.io/v1" %}}
 ```yaml
 apiVersion: apiserver.config.k8s.io/v1
 kind: AdmissionConfiguration
 plugins:
 - name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: <path-to-kubeconfig-file>
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: true
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
 - name: ImagePolicyWebhook
  configuration:
    imagePolicy:
-      kubeConfigFile: <path-to-kubeconfig-file>
+      kubeConfigFile: /path/to/file
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: true
 ```
 {{% /tab %}}
 {{< /tabs >}}
 The ImagePolicyWebhook config file must reference a [kubeconfig](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/) formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS.
@@ -491,11 +441,8 @@ and kubelets will not be allowed to modify labels with that prefix.
  * `kubernetes.io/arch`
  * `kubernetes.io/os`
  * `beta.kubernetes.io/instance-type`
  * `node.kubernetes.io/instance-type`
  * `failure-domain.beta.kubernetes.io/region`
  * `failure-domain.beta.kubernetes.io/zone`
  * `topology.kubernetes.io/region`
  * `topology.kubernetes.io/zone`
  * `kubelet.kubernetes.io/`-prefixed labels
  * `node.kubernetes.io/`-prefixed labels
@@ -542,20 +489,7 @@ podNodeSelectorPluginConfig:
 Reference the `PodNodeSelector` configuration file from the file provided to the API server's command line flag `--admission-control-config-file`:
 {{< tabs name="podnodeselector_example1" >}}
 {{% tab name="apiserver.config.k8s.io/v1" %}}
 ```yaml
 apiVersion: apiserver.config.k8s.io/v1
 kind: AdmissionConfiguration
 plugins:
 - name: PodNodeSelector
  path: podnodeselector.yaml
 ...
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
@@ -563,8 +497,6 @@ plugins:
  path: podnodeselector.yaml
 ...
 ```
 {{% /tab %}}
 {{< /tabs >}}
 #### Configuration Annotation Format
 `PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces.
@@ -184,48 +184,24 @@ the webhooks. There are three steps to complete the configuration.
  (yes, the same schema that's used by kubectl), so the field name is
  `kubeConfigFile`. Here is an example admission control configuration file:
-{{< tabs name="admissionconfiguration_example1" >}}
+    ```yaml
-{{% tab name="apiserver.config.k8s.io/v1" %}}
+    apiVersion: apiserver.k8s.io/v1alpha1
-```yaml
+    kind: AdmissionConfiguration
-apiVersion: apiserver.config.k8s.io/v1
+    plugins:
-kind: AdmissionConfiguration
+    - name: ValidatingAdmissionWebhook
-plugins:
+      configuration:
- name: ValidatingAdmissionWebhook
+        apiVersion: apiserver.config.k8s.io/v1alpha1
-  configuration:
+        kind: WebhookAdmission
-    apiVersion: apiserver.config.k8s.io/v1
+        kubeConfigFile: "<path-to-kubeconfig-file>"
-    kind: WebhookAdmissionConfiguration
+    - name: MutatingAdmissionWebhook
-    kubeConfigFile: "<path-to-kubeconfig-file>"
+      configuration:
- name: MutatingAdmissionWebhook
+        apiVersion: apiserver.config.k8s.io/v1alpha1
-  configuration:
+        kind: WebhookAdmission
-    apiVersion: apiserver.config.k8s.io/v1
+        kubeConfigFile: "<path-to-kubeconfig-file>"
-    kind: WebhookAdmissionConfiguration
+    ```
    kubeConfigFile: "<path-to-kubeconfig-file>"
 ```
 {{% /tab %}}
 {{% tab name="apiserver.k8s.io/v1alpha1" %}}
 ```yaml
 # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
 apiVersion: apiserver.k8s.io/v1alpha1
 kind: AdmissionConfiguration
 plugins:
 - name: ValidatingAdmissionWebhook
  configuration:
    # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, kind=WebhookAdmissionConfiguration
    apiVersion: apiserver.config.k8s.io/v1alpha1
    kind: WebhookAdmission
    kubeConfigFile: "<path-to-kubeconfig-file>"
 - name: MutatingAdmissionWebhook
  configuration:
    # Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, kind=WebhookAdmissionConfiguration
    apiVersion: apiserver.config.k8s.io/v1alpha1
    kind: WebhookAdmission
    kubeConfigFile: "<path-to-kubeconfig-file>"
 ```
 {{% /tab %}}
 {{< /tabs >}}
-For more information about `AdmissionConfiguration`, see the
+The schema of `admissionConfiguration` is defined
-[AdmissionConfiguration schema](https://github.com/kubernetes/kubernetes/blob/v1.17.0/staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1/types.go#L27).
+[here](https://github.com/kubernetes/kubernetes/blob/v1.13.0/staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1alpha1/types.go#L27).
 See the [webhook configuration](#webhook-configuration) section for details about each config field.
 * In the kubeConfig file, provide the credentials:
@@ -218,7 +218,7 @@ cloud-controller-manager [flags]
      <td colspan="2">--feature-gates mapStringBool</td>
    </tr>
    <tr>
-      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
+      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
    </tr>
    <tr>
@@ -52,6 +52,8 @@ different Kubernetes components.
 | `APIListChunking` | `true` | Beta | 1.9 | |
 | `APIResponseCompression` | `false` | Alpha | 1.7 | |
 | `AppArmor` | `true` | Beta | 1.4 | |
 | `AttachVolumeLimit` | `true` | Alpha | 1.11 | 1.11 |
 | `AttachVolumeLimit` | `true` | Beta | 1.12 | |
 | `BalanceAttachedNodeVolumes` | `false` | Alpha | 1.11 | |
 | `BlockVolume` | `false` | Alpha | 1.9 | 1.12 |
 | `BlockVolume` | `true` | Beta | 1.13 | - |
@@ -66,20 +68,14 @@ different Kubernetes components.
 | `CSIDriverRegistry` | `true` | Beta | 1.14 | |
 | `CSIInlineVolume` | `false` | Alpha | 1.15 | 1.15 |
 | `CSIInlineVolume` | `true` | Beta | 1.16 | - |
-| `CSIMigration` | `false` | Alpha | 1.14 | 1.16 |
+| `CSIMigration` | `false` | Alpha | 1.14 | |
 | `CSIMigration` | `false` | Beta | 1.17 | |
 | `CSIMigrationAWS` | `false` | Alpha | 1.14 | |
 | `CSIMigrationAWS` | `false` | Beta | 1.17 | |
 | `CSIMigrationAWSComplete` | `false` | Alpha | 1.17 | |
 | `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | |
 | `CSIMigrationAzureDiskComplete` | `false` | Alpha | 1.17 | |
 | `CSIMigrationAzureFile` | `false` | Alpha | 1.15 | |
-| `CSIMigrationAzureFileComplete` | `false` | Alpha | 1.17 | |
+| `CSIMigrationGCE` | `false` | Alpha | 1.14 | |
 | `CSIMigrationGCE` | `false` | Alpha | 1.14 | 1.16 |
 | `CSIMigrationGCE` | `false` | Beta | 1.17 | |
 | `CSIMigrationGCEComplete` | `false` | Alpha | 1.17 | |
 | `CSIMigrationOpenStack` | `false` | Alpha | 1.14 | |
-| `CSIMigrationOpenStackComplete` | `false` | Alpha | 1.17 | |
+| `CSINodeInfo` | `false` | Alpha | 1.12 | 1.13 |
 | `CSINodeInfo` | `true` | Beta | 1.14 | |
 | `CustomCPUCFSQuotaPeriod` | `false` | Alpha | 1.12 | |
 | `CustomResourceDefaulting` | `false` | Alpha| 1.15 | 1.15 |
 | `CustomResourceDefaulting` | `true` | Beta | 1.16 | |
@@ -90,8 +86,7 @@ different Kubernetes components.
 | `DynamicAuditing` | `false` | Alpha | 1.13 | |
 | `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 |
 | `DynamicKubeletConfig` | `true` | Beta | 1.11 | |
-| `EndpointSlice` | `false` | Alpha | 1.16 | 1.16 |
+| `EndpointSlice` | `false` | Alpha | 1.16 | |
 | `EndpointSlice` | `false` | Beta | 1.17 | |
 | `EphemeralContainers` | `false` | Alpha | 1.16 | |
 | `ExpandCSIVolumes` | `false` | Alpha | 1.14 | 1.15 |
 | `ExpandCSIVolumes` | `true` | Beta | 1.16 | |
@@ -111,24 +106,33 @@ different Kubernetes components.
 | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | |
 | `MountContainers` | `false` | Alpha | 1.9 | |
 | `NodeDisruptionExclusion` | `false` | Alpha | 1.16 | |
 | `NodeLease` | `false` | Alpha | 1.12 | 1.13 |
 | `NodeLease` | `true` | Beta | 1.14 | |
 | `NonPreemptingPriority` | `false` | Alpha | 1.15 | |
 | `PodOverhead` | `false` | Alpha | 1.16 | - |
 | `PodShareProcessNamespace` | `false` | Alpha | 1.10 | 1.11 |
 | `PodShareProcessNamespace` | `true` | Beta | 1.12 | |
 | `ProcMountType` | `false` | Alpha | 1.12 | |
 | `QOSReserved` | `false` | Alpha | 1.11 | |
 | `RemainingItemCount` | `false` | Alpha | 1.15 | |
 | `RequestManagement` | `false` | Alpha | 1.15 | |
 | `ResourceLimitsPriorityFunction` | `false` | Alpha | 1.9 | |
 | `ResourceQuotaScopeSelectors` | `false` | Alpha | 1.11 | 1.11 |
 | `ResourceQuotaScopeSelectors` | `true` | Beta | 1.12 | |
 | `RotateKubeletClientCertificate` | `true` | Beta | 1.8 | |
 | `RotateKubeletServerCertificate` | `false` | Alpha | 1.7 | 1.11 |
 | `RotateKubeletServerCertificate` | `true` | Beta | 1.12 | |
 | `RunAsGroup` | `true` | Beta | 1.14 | |
 | `RuntimeClass` | `false` | Alpha | 1.12 | 1.13 |
 | `RuntimeClass` | `true` | Beta | 1.14 | |
 | `ScheduleDaemonSetPods` | `false` | Alpha | 1.11 | 1.11 |
 | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | |
 | `SCTPSupport` | `false` | Alpha | 1.12 | |
 | `ServerSideApply` | `false` | Alpha | 1.14 | 1.15 |
 | `ServerSideApply` | `true` | Beta | 1.16 | |
 | `ServiceLoadBalancerFinalizer` | `false` | Alpha | 1.15 | |
 | `ServiceNodeExclusion` | `false` | Alpha | 1.8 | |
-| `StartupProbe` | `true` | Beta | 1.17 | |
+| `StartupProbe` | `false` | Alpha | 1.16 | |
 | `StorageVersionHash` | `false` | Alpha | 1.14 | 1.14 |
 | `StorageVersionHash` | `true` | Beta | 1.15 | |
 | `StreamingProxyRedirects` | `false` | Beta | 1.5 | 1.5 |
@@ -140,6 +144,8 @@ different Kubernetes components.
 | `Sysctls` | `true` | Beta | 1.11 | |
 | `TaintBasedEvictions` | `false` | Alpha | 1.6 | 1.12 |
 | `TaintBasedEvictions` | `true` | Beta | 1.13 | |
 | `TaintNodesByCondition` | `false` | Alpha | 1.8 | 1.11 |
 | `TaintNodesByCondition` | `true` | Beta | 1.12 | |
 | `TokenRequest` | `false` | Alpha | 1.10 | 1.11 |
 | `TokenRequest` | `true` | Beta | 1.12 | |
 | `TokenRequestProjection` | `false` | Alpha | 1.11 | 1.11 |
@@ -150,8 +156,11 @@ different Kubernetes components.
 | `ValidateProxyRedirects` | `true` | Beta | 1.14 | |
 | `VolumePVCDataSource` | `false` | Alpha | 1.15 | 1.15 |
 | `VolumePVCDataSource` | `true` | Beta | 1.16 | |
-| `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | 1.16 |
+| `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.14 | 1.14 |
-| `VolumeSnapshotDataSource` | `true` | Beta | 1.17 | - |
+| `VolumeSubpathEnvExpansion` | `true` | Beta | 1.15 | |
 | `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | - |
 | `WatchBookmark` | `false` | Alpha | 1.15 | 1.15 |
 | `WatchBookmark` | `true` | Beta | 1.16 | |
 | `WindowsGMSA` | `false` | Alpha | 1.14 | |
 | `WindowsGMSA` | `true` | Beta | 1.16 | |
 | `WinDSR` | `false` | Alpha | 1.14 | |
@@ -173,12 +182,6 @@ different Kubernetes components.
 | `AffinityInAnnotations` | - | Deprecated | 1.8 | - |
 | `AllowExtTrafficLocalEndpoints` | `false` | Beta | 1.4 | 1.6 |
 | `AllowExtTrafficLocalEndpoints` | `true` | GA | 1.7 | - |
 | `CSINodeInfo` | `false` | Alpha | 1.12 | 1.13 |
 | `CSINodeInfo` | `true` | Beta | 1.14 | 1.16 |
 | `CSINodeInfo` | `true` | GA | 1.17 | |
 | `AttachVolumeLimit` | `false` | Alpha | 1.11 | 1.11 |
 | `AttachVolumeLimit` | `true` | Beta | 1.12 | 1.16 |
 | `AttachVolumeLimit` | `true` | GA | 1.17 | - |
 | `CSIPersistentVolume` | `false` | Alpha | 1.9 | 1.9 |
 | `CSIPersistentVolume` | `true` | Beta | 1.10 | 1.12 |
 | `CSIPersistentVolume` | `true` | GA | 1.13 | - |
@@ -220,9 +223,6 @@ different Kubernetes components.
 | `MountPropagation` | `false` | Alpha | 1.8 | 1.9 |
 | `MountPropagation` | `true` | Beta | 1.10 | 1.11 |
 | `MountPropagation` | `true` | GA | 1.12 | - |
 | `NodeLease` | `false` | Alpha | 1.12 | 1.13 |
 | `NodeLease` | `true` | Beta | 1.14 | 1.16 |
 | `NodeLease` | `true` | GA | 1.17 | - |
 | `PersistentLocalVolumes` | `false` | Alpha | 1.7 | 1.9 |
 | `PersistentLocalVolumes` | `true` | Beta | 1.10 | 1.13 |
 | `PersistentLocalVolumes` | `true` | GA | 1.14 | - |
@@ -232,39 +232,18 @@ different Kubernetes components.
 | `PodReadinessGates` | `false` | Alpha | 1.11 | 1.11 |
 | `PodReadinessGates` | `true` | Beta | 1.12 | 1.13 |
 | `PodReadinessGates` | `true` | GA | 1.14 | - |
 | `PodShareProcessNamespace` | `false` | Alpha | 1.10 | 1.11 |
 | `PodShareProcessNamespace` | `true` | Beta | 1.12 | 1.16 |
 | `PodShareProcessNamespace` | `true` | GA | 1.17 | - |
 | `PVCProtection` | `false` | Alpha | 1.9 | 1.9 |
 | `PVCProtection` | - | Deprecated | 1.10 | - |
 | `ResourceQuotaScopeSelectors` | `false` | Alpha | 1.11 | 1.11 |
 | `ResourceQuotaScopeSelectors` | `true` | Beta | 1.12 | 1.16 |
 | `ResourceQuotaScopeSelectors` | `true` | GA | 1.17 | - |
 | `ScheduleDaemonSetPods` | `false` | Alpha | 1.11 | 1.11 |
 | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | 1.16  |
 | `ScheduleDaemonSetPods` | `true` | GA | 1.17 | - |
 | `ServiceLoadBalancerFinalizer` | `false` | Alpha | 1.15 | 1.15 |
 | `ServiceLoadBalancerFinalizer` | `true` | Beta | 1.16 | 1.16 |
 | `ServiceLoadBalancerFinalizer` | `true` | GA | 1.17 | - |
 | `StorageObjectInUseProtection` | `true` | Beta | 1.10 | 1.10 |
 | `StorageObjectInUseProtection` | `true` | GA | 1.11 | - |
 | `SupportIPVSProxyMode` | `false` | Alpha | 1.8 | 1.8 |
 | `SupportIPVSProxyMode` | `false` | Beta | 1.9 | 1.9 |
 | `SupportIPVSProxyMode` | `true` | Beta | 1.10 | 1.10 |
 | `SupportIPVSProxyMode` | `true` | GA | 1.11 | - |
 | `TaintNodesByCondition` | `false` | Alpha | 1.8 | 1.11 |
 | `TaintNodesByCondition` | `true` | Beta | 1.12 | 1.16 |
 | `TaintNodesByCondition` | `true` | GA | 1.17 | - |
 | `VolumeScheduling` | `false` | Alpha | 1.9 | 1.9 |
 | `VolumeScheduling` | `true` | Beta | 1.10 | 1.12 |
 | `VolumeScheduling` | `true` | GA | 1.13 | - |
 | `VolumeSubpath` | `true` | GA | 1.13 | - |
 | `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.14 | 1.14 |
 | `VolumeSubpathEnvExpansion` | `true` | Beta | 1.15 | 1.16 |
 | `VolumeSubpathEnvExpansion` | `true` | GA | 1.17 | - |
 | `WatchBookmark` | `false` | Alpha | 1.15 | 1.15 |
 | `WatchBookmark` | `true` | Beta | 1.16 | 1.16 |
 | `WatchBookmark` | `true` | GA | 1.17 | - |
 {{< /table >}}
 ## Using a feature
@@ -337,16 +316,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `CSIDriverRegistry`: Enable all logic related to the CSIDriver API object in csi.storage.k8s.io.
 - `CSIInlineVolume`: Enable CSI Inline volumes support for pods.
 - `CSIMigration`: Enables shims and translation logic to route volume operations from in-tree plugins to corresponding pre-installed CSI plugins
- `CSIMigrationAWS`: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Supports falling back to in-tree EBS plugin if a node does not have EBS CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
+- `CSIMigrationAWS`: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin
- `CSIMigrationAWSComplete`: Stops registering the EBS in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Requires CSIMigration and CSIMigrationAWS feature flags enabled and EBS CSI plugin installed and configured on all nodes in the cluster.
+- `CSIMigrationAzureDisk`: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to Azure Disk CSI plugin
- `CSIMigrationAzureDisk`: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Supports falling back to in-tree AzureDisk plugin if a node does not have AzureDisk CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
+- `CSIMigrationAzureFile`: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to Azure File CSI plugin
- `CSIMigrationAzureDiskComplete`: Stops registering the Azure-Disk in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin. Requires CSIMigration and CSIMigrationAzureDisk feature flags enabled and AzureDisk CSI plugin installed and configured on all nodes in the cluster.
+- `CSIMigrationGCE`: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin
- `CSIMigrationAzureFile`: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Supports falling back to in-tree AzureFile plugin if a node does not have AzureFile CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
+- `CSIMigrationOpenStack`: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin
 - `CSIMigrationAzureFileComplete`: Stops registering the Azure-File in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to AzureFile CSI plugin. Requires CSIMigration and CSIMigrationAzureFile feature flags  enabled and AzureFile CSI plugin installed and configured on all nodes in the cluster.
 - `CSIMigrationGCE`: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling back to in-tree GCE plugin if a node does not have PD CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
 - `CSIMigrationGCEComplete`: Stops registering the GCE-PD in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin. Requires CSIMigration and CSIMigrationGCE feature flags enabled and PD CSI plugin installed and configured on all nodes in the cluster.
 - `CSIMigrationOpenStack`: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Supports falling back to in-tree Cinder plugin if a node does not have Cinder CSI plugin installed and configured. Requires CSIMigration feature flag enabled.
 - `CSIMigrationOpenStackComplete`: Stops registering the Cinder in-tree plugin in kubelet and volume controllers and enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin. Requires CSIMigration and CSIMigrationOpenStack feature flags enabled and Cinder CSI plugin installed and configured on all nodes in the cluster.
 - `CSINodeInfo`: Enable all logic related to the CSINodeInfo API object in csi.storage.k8s.io.
 - `CSIPersistentVolume`: Enable discovering and mounting volumes provisioned through a
  [CSI (Container Storage Interface)](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md)
@@ -570,7 +570,7 @@ kube-apiserver [flags]
      <td colspan="2">--feature-gates mapStringBool</td>
    </tr>
    <tr>
-      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
+      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
    </tr>
    <tr>
@@ -371,7 +371,7 @@ kube-controller-manager [flags]
      <td colspan="2">--feature-gates mapStringBool</td>
    </tr>
    <tr>
-      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
+      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
    </tr>
    <tr>
@@ -111,7 +111,7 @@ kube-proxy [flags]
      <td colspan="2">--feature-gates mapStringBool</td>
    </tr>
    <tr>
-      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
+      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
    </tr>
    <tr>
@@ -160,7 +160,7 @@ kube-scheduler [flags]
      <td colspan="2">--feature-gates mapStringBool</td>
    </tr>
    <tr>
-      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
+      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)<br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)<br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false)</td>
    </tr>
    <tr>
@@ -552,7 +552,7 @@ kubelet [flags]
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:<br/>APIListChunking=true|false (BETA - default=true)<br/>APIResponseCompression=true|false (BETA - default=true)<br/>AllAlpha=true|false (ALPHA - default=false)<br/>AppArmor=true|false (BETA - default=true)<br/>AttachVolumeLimit=true|false (BETA - default=true)<br/>BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)<br/>BlockVolume=true|false (BETA - default=true)<br/>BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)<br/>CPUManager=true|false (BETA - default=true)<br/>CRIContainerLogRotation=true|false (BETA - default=true)
-      <br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (BETA - default=true)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
+      <br/>CSIBlockVolume=true|false (BETA - default=true)<br/>CSIDriverRegistry=true|false (BETA - default=true)<br/>CSIInlineVolume=true|false (BETA - default=true)<br/>CSIMigration=true|false (ALPHA - default=false)<br/>CSIMigrationAWS=true|false (ALPHA - default=false)<br/>CSIMigrationAzureDisk=true|false (ALPHA - default=false)<br/>CSIMigrationAzureFile=true|false (ALPHA - default=false)<br/>CSIMigrationGCE=true|false (ALPHA - default=false)<br/>CSIMigrationOpenStack=true|false (ALPHA - default=false)<br/>CSINodeInfo=true|false (BETA - default=true)<br/>CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)<br/>CustomResourceDefaulting=true|false (BETA - default=true)<br/>DevicePlugins=true|false (BETA - default=true)<br/>DryRun=true|false (BETA - default=true)<br/>DynamicAuditing=true|false (ALPHA - default=false)<br/>DynamicKubeletConfig=true|false (BETA - default=true)<br/>EndpointSlice=true|false (ALPHA - default=false)<br/>EphemeralContainers=true|false (ALPHA - default=false)<br/>EvenPodsSpread=true|false (ALPHA - default=false)<br/>ExpandCSIVolumes=true|false (BETA - default=true)<br/>ExpandInUsePersistentVolumes=true|false (BETA - default=true)<br/>ExpandPersistentVolumes=true|false (BETA - default=true)<br/>ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)<br/>HPAScaleToZero=true|false (ALPHA - default=false)<br/>HyperVContainer=true|false (ALPHA - default=false)<br/>IPv6DualStack=true|false (ALPHA - default=false)<br/>KubeletPodResources=true|false (BETA - default=true)<br/>LegacyNodeRoleBehavior=true|false (ALPHA - default=true)<br/>LocalStorageCapacityIsolation=true|false (BETA - default=true)<br/>LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)<br/>MountContainers=true|false (ALPHA - default=false)<br/>NodeDisruptionExclusion=true|false (ALPHA - default=false)<br/>NodeLease=true|false (BETA - default=true)<br/>NonPreemptingPriority=true|false (ALPHA - default=false)<br/>PodOverhead=true|false (ALPHA - default=false)<br/>PodShareProcessNamespace=true|false (BETA - default=true)<br/>ProcMountType=true|false (ALPHA - default=false)br/>QOSReserved=true|false (ALPHA - default=false)<br/>RemainingItemCount=true|false (BETA - default=true)<br/>RemoveSelfLink=true|false (ALPHA - default=false)<br/>RequestManagement=true|false (ALPHA - default=false)<br/>ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)<br/>ResourceQuotaScopeSelectors=true|false (BETA - default=true)<br/>RotateKubeletClientCertificate=true|false (BETA - default=true)<br/>RotateKubeletServerCertificate=true|false (BETA - default=true)<br/>RunAsGroup=true|false (BETA - default=true)<br/>RuntimeClass=true|false (BETA - default=true)<br/>SCTPSupport=true|false (ALPHA - default=false)<br/>ScheduleDaemonSetPods=true|false (BETA - default=true)<br/>ServerSideApply=true|false (BETA - default=true)<br/>ServiceLoadBalancerFinalizer=true|false (BETA - default=true)<br/>ServiceNodeExclusion=true|false (ALPHA - default=false)<br/>StartupProbe=true|false (ALPHA - default=false)<br/>StorageVersionHash=true|false (BETA - default=true)<br/>StreamingProxyRedirects=true|false (BETA - default=true)<br/>SupportNodePidsLimit=true|false (BETA - default=true)<br/>SupportPodPidsLimit=true|false (BETA - default=true)<br/>Sysctls=true|false (BETA - default=true)<br/>TTLAfterFinished=true|false (ALPHA - default=false)<br/>TaintBasedEvictions=true|false (BETA - default=true)<br/>TaintNodesByCondition=true|false (BETA - default=true)<br/>TokenRequest=true|false (BETA - default=true)<br/>TokenRequestProjection=true|false (BETA - default=true)<br/>TopologyManager=true|false (ALPHA - default=false)<br/>ValidateProxyRedirects=true|false (BETA - default=true)<br/>VolumePVCDataSource=true|false (BETA - default=true)<br/>VolumeSnapshotDataSource=true|false (ALPHA - default=false)<br/>VolumeSubpathEnvExpansion=true|false (BETA - default=true)<br/>WatchBookmark=true|false (BETA - default=true)<br/>WinDSR=true|false (ALPHA - default=false)<br/>WinOverlay=true|false (ALPHA - default=false)<br/>WindowsGMSA=true|false (BETA - default=true)<br/>WindowsRunAsUserName=true|false (ALPHA - default=false) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
    </tr>
     <tr>
@@ -28,47 +28,51 @@ For `kubectl run` to satisfy infrastructure as code:
 * Tag the image with a version-specific tag and don't move that tag to a new version. For example, use `:v1234`, `v1.2.3`, `r03062016-1-4`, rather than `:latest` (For more information, see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images)).
 * Capture the parameters in a checked-in script, or at least use `--record` to annotate the created objects with the command line for an image that is lightly parameterized.
 * Pin to a specific [generator](#generators) version, such as `kubectl run --generator=run-pod/v1`.
 * Check in the script for an image that is heavily parameterized.
 * Switch to configuration files checked into source control for features that are needed, but not expressible via `kubectl run` flags.
 * Pin to a specific [generator](#generators) version, such as `kubectl run --generator=deployment/v1beta1`.
 #### Generators
 You can create the following resources using `kubectl run` with the `--generator` flag:
-{{< table caption="Resources you can create using kubectl run" >}}
+| Resource                            | api group          | kubectl command                                   |
-| Resource                             | API group          | kubectl command                                   |
+|-------------------------------------|--------------------|---------------------------------------------------|
-|--------------------------------------|--------------------|---------------------------------------------------|
+| Pod                                 | v1                 | `kubectl run --generator=run-pod/v1`              |
-| Pod                                  | v1                 | `kubectl run --generator=run-pod/v1`              |
+| Replication controller (deprecated) | v1                 | `kubectl run --generator=run/v1`                  |
-| ReplicationController _(deprecated)_ | v1                 | `kubectl run --generator=run/v1`                  |
+| Deployment (deprecated)             | extensions/v1beta1 | `kubectl run --generator=deployment/v1beta1`      |
-| Deployment _(deprecated)_            | extensions/v1beta1 | `kubectl run --generator=deployment/v1beta1`      |
+| Deployment (deprecated)             | apps/v1beta1       | `kubectl run --generator=deployment/apps.v1beta1` |
-| Deployment _(deprecated)_            | apps/v1beta1       | `kubectl run --generator=deployment/apps.v1beta1` |
+| Job (deprecated)                    | batch/v1           | `kubectl run --generator=job/v1`                  |
-| Job _(deprecated)_                   | batch/v1           | `kubectl run --generator=job/v1`                  |
+| CronJob (deprecated)                | batch/v1beta1      | `kubectl run --generator=cronjob/v1beta1`         |
-| CronJob _(deprecated)_               | batch/v2alpha1     | `kubectl run --generator=cronjob/v2alpha1`        |
+| CronJob (deprecated)                | batch/v2alpha1     | `kubectl run --generator=cronjob/v2alpha1`        |
 | CronJob _(deprecated)_               | batch/v1beta1      | `kubectl run --generator=cronjob/v1beta1`         |
 {{< /table >}}
 {{< note >}}
-Generators other than `run-pod/v1` are deprecated.
+`kubectl run --generator` except for `run-pod/v1` is deprecated in v1.12.
 {{< /note >}}
-If you explicitly set `--generator`, kubectl uses the generator you specified. If you invoke `kubectl run` and don't specify a generator, kubectl automatically selects which generator to use based on the other flags you set. The following table lists flags and the generators that are activated if you didn't specify one yourself:
+If you do not specify a generator flag, other flags prompt you to use a specific generator. The following table lists the flags that force you to use specific generators, depending on the version of the cluster:
-{{< table caption="kubectl run flags and the resource they imply" >}}
+|   Generated Resource   | Cluster v1.4 and later | Cluster v1.3          | Cluster v1.2                               | Cluster v1.1 and earlier                   |
-| Flag                    | Generated Resource    |
+|:----------------------:|------------------------|-----------------------|--------------------------------------------|--------------------------------------------|
-|-------------------------|-----------------------|
+| Pod                    | `--restart=Never`      | `--restart=Never`     | `--generator=run-pod/v1`                   | `--restart=OnFailure` OR `--restart=Never` |
-| `--schedule=<schedule>` | CronJob               |
+| Replication Controller | `--generator=run/v1`   | `--generator=run/v1`  | `--generator=run/v1`                       | `--restart=Always`                         |
-| `--restart=Always`      | Deployment            |
+| Deployment             | `--restart=Always`     | `--restart=Always`    | `--restart=Always`                         | N/A                                        |
-| `--restart=OnFailure`   | Job                   |
+| Job                    | `--restart=OnFailure`  | `--restart=OnFailure` | `--restart=OnFailure` OR `--restart=Never` | N/A                                        |
-| `--restart=Never`       | Pod                   |
+| Cron Job               | `--schedule=<cron>`    | N/A                   | N/A                                        | N/A                                        |
 {{< /table >}}
-If you don't specify a generator, kubectl pays attention to other flags in the following order:
+{{< note >}}
 These flags use a default generator only when you have not specified any flag.
 This means that when you combine `--generator` with other flags the generator that you specified later does not change. For example, in a cluster v1.4, if you initially specify
 `--restart=Always`, a Deployment is created; if you later specify `--restart=Always`
 and `--generator=run/v1`, a Replication Controller is created.
 This enables you to pin to a specific behavior with the generator,
 even when the default generator is changed later.
 {{< /note >}}
-1. `--schedule`
+The flags set the generator in the following order: first the `--schedule` flag, then the `--restart` policy flag, and finally the `--generator` flag.
 1. `--restart`
-You can use the `--dry-run` flag to preview the object that would be sent to your cluster, without really submitting it.
+To check the final resource that was created, use the `--dry-run`
 flag, which only prints the object that would be sent to the cluster without really sending it.
 ### `kubectl apply`
@@ -40,6 +40,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">log to standard error as well as files</td>
    </tr>
    <tr>
      <td colspan="2">--application-metrics-count-limit int&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 100</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Max number of application metrics to store (per container)</td>
    </tr>
    <tr>
      <td colspan="2">--as string</td>
    </tr>
@@ -61,6 +68,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Path to the file containing Azure container registry configuration information.</td>
    </tr>
    <tr>
      <td colspan="2">--boot-id-file string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/proc/sys/kernel/random/boot_id"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Comma-separated list of files to check for boot-id. Use the first one that exists.</td>
    </tr>
    <tr>
      <td colspan="2">--cache-dir string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "~/.kube/http-cache"</td>
    </tr>
@@ -103,6 +117,27 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">The name of the kubeconfig cluster to use</td>
    </tr>
    <tr>
      <td colspan="2">--container-hints string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/etc/cadvisor/container_hints.json"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">location of the container hints file</td>
    </tr>
    <tr>
      <td colspan="2">--containerd string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/run/containerd/containerd.sock"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">containerd endpoint</td>
    </tr>
    <tr>
      <td colspan="2">--containerd-namespace string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "k8s.io"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">containerd namespace</td>
    </tr>
    <tr>
      <td colspan="2">--context string</td>
    </tr>
@@ -124,6 +159,97 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.</td>
    </tr>
    <tr>
      <td colspan="2">--disable-root-cgroup-stats</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Disable collecting root Cgroup stats</td>
    </tr>
    <tr>
      <td colspan="2">--docker string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "unix:///var/run/docker.sock"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">docker endpoint</td>
    </tr>
    <tr>
      <td colspan="2">--docker-env-metadata-whitelist string</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">a comma-separated list of environment variable keys that needs to be collected for docker containers</td>
    </tr>
    <tr>
      <td colspan="2">--docker-only</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Only report docker containers in addition to root stats</td>
    </tr>
    <tr>
      <td colspan="2">--docker-root string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/var/lib/docker"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker)</td>
    </tr>
    <tr>
      <td colspan="2">--docker-tls</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">use TLS to connect to docker</td>
    </tr>
    <tr>
      <td colspan="2">--docker-tls-ca string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "ca.pem"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">path to trusted CA</td>
    </tr>
    <tr>
      <td colspan="2">--docker-tls-cert string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "cert.pem"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">path to client certificate</td>
    </tr>
    <tr>
      <td colspan="2">--docker-tls-key string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "key.pem"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">path to private key</td>
    </tr>
    <tr>
      <td colspan="2">--enable-load-reader</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Whether to enable cpu load reader</td>
    </tr>
    <tr>
      <td colspan="2">--event-storage-age-limit string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "default=0"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Max length of time for which to store events (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is a duration. Default is applied to all non-specified event types</td>
    </tr>
    <tr>
      <td colspan="2">--event-storage-event-limit string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "default=0"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Max number of events to store (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is an integer. Default is applied to all non-specified event types</td>
    </tr>
    <tr>
      <td colspan="2">--global-housekeeping-interval duration&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 1m0s</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Interval between global housekeepings</td>
    </tr>
    <tr>
      <td colspan="2">-h, --help</td>
    </tr>
@@ -131,6 +257,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">help for kubectl</td>
    </tr>
    <tr>
      <td colspan="2">--housekeeping-interval duration&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 10s</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Interval between container housekeepings</td>
    </tr>
    <tr>
      <td colspan="2">--insecure-skip-tls-verify</td>
    </tr>
@@ -152,6 +285,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">when logging hits line file:N, emit a stack trace</td>
    </tr>
    <tr>
      <td colspan="2">--log-cadvisor-usage</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Whether to log the usage of the cAdvisor container</td>
    </tr>
    <tr>
      <td colspan="2">--log-dir string</td>
    </tr>
@@ -187,6 +327,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">log to standard error instead of files</td>
    </tr>
    <tr>
      <td colspan="2">--machine-id-file string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "/etc/machine-id,/var/lib/dbus/machine-id"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Comma-separated list of files to check for machine-id. Use the first one that exists.</td>
    </tr>
    <tr>
      <td colspan="2">--match-server-version</td>
    </tr>
@@ -257,6 +404,55 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">logs at or above this threshold go to stderr</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-buffer-duration duration&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 1m0s</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-db string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "cadvisor"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">database name</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-host string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "localhost:8086"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">database host:port</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-password string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "root"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">database password</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-secure</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">use secure connection with database</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-table string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "stats"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">table name</td>
    </tr>
    <tr>
      <td colspan="2">--storage-driver-user string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: "root"</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">database username</td>
    </tr>
    <tr>
      <td colspan="2">--token string</td>
    </tr>
@@ -264,6 +460,13 @@ kubectl [flags]
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Bearer token for authentication to the API server</td>
    </tr>
    <tr>
      <td colspan="2">--update-machine-info-interval duration&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 5m0s</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Interval between machine info updates.</td>
    </tr>
    <tr>
      <td colspan="2">--user string</td>
    </tr>
@@ -1,6 +1,6 @@
 ---
-title: v1.17
+title: v1.16
 weight: 50
 ---
-[Kubernetes API v1.17](/docs/reference/generated/kubernetes-api/v1.17/)
+[Kubernetes API v1.16](/docs/reference/generated/kubernetes-api/v1.16/)
@@ -7,7 +7,7 @@ weight: 60
 {{% capture overview %}}
 Kubernetes reserves all labels and annotations in the kubernetes.io namespace.
-
+  
 This document serves both as a reference to the values and as a coordination point for assigning values.
 {{% /capture %}}
@@ -46,13 +46,9 @@ Used on: Node
 The Kubelet populates this label with the hostname. Note that the hostname can be changed from the "actual" hostname by passing the `--hostname-override` flag to the `kubelet`.
-## beta.kubernetes.io/instance-type (deprecated)
+## beta.kubernetes.io/instance-type
-{{< note >}} Starting in v1.17, this label is deprecated in favor of [node.kubernetes.io/instance-type](#nodekubernetesioinstance-type). {{< /note >}}
+Example: `beta.kubernetes.io/instance-type=m3.medium`
 ## node.kubernetes.io/instance-type {#nodekubernetesioinstance-type}
 Example: `node.kubernetes.io/instance-type=m3.medium`
 Used on: Node
@@ -61,13 +57,11 @@ This will be set only if you are using a `cloudprovider`. This setting is handy
 if you want to target certain workloads to certain instance types, but typically you want
 to rely on the Kubernetes scheduler to perform resource-based scheduling. You should aim to schedule based on properties rather than on instance types (for example: require a GPU, instead of requiring a `g2.2xlarge`).
-## failure-domain.beta.kubernetes.io/region (deprecated) {#failure-domainbetakubernetesioregion}
+## failure-domain.beta.kubernetes.io/region
 See [failure-domain.beta.kubernetes.io/zone](#failure-domainbetakubernetesiozone).
-{{< note >}} Starting in v1.17, this label is deprecated in favor of [topology.kubernetes.io/region](#topologykubernetesioregion). {{< /note >}}
+## failure-domain.beta.kubernetes.io/zone {#failure-domainbetakubernetesiozone}
 ## failure-domain.beta.kubernetes.io/zone (deprecated) {#failure-domainbetakubernetesiozone}
 Example:
@@ -95,39 +89,4 @@ The expectation is that failures of nodes in different zones should be uncorrela
 If `PersistentVolumeLabel` does not support automatic labeling of your PersistentVolumes, you should consider
 adding the labels manually (or adding support for `PersistentVolumeLabel`). With `PersistentVolumeLabel`, the scheduler prevents Pods from mounting volumes in a different zone. If your infrastructure doesn't have this constraint, you don't need to add the zone labels to the volumes at all.
 {{< note >}} Starting in v1.17, this label is deprecated in favor of [topology.kubernetes.io/zone](#topologykubernetesiozone). {{< /note >}}
 ## topology.kubernetes.io/region {#topologykubernetesioregion}
 See [topology.kubernetes.io/zone](#topologykubernetesiozone).
 ## topology.kubernetes.io/zone {#topologykubernetesiozone}
 Example:
 `topology.kubernetes.io/region=us-east-1`
 `topology.kubernetes.io/zone=us-east-1c`
 Used on: Node, PersistentVolume
 On the Node: The `kubelet` populates this with the zone information as defined by the `cloudprovider`.
 This will be set only if you are using a `cloudprovider`. However, you should consider setting this
 on the nodes if it makes sense in your topology.
 On the PersistentVolume: The `PersistentVolumeLabel` admission controller will automatically add zone labels to PersistentVolumes, on GCE and AWS.
 Kubernetes will automatically spread the Pods in a replication controller or service across nodes in a single-zone cluster (to reduce the impact of failures). With multiple-zone clusters, this spreading behaviour is extended across zones (to reduce the impact of zone failures). This is achieved via _SelectorSpreadPriority_.
 _SelectorSpreadPriority_ is a best effort placement. If the zones in your cluster are heterogeneous (for example: different numbers of nodes, different types of nodes, or different pod resource requirements), this placement might prevent equal spreading of your Pods across zones. If desired, you can use homogenous zones (same number and types of nodes) to reduce the probability of unequal spreading.
 The scheduler (through the _VolumeZonePredicate_ predicate) also will ensure that Pods, that claim a given volume, are only placed into the same zone as that volume. Volumes cannot be attached across zones.
 The actual values of zone and region don't matter. Nor is the node hierarchy rigidly defined.
 The expectation is that failures of nodes in different zones should be uncorrelated unless the entire region has failed. For example, zones should typically avoid sharing a single network switch. The exact mapping depends on your particular infrastructure - a three rack installation will choose a very different setup to a multi-datacenter configuration.
 If `PersistentVolumeLabel` does not support automatic labeling of your PersistentVolumes, you should consider
 adding the labels manually (or adding support for `PersistentVolumeLabel`). With `PersistentVolumeLabel`, the scheduler prevents Pods from mounting volumes in a different zone. If your infrastructure doesn't have this constraint, you don't need to add the zone labels to the volumes at all.
 {{% /capture %}}
@@ -39,7 +39,7 @@ kubeadm upgrade node [flags]
    </tr>
    <tr>
-      <td colspan="2">--etcd-upgrade&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: true</td>
+      <td colspan="2">--etcd-upgrade</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Perform the upgrade of etcd.</td>
@@ -32,7 +32,7 @@ kubeadm upgrade node phase control-plane [flags]
    </tr>
    <tr>
-      <td colspan="2">--etcd-upgrade&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: true</td>
+      <td colspan="2">--etcd-upgrade</td>
    </tr>
    <tr>
      <td></td><td style="line-height: 130%; word-wrap: break-word;">Perform the upgrade of etcd.</td>
@@ -51,7 +51,7 @@ The verbs supported for each subresource will differ depending on the object - s
 ## Efficient detection of changes
-To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a **watch**.  Every Kubernetes object has a `resourceVersion` field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a `resourceVersion` value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied `resourceVersion`. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned `resourceVersion`, or perform a new collection request and begin again. See [Resource Version Semantics](#resource-versions) for more detail.
+To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a **watch**.  Every Kubernetes object has a `resourceVersion` field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a `resourceVersion` value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied `resourceVersion`. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned `resourceVersion`, or perform a new collection request and begin again.
 For example:
@@ -89,6 +89,8 @@ A given Kubernetes server will only preserve a historical list of changes for a
 ### Watch bookmarks
 {{< feature-state for_k8s_version="v1.16" state="beta" >}}
 To mitigate the impact of short history window, we introduced a concept of `bookmark` watch event. It is a special kind of event to pass an information that all changes up to a given `resourceVersion` client is requesting has already been send. Object returned in that event is of the type requested by the request, but only `resourceVersion` field is set, e.g.:
        GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245&allowWatchBookmarks=true
@@ -106,7 +108,7 @@ To mitigate the impact of short history window, we introduced a concept of `book
          "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "12746"} }
        }
-`Bookmark` events can be requested by `allowWatchBookmarks=true` option in watch requests, but clients shouldn't assume bookmarks are returned at any specific interval, nor may they assume the server will send any `bookmark` event.
+`Bookmark` events can be requested by `allowWatchBookmarks=true` option in watch requests, but clients shouldn't assume bookmarks are returned at any specific interval, nor may they assume the server will send any `bookmark` event. Since version 1.16, watch bookmarks feature is enabled by default.
 ## Retrieving large results sets in chunks
@@ -644,64 +646,3 @@ Server Side Apply is a beta feature, so it is enabled by default. To turn this
 you need to include the `--feature-gates ServerSideApply=false` flag when
 starting `kube-apiserver`. If you have multiple `kube-apiserver` replicas, all
 should have the same flag setting.
 ## Resource Versions
 Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server. For example, clients must not assume resource versions are numeric, and may only compare two resource version for equality (i.e. must not compare resource versions for greater-than or less-than relationships).
 ### ResourceVersion in metadata
 Clients find resource versions in resources, including the resources in watch events, and list responses returned from the server:
 [v1.meta/ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#objectmeta-v1-meta) - The `metadata.resourceVersion` of a resource instance identifies the resource version the instance was last modified at.
 [v1.meta/ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#listmeta-v1-meta) - The `metadata.resourceVersion` of a resource collection (i.e. a list response) identifies the resource version at which the list response was constructed.
 ### The ResourceVersion Parameter
 The get, list and watch operations support the `resourceVersion` parameter.
 The exact meaning of this parameter differs depending on the operation and the value of the resource version.
 For get and list, the semantics of resource version are:
 **Get:**
 | resourceVersion unset | resourceVersion="0" | resourceVersion="{non-zero version}" |
 |-----------------------|---------------------|--------------------------------------|
 | Most Recent           | Any                 | Not older than                       |
 **List:**
 | paging    | resourceVersion unset | resourceVersion="0" | resourceVersion="{non-zero version}" |
 |-----------|-----------------------|---------------------|--------------------------------------|
 | no limit  | Most Recent           | Any                 | Not older than                       |
 | limit="n" | Most Recent           | Any                 | Exact                                |
 The meaning of the get and list semantics are:
 - **Most Recent:** Return data at the most recent resource version. The returned data must be consistent (i.e. served from etcd via a quorum read).
 - **Any:** Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served. It is possible for the request to return data at a much older resource version that the client has previously observed, particularly in high availabiliy configurations, due to partitions or stale caches. Clients that cannot tolerate this should not use this semantic.
 - **Not older than:** Return data at least as new as the provided resource version. The newest available resource version is preferred, but any data not older than this resource version may be served.
 - **Exact:** Return data at the exact resource version provided.
 For watch, the semantics of resource version are:
 **Watch:**
 | resourceVersion unset               | resourceVersion="0"        | resourceVersion="{non-zero version}" |
 |-------------------------------------|----------------------------|--------------------------------------|
 | Get State and Start at Most Recent  | Get State and Start at Any | Start at Exact                       |
 The meaning of the watch semantics are:
 - **Get State and Start at Most Recent:** Start a watch at the most recent resource version, which must be consistent (i.e. served from etcd via a quorum read). To establish initial state, the watch begins with synthetic “Added” events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occured after the resource version the watch started at.
 - **Get State and Start at Any:** Warning: Watches initialize this way may return arbitrarily stale data! Please review this semantic before using it, and favor the other semantics where possible. Start a watch at any resource version, the most recent resource version available is preferred, but not required; any starting resource version is allowed. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availabiliy configurations, due to partitions or stale caches. Clients that cannot tolerate this should not start a watch with this semantic. To establish initial state, the watch begins with synthetic “Added” events for all resources instances that exist at the starting resource version. All following watch events are for all changes that occured after the resource version the watch started at.
 - **Start at Exact:** Start a watch at an exact resource version. The watch events are for all changes after the provided resource version. Unlike "Get State and Start at Most Recent" and "Get State and Start at Any", the watch is not started with synthetic "Added" events for the provided resource version. The client is assumed to already have the initial state at the starting resource version since the client provided the resource version.
 ### "410 Gone" responses
 Servers are not required to serve all older resource versions and may return a HTTP `410 (Gone)` status code if a client requests a resourceVersion older than the server has retained. Clients must be able to tolerate `410 (Gone)` responses. See [Efficient detection of changes](#efficient-detection-of-changes) for details on how to handle `410 (Gone)` responses when watching resources.
 For example, the kube-apiserver periodically compacts old resource versions from etcd based on its `--etcd-compaction-interval` setting. Also, the kube-apiserver's watch cache keeps `--watch-cache-sizes` resource versions in each resource cache. It depends on if a request is served from cache on which one of these limits applies, but if a resource version is unavailable in the one that applies, a `410 (Gone)` will be returned by the kube-apiserver.
@@ -64,18 +64,17 @@ is to drain the Node from its workloads, remove it from the cluster and re-join
 ## Docker
 On each of your machines, install Docker.
-Version 19.03.4 is recommended, but 1.13.1, 17.03, 17.06, 17.09, 18.06 and 18.09 are known to work as well.
+Version 18.06.2 is recommended, but 1.11, 1.12, 1.13, 17.03 and 18.09 are known to work as well.
 Keep track of the latest verified Docker version in the Kubernetes release notes.
 Use the following commands to install Docker on your system:
 {{< tabs name="tab-cri-docker-installation" >}}
-{{< tab name="Ubuntu 16.04+" codelang="bash" >}}
+{{< tab name="Ubuntu 16.04" codelang="bash" >}}
 # Install Docker CE
 ## Set up the repository:
 ### Install packages to allow apt to use a repository over HTTPS
-apt-get update && apt-get install \
+apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common
  apt-transport-https ca-certificates curl software-properties-common
 ### Add Docker’s official GPG key
 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
@@ -87,10 +86,7 @@ add-apt-repository \
  stable"
 ## Install Docker CE.
-apt-get update && apt-get install \
+apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
  containerd.io=1.2.10-3 \
  docker-ce=5:19.03.4~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:19.03.4~3-0~ubuntu-$(lsb_release -cs)
 # Setup daemon.
 cat > /etc/docker/daemon.json <<EOF
@@ -118,14 +114,12 @@ systemctl restart docker
 yum install yum-utils device-mapper-persistent-data lvm2
 ### Add Docker repository.
-yum-config-manager --add-repo \
+yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo
 ## Install Docker CE.
-yum update && yum install \
+yum update && yum install docker-ce-18.06.2.ce
  containerd.io-1.2.10 \
  docker-ce-19.03.4 \
  docker-ce-cli-19.03.4
 ## Create /etc/docker directory.
 mkdir /etc/docker
@@ -123,120 +123,25 @@ Users today need to use some combination of taints and node selectors in order t
 Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All Kubernetes nodes today have the following default labels:
-* kubernetes.io/os = [windows|linux]
+* beta.kubernetes.io/os = [windows|linux]
-* kubernetes.io/arch = [amd64|arm64|...]
+* beta.kubernetes.io/arch = [amd64|arm64|...]
-If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector.
+If a Pod specification does not specify a nodeSelector like `"beta.kubernetes.io/os": windows`, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector.
 However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. In those situations, you may be hesitant to make the configuration change to add nodeSelectors. The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only.
-For example:  `--register-with-taints='os=windows:NoSchedule'`
+For example:  `--register-with-taints='os=Win1809:NoSchedule'`
 By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector to choose Windows, and the appropriate matching toleration.
 ```yaml
 nodeSelector:
-    kubernetes.io/os: windows
+    "beta.kubernetes.io/os": windows
    node.kubernetes.io/windows-build: '10.0.17763'
 tolerations:
    - key: "os"
      operator: "Equal"
-      value: "windows"
+      value: "Win1809"
      effect: "NoSchedule"
 ```
 ### Handling multiple Windows versions in the same cluster
 The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows
 Server versions in the same cluster, then you should set additional node labels and nodeSelectors.
 Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this. If you're running an older version, then it's recommended to add this label manually to Windows nodes.
 This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used today for each Windows Server version.
 | Product Name                         |   Build Number(s)      |
 |--------------------------------------|------------------------|
 | Windows Server 2019                  | 10.0.17763             |
 | Windows Server version 1809          | 10.0.17763             |
 | Windows Server version 1903          | 10.0.18362             |
 ### Simplifying with RuntimeClass
 [RuntimeClass] can be used to simplify the process of using taints and tolerations. A cluster administrator can create a `RuntimeClass` object which is used to encapsulate these taints and tolerations.
 1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector` for the Windows OS, architecture, and version.
 ```yaml
 apiVersion: node.k8s.io/v1beta1
 kind: RuntimeClass
 metadata:
  name: windows-2019
 handler: 'docker'
 scheduling:
  nodeSelector:
    kubernetes.io/os: 'windows'
    kubernetes.io/arch: 'amd64'
    node.kubernetes.io/windows-build: '10.0.17763'
  tolerations:
  - effect: NoSchedule
    key: os
    operator: Equal
    value: "windows"
 ```
 1. Run `kubectl create -f runtimeClasses.yml` using as a cluster administrator
 1. Add `runtimeClassName: windows-2019` as appropriate to Pod specs
 For example:
 ```yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: iis-2019
  labels:
    app: iis-2019
 spec:
  replicas: 1
  template:
    metadata:
      name: iis-2019
      labels:
        app: iis-2019
    spec:
      runtimeClassName: windows-2019
      containers:
      - name: iis
        image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
        resources:
          limits:
            cpu: 1
            memory: 800Mi
          requests:
            cpu: .1
            memory: 300Mi
        ports:
          - containerPort: 80
 selector:
    matchLabels:
      app: iis-2019
 ---
 apiVersion: v1
 kind: Service
 metadata:
  name: iis
 spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
  selector:
    app: iis-2019
 ```
 {{% /capture %}}
 [RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/
@@ -154,8 +154,6 @@ spec:
 ## Garbage Collecting Load Balancers
 {{< feature-state for_k8s_version="v1.17" state="stable" >}}
 In usual case, the correlating load balancer resources in cloud provider should
 be cleaned up soon after a LoadBalancer type Service is deleted. But it is known
 that there are various corner cases where cloud resources are orphaned after the
@@ -169,6 +167,10 @@ The finalizer will only be removed after the load balancer resource is cleaned u
 This prevents dangling load balancer resources even in corner cases such as the
 service controller crashing.
 This feature is beta and enabled by default since Kubernetes v1.16. You can also
 enable it in v1.15 (alpha) via the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
 `ServiceLoadBalancerFinalizer`.
 ## External Load Balancer Providers
 It is important to note that the datapath for this functionality is provided by a load balancer external to the Kubernetes cluster.
@@ -628,7 +628,7 @@ Additionally, the following restrictions are applied to the schema:
 These fields can only be set with specific features enabled:
- `default`: can be set for `apiextensions.k8s.io/v1` CustomResourceDefinitions. Defaulting is in GA since 1.17 (beta since 1.16 with the `CustomResourceDefaulting` feature gate to be enabled, which is the case automatically for many clusters for beta features). Compare [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting).
+- `default`: can be set for `apiextensions.k8s.io/v1` CustomResourceDefinitions. Defaulting is in beta since 1.16 and requires the `CustomResourceDefaulting` feature gate to be enabled (which is the case automatically for many clusters for beta features). Compare [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting).
 Note: compare with [structural schemas](#specifying-a-structural-schema) for further restriction required for certain CustomResourceDefinition features.
@@ -781,10 +781,10 @@ crontab "my-new-cron-object" created
 ### Defaulting
-{{< feature-state state="stable" for_kubernetes_version="1.17" >}}
+{{< feature-state state="beta" for_kubernetes_version="1.16" >}}
 {{< note >}}
-To use defaulting, your CustomResourceDefinition must use API version `apiextensions.k8s.io/v1`.
+Defaulting is available as beta since 1.16 in `apiextensions.k8s.io/v1` CustomResourceDefinitions, and hence enabled by default for most clusters (feature gate `CustomResourceDefaulting`, refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation).
 {{< /note >}}
 Defaulting allows to specify default values in the [OpenAPI v3 validation schema](#validation):
@@ -85,10 +85,7 @@ state file `cpu_manager_state` in the kubelet root directory.
 This policy manages a shared pool of CPUs that initially contains all CPUs in the
 node. The amount of exclusively allocatable CPUs is equal to the total
 number of CPUs in the node minus any CPU reservations by the kubelet `--kube-reserved` or
-`--system-reserved` options. From 1.17, the CPU reservation list can be specified
+`--system-reserved` options. CPUs reserved by these options are taken, in
 explictly by kubelet `--reserved-cpus` option. The explicit CPU list specified by
 `--reserved-cpus` takes precedence over the CPU reservation specified by
 `--kube-reserved` and `--system-reserved`. CPUs reserved by these options are taken, in
 integer quantity, from the initial shared pool in ascending order by physical
 core ID.  This shared pool is the set of CPUs on which any containers in
 `BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional
@@ -98,8 +95,8 @@ exclusive CPUs.
 {{< note >}}
 The kubelet requires a CPU reservation greater than zero be made
-using either `--kube-reserved` and/or `--system-reserved` or `--reserved-cpus` when
+using either `--kube-reserved` and/or `--system-reserved` when the static
-the static policy is enabled. This is because zero CPU reservation would allow the shared
+policy is enabled. This is because zero CPU reservation would allow the shared
 pool to become empty.
 {{< /note >}}
@@ -27,7 +27,7 @@ resources instead of a single large Endpoints resource.
 ## Enabling Endpoint Slices
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
 {{< note >}}
 Although Endpoint Slices may eventually replace Endpoints, many Kubernetes
@@ -36,7 +36,19 @@ seen as an addition to Endpoints in a cluster, not a replacement for them.
 {{< /note >}}
 As an alpha feature, Endpoint Slices are not enabled by default in Kubernetes.
-To enable them, the EndpointSlice feature gate will need to be enabled
+Enabling Endpoint Slices requires as many as 3 changes to Kubernetes cluster
 configuration.
 To enable the Discovery API group that includes Endpoint Slices, use the runtime
 config flag (`--runtime-config=discovery.k8s.io/v1alpha1=true`).
 The logic responsible for watching services, pods, and nodes and creating or
 updating associated Endpoint Slices lives within the EndpointSlice controller.
 This is disabled by default but can be enabled with the controllers flag on
 kube-controller-manager (`--controllers=endpointslice`).
 For Kubernetes components like kube-proxy to actually start using Endpoint
 Slices, the EndpointSlice feature gate will need to be enabled
 (`--feature-gates=EndpointSlice=true`).
 ## Using Endpoint Slices
@@ -45,4 +57,4 @@ With Endpoint Slices fully enabled in your cluster, you should see corresponding
 EndpointSlice resources for each Endpoints resource. In addition to supporting
 existing Endpoints functionality, Endpoint Slices should include new bits of
 information such as topology. They will allow for greater scalability and
-extensibility of network endpoints in your cluster.
+extensibility of network endpoints in your cluster.
@@ -8,14 +8,14 @@ content_template: templates/task
 {{% capture overview %}}
 This page explains how to upgrade a Kubernetes cluster created with kubeadm from version
-1.16.x to version 1.17.x, and from version 1.17.x to 1.17.y (where `y > x`).
+1.15.x to version 1.16.x, and from version 1.16.x to 1.16.y (where `y > x`).
 To see information about upgrading clusters created using older versions of kubeadm,
 please refer to following pages instead:
 - [Upgrading kubeadm cluster from 1.15 to 1.16](https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)
 - [Upgrading kubeadm cluster from 1.14 to 1.15](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/)
 - [Upgrading kubeadm cluster from 1.13 to 1.14](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-14/)
 - [Upgrading kubeadm cluster from 1.12 to 1.13](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/) or [Upgrading kubeadm HA clusters from v1.12 to v1.13](https://v1-15.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13/)
 The upgrade workflow at high level is the following:
@@ -27,7 +27,7 @@ The upgrade workflow at high level is the following:
 {{% capture prerequisites %}}
- You need to have a kubeadm Kubernetes cluster running version 1.16.0 or later.
+- You need to have a kubeadm Kubernetes cluster running version 1.15.0 or later.
 - [Swap must be disabled](https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux).
 - The cluster should use a static control plane and etcd pods or external etcd.
 - Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md) carefully.
@@ -47,19 +47,19 @@ The upgrade workflow at high level is the following:
 ## Determine which version to upgrade to
-1.  Find the latest stable 1.17 version:
+1.  Find the latest stable 1.16 version:
    {{< tabs name="k8s_install_versions" >}}
    {{% tab name="Ubuntu, Debian or HypriotOS" %}}
    apt update
    apt-cache policy kubeadm
-    # find the latest 1.17 version in the list
+    # find the latest 1.16 version in the list
-    # it should look like 1.17.x-00, where x is the latest patch
+    # it should look like 1.16.x-00, where x is the latest patch
    {{% /tab %}}
    {{% tab name="CentOS, RHEL or Fedora" %}}
    yum list --showduplicates kubeadm --disableexcludes=kubernetes
-    # find the latest 1.17 version in the list
+    # find the latest 1.16 version in the list
-    # it should look like 1.17.x-0, where x is the latest patch
+    # it should look like 1.16.x-0, where x is the latest patch
    {{% /tab %}}
    {{< /tabs >}}
@@ -71,14 +71,14 @@ The upgrade workflow at high level is the following:
    {{< tabs name="k8s_install_kubeadm_first_cp" >}}
    {{% tab name="Ubuntu, Debian or HypriotOS" %}}
-    # replace x in 1.17.x-00 with the latest patch version
+    # replace x in 1.16.x-00 with the latest patch version
    apt-mark unhold kubeadm && \
-    apt-get update && apt-get install -y kubeadm=1.17.x-00 && \
+    apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
    apt-mark hold kubeadm
    {{% /tab %}}
    {{% tab name="CentOS, RHEL or Fedora" %}}
-    # replace x in 1.17.x-0 with the latest patch version
+    # replace x in 1.16.x-0 with the latest patch version
-    yum install -y kubeadm-1.17.x-0 --disableexcludes=kubernetes
+    yum install -y kubeadm-1.16.x-0 --disableexcludes=kubernetes
    {{% /tab %}}
    {{< /tabs >}}
@@ -91,7 +91,7 @@ The upgrade workflow at high level is the following:
 1.  Drain the control plane node:
    ```shell
-    kubectl drain $CP_NODE --ignore-daemonsets
+    kubectl drain $MASTER --ignore-daemonsets
    ```
 1.  On the control plane node, run:
@@ -109,26 +109,26 @@ The upgrade workflow at high level is the following:
    [preflight] Running pre-flight checks.
    [upgrade] Making sure the cluster is healthy:
    [upgrade] Fetching available versions to upgrade to
-    [upgrade/versions] Cluster version: v1.16.0
+    [upgrade/versions] Cluster version: v1.15.2
-    [upgrade/versions] kubeadm version: v1.17.0
+    [upgrade/versions] kubeadm version: v1.16.0
    Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
    COMPONENT   CURRENT       AVAILABLE
-    Kubelet     1 x v1.16.0   v1.17.0
+    Kubelet     1 x v1.15.2   v1.16.0
    Upgrade to the latest version in the v1.16 series:
    COMPONENT            CURRENT   AVAILABLE
-    API Server           v1.16.0   v1.17.0
+    API Server           v1.15.2   v1.16.0
-    Controller Manager   v1.16.0   v1.17.0
+    Controller Manager   v1.15.2   v1.16.0
-    Scheduler            v1.16.0   v1.17.0
+    Scheduler            v1.15.2   v1.16.0
-    Kube Proxy           v1.16.0   v1.17.0
+    Kube Proxy           v1.15.2   v1.16.0
-    CoreDNS              1.6.2     1.6.5
+    CoreDNS              1.3.1     1.6.2
-    Etcd                 3.3.15    3.4.3-0
+    Etcd                 3.3.10    3.3.15
    You can now apply the upgrade by executing the following command:
-        kubeadm upgrade apply v1.17.0
+        kubeadm upgrade apply v1.16.0
    _____________________________________________________________________
    ```
@@ -144,7 +144,7 @@ The upgrade workflow at high level is the following:
 1.  Choose a version to upgrade to, and run the appropriate command. For example:
    ```shell
-    sudo kubeadm upgrade apply v1.17.x
+    sudo kubeadm upgrade apply v1.16.x
    ```
    - Replace `x` with the patch version you picked for this upgrade.
@@ -157,9 +157,9 @@ The upgrade workflow at high level is the following:
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
-    [upgrade/version] You have chosen to change the cluster version to "v1.17.0"
+    [upgrade/version] You have chosen to change the cluster version to "v1.16.0"
-    [upgrade/versions] Cluster version: v1.16.0
+    [upgrade/versions] Cluster version: v1.15.2
-    [upgrade/versions] kubeadm version: v1.17.0
+    [upgrade/versions] kubeadm version: v1.16.0
    [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
    [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
    [upgrade/prepull] Prepulling image for component etcd.
@@ -177,7 +177,7 @@ The upgrade workflow at high level is the following:
    [upgrade/prepull] Prepulled image for component kube-apiserver.
    [upgrade/prepull] Prepulled image for component kube-scheduler.
    [upgrade/prepull] Successfully prepulled the images for all the control plane components
-    [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.17.0"...
+    [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.16.0"...
    Static pod: kube-apiserver-luboitvbox hash: 8d931c2296a38951e95684cbcbe3b923
    Static pod: kube-controller-manager-luboitvbox hash: 2480bf6982ad2103c05f6764e20f2787
    Static pod: kube-scheduler-luboitvbox hash: 9b290132363a92652555896288ca3f88
@@ -215,8 +215,8 @@ The upgrade workflow at high level is the following:
    [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
    [upgrade/staticpods] Renewing certificate embedded in "admin.conf"
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
-    [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
+    [kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
-    [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
+    [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
@@ -224,7 +224,7 @@ The upgrade workflow at high level is the following:
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
-    [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.17.0". Enjoy!
+    [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.0". Enjoy!
    [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
    ```
@@ -240,7 +240,7 @@ The upgrade workflow at high level is the following:
 1.  Uncordon the control plane node
    ```shell
-    kubectl uncordon $CP_NODE
+    kubectl uncordon $MASTER
    ```
 ### Upgrade additional control plane nodes
@@ -265,14 +265,14 @@ Also `sudo kubeadm upgrade plan` is not needed.
    {{< tabs name="k8s_install_kubelet" >}}
    {{% tab name="Ubuntu, Debian or HypriotOS" %}}
-    # replace x in 1.17.x-00 with the latest patch version
+    # replace x in 1.16.x-00 with the latest patch version
    apt-mark unhold kubelet kubectl && \
-    apt-get update && apt-get install -y kubelet=1.17.x-00 kubectl=1.17.x-00 && \
+    apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
    apt-mark hold kubelet kubectl
    {{% /tab %}}
    {{% tab name="CentOS, RHEL or Fedora" %}}
-    # replace x in 1.17.x-0 with the latest patch version
+    # replace x in 1.16.x-0 with the latest patch version
-    yum install -y kubelet-1.17.x-0 kubectl-1.17.x-0 --disableexcludes=kubernetes
+    yum install -y kubelet-1.16.x-0 kubectl-1.16.x-0 --disableexcludes=kubernetes
    {{% /tab %}}
    {{< /tabs >}}
@@ -293,14 +293,14 @@ without compromising the minimum required capacity for running your workloads.
    {{< tabs name="k8s_install_kubeadm_worker_nodes" >}}
    {{% tab name="Ubuntu, Debian or HypriotOS" %}}
-    # replace x in 1.17.x-00 with the latest patch version
+    # replace x in 1.16.x-00 with the latest patch version
    apt-mark unhold kubeadm && \
-    apt-get update && apt-get install -y kubeadm=1.17.x-00 && \
+    apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
    apt-mark hold kubeadm
    {{% /tab %}}
    {{% tab name="CentOS, RHEL or Fedora" %}}
-    # replace x in 1.17.x-0 with the latest patch version
+    # replace x in 1.16.x-0 with the latest patch version
-    yum install -y kubeadm-1.17.x-0 --disableexcludes=kubernetes
+    yum install -y kubeadm-1.16.x-0 --disableexcludes=kubernetes
    {{% /tab %}}
    {{< /tabs >}}
@@ -334,14 +334,14 @@ without compromising the minimum required capacity for running your workloads.
    {{< tabs name="k8s_kubelet_and_kubectl" >}}
    {{% tab name="Ubuntu, Debian or HypriotOS" %}}
-    # replace x in 1.17.x-00 with the latest patch version
+    # replace x in 1.16.x-00 with the latest patch version
    apt-mark unhold kubelet kubectl && \
-    apt-get update && apt-get install -y kubelet=1.17.x-00 kubectl=1.17.x-00 && \
+    apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
    apt-mark hold kubelet kubectl
    {{% /tab %}}
    {{% tab name="CentOS, RHEL or Fedora" %}}
-    # replace x in 1.17.x-0 with the latest patch version
+    # replace x in 1.16.x-0 with the latest patch version
-    yum install -y kubelet-1.17.x-0 kubectl-1.17.x-0 --disableexcludes=kubernetes
+    yum install -y kubelet-1.16.x-0 kubectl-1.16.x-0 --disableexcludes=kubernetes
    {{% /tab %}}
    {{< /tabs >}}
@@ -146,28 +146,6 @@ control group (`system.slice` on systemd machines for example).
 Note that Kubelet **does not** create `--system-reserved-cgroup` if it doesn't
 exist. Kubelet will fail if an invalid cgroup is specified.
 ### Explictly Reserved CPU List
 {{< feature-state for_k8s_version="v1.17" state="stable" >}}
 - **Kubelet Flag**: `--reserved-cpus=0-3`
 `reserved-cpus` is meant to define an explict CPU set for OS system daemons and
 kubernetes system daemons. This option is added in 1.17 release. `reserved-cpus`
 is for systems that do not intent to define seperate top level cgroups for
 OS system daemons and kubernetes system daemons with regard to cpuset resource.
 If the Kubelet **does not** have `--system-reserved-cgroup` and `--kube-reserved-cgroup`,
 the explicit cpuset provided by `reserved-cpus` will take precedence over the CPUs
 defined by `--kube-reserved` and `--system-reserved` options.
 This option is specifically designed for Telco/NFV use cases where uncontrolled
 interrupts/timers may impact the workload performance. you can use this option
 to define the explict cpuset for the system/kubernetes daemons as well as the
 interrupts/timers, so the rest CPUs on the system can be used exclusively for
 workloads, with less impact from uncontrolled interrupts/timers. To move the
 system daemon, kubernetes daemons and interrupts/timers to the explict cpuset
 defined by this option, other mechanism outside Kubernetes should be used.
 For example: in Centos, you can do this using the tuned toolset.
 ### Eviction Thresholds
 - **Kubelet Flag**: `--eviction-hard=[memory.available<500Mi]`
@@ -281,8 +259,4 @@ for `kube-reserved` and `system-reserved`.
 As of Kubernetes version 1.8, the `storage` key name was changed to `ephemeral-storage`
 for the alpha release.
 As of Kubernetes version 1.17, you can optionally specify
 explicit cpuset by `reserved-cpus` as CPUs reserved for OS system
 daemons/interrupts/timers and Kubernetes daemons.
 {{% /capture %}}
@@ -1,6 +1,5 @@
 ---
 title: Control Topology Management Policies on a node
 reviewers:
 - ConnorDoyle
 - klueska
@@ -49,9 +48,9 @@ The hint is then stored in the Topology Manager for use by the *Hint Providers*
 The Topology Manager currently:
 - Works on Nodes with the `static` CPU Manager Policy enabled. See [control CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/)
- - Works on Pods making CPU requests or Device requests via extended resources
+ - Works on Pods in the `Guaranteed` {{< glossary_tooltip text="QoS class" term_id="qos-class" >}}
-If these conditions are met, Topology Manager will align the requested resources.
+If these conditions are met, Topology Manager will align CPU and device requests.
 Topology Manager supports four allocation policies. You can set a policy via a Kubelet flag, `--topology-manager-policy`.
 There are four supported policies:
@@ -84,9 +83,6 @@ Using this information, the Topology Manager stores the
 preferred NUMA Node affinity for that container. If the affinity is not preferred, 
 Topology Manager will reject this pod from the node. This will result in a pod in a `Terminated` state with a pod admission failure.
 Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeploy of the pod.
 An external control loop could be also implemented to trigger a redeployment of pods that have the `Topology Affinity` error.
 If the pod is admitted, the *Hint Providers* can then use this information when making the 
 resource allocation decision.
@@ -99,8 +95,6 @@ If it is, Topology Manager will store this and the *Hint Providers* can then use
 resource allocation decision.
 If, however, this is not possible then the Topology Manager will reject the pod from the node. This will result in a pod in a `Terminated` state with a pod admission failure.
 Once the pod is in a `Terminated` state, the Kubernetes scheduler will **not** attempt to reschedule the pod. It is recommended a Deployment with Replicas to trigger a redeploy of the pod.
 An external control loop could be also implemented to trigger a redeployment of pods that have the `Topology Affinity` error.
 ### Pod Interactions with Topology Manager Policies
@@ -152,36 +146,9 @@ spec:
 This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
 Topology Manager would consider this Pod. The Topology Manager consults the CPU Manager `static` policy, which returns the topology of available CPUs. 
 Topology Manager also consults Device Manager to discover the topology of available devices for example.com/device.
-```yaml
+Topology Manager will use this information to store the best Topology for this container. In the case of this Pod, CPU and Device Manager will use this stored information at the resource allocation stage.
 spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        example.com/deviceA: "1"
        example.com/deviceB: "1"
      requests:
        example.com/deviceA: "1"
        example.com/deviceB: "1"
 ```
 This pod runs in the `BestEffort` QoS class because there are no CPU and memory requests.
 The Topology Manager would consider both of the above pods. The Topology Manager would consult the Hint Providers, which are CPU and Device Manager to get topology hints for the pods. 
 In the case of the `Guaranteed` pod the `static` CPU Manager policy would return hints relating to the CPU request and the Device Manager would send back hints for the requested device.
 In the case of the `BestEffort` pod the CPU Manager would send back the default hint as there is no CPU request and the Device Manager would send back the hints for each of the requested devices.
 Using this information the Topology Manager calculates the optimal hint for the pod and stores this information, which will be used by the Hint Providers when they are making their resource assignments. 
 ### Known Limitations
 1. As of K8s 1.16 the Topology Manager is currently only guaranteed to work if a *single* container in the pod spec requires aligned resources. This is due to the hint generation being based on current resource allocations, and all containers in a pod generate hints before any resource allocation has been made. This results in unreliable hints for all but the first container in a pod.
 *Due to this limitation if multiple pods/containers are considered by Kubelet in quick succession they may not respect the Topology Manager policy.
 2. The maximum number of NUMA nodes that Topology Manager will allow is 8, past this there will be a state explosion when trying to enumerate the possible NUMA affinities and generating their hints.
 3. The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail on the node due to the Topology Manager. 
 {{% /capture %}}
@@ -6,12 +6,12 @@ weight: 20
 {{% capture overview %}}
-{{< feature-state for_k8s_version="v1.17" state="beta" >}}
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
 This page shows how to enable and use the `RunAsUserName` feature for pods and containers that will run on Windows nodes. This feature is meant to be the Windows equivalent of the Linux-specific `runAsUser` feature, allowing users to run the container entrypoints with a different username that their default ones.
 {{< note >}}
-This feature is in beta. The overall functionality for `RunAsUserName` will not change, but there may be some changes regarding the username validation.
+Currently this feature is in alpha state. The overall functionality of the feature will not change, but there may be some changes regarding the username validation. Please take this into consideration when testing or adopting this feature.
 {{< /note >}}
 {{% /capture %}}
@@ -20,6 +20,16 @@ This feature is in beta. The overall functionality for `RunAsUserName` will not
 You need to have a Kubernetes cluster and the kubectl command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes where pods with containers running Windows workloads will get scheduled.
 ### Enable the WindowsRunAsUserName feature gate
 In the alpha state, the `WindowsRunAsUserName` feature gate needs to be enabled on the `kube-apiserver` service. Without it, the `runAsUserName` field will be dropped from the pod's, container's, and init container's SecurityContexts. See [Feature Gates](/docs/reference/command-line-tools-reference/feature-gates/) for an explanation of enabling feature gates. Please make sure `feature-gates=WindowsRunAsUserName=true` parameter exists in the `kube-apiserver` command line.
 {{% /capture %}}
 {{% capture steps %}}
 ## Set the Username for a Pod
 To specify the username with which to execute the Pod's container processes, include the `securityContext` field ([PodSecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritycontext-v1-core) in the Pod specification, and within it, the `windowsOptions` ([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core) field containing the `runAsUserName` field.
@@ -11,7 +11,7 @@ weight: 160
 {{% capture overview %}}
-{{< feature-state state="stable" for_k8s_version="v1.17" >}}
+{{< feature-state state="beta" >}}
 This page shows how to configure process namespace sharing for a pod. When
 process namespace sharing is enabled, processes in a container are visible
@@ -27,6 +27,9 @@ include debugging utilities like a shell.
 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
 Process Namespace Sharing is a **beta** feature that is enabled by default. It
 may be disabled by setting `--feature-gates=PodShareProcessNamespace=false`.
 {{% /capture %}}
 {{% capture steps %}}
@@ -58,41 +58,6 @@ kubectl get pods pod01 -o go-template --template='{{range .status.podIPs}}{{prin
 a00:100::4
 ```
 You can also validate Pod IPs using the Downward API via the `status.podIPs` fieldPath. The following snippet demonstrates how you can expose the Pod IPs via an environment variable called `MY_POD_IPS` within a container.
 ```
        env:
        - name: MY_POD_IPS
          valueFrom:
            fieldRef:
              fieldPath: status.podIPs
 ```
 The following command prints the value of the `MY_POD_IPS` environment variable from within a container. The value is a comma separated list that corresponds to the Pod's IPv4 and IPv6 addresses.
 ```shell
 kubectl exec -it pod01 -- set | grep MY_POD_IPS
 ```
 ```
 MY_POD_IPS=10.244.1.4,a00:100::4
 ```
 The Pod's IP addresses will also be written to `/etc/hosts` within a container. The following command executes a cat on `/etc/hosts` on a dual stack Pod. From the output you can verify both the IPv4 and IPv6 IP address for the Pod.
 ```shell
 kubectl exec -it pod01 -- cat /etc/hosts
 ```
 ```
 # Kubernetes-managed hosts file.
 127.0.0.1    localhost
 ::1    localhost ip6-localhost ip6-loopback
 fe00::0    ip6-localnet
 fe00::0    ip6-mcastprefix
 fe00::1    ip6-allnodes
 fe00::2    ip6-allrouters
 10.244.1.4    pod01
 a00:100::4    pod01
 ```
 ## Validate Services
 Create the following Service without the `ipFamily` field set. When this field is not set, the Service gets an IP from the first configured range via `--service-cluster-ip-range` flag on the kube-controller-manager.
@@ -158,3 +123,4 @@ my-service   ClusterIP   fe80:20d::d06b   2001:db8:f100:4002::9d37:c0d7   80:318
 {{% /capture %}}
@@ -569,6 +569,8 @@ func TestExampleObjectSchemas(t *testing.T) {
 	capabilities.SetForTests(capabilities.Capabilities{
 		AllowPrivileged: true,
 	})
 	// PodShareProcessNamespace needed for example share-process-namespace.yaml
 	utilfeature.DefaultFeatureGate.Set("PodShareProcessNamespace=true")
 	for dir, expected := range cases {
 		tested := 0
@@ -14,4 +14,4 @@ spec:
        windowsOptions:
            runAsUserName: "ContainerAdministrator"
  nodeSelector:
-    kubernetes.io/os: windows
+    beta.kubernetes.io/os: windows
@@ -11,4 +11,4 @@ spec:
    image: mcr.microsoft.com/windows/servercore:ltsc2019
    command: ["ping", "-t", "localhost"]
  nodeSelector:
-    kubernetes.io/os: windows
+    beta.kubernetes.io/os: windows
@@ -543,6 +543,8 @@ func TestExampleObjectSchemas(t *testing.T) {
 	capabilities.SetForTests(capabilities.Capabilities{
 		AllowPrivileged: true,
 	})
 	// PodShareProcessNamespace needed for example share-process-namespace.yaml
 	utilfeature.DefaultFeatureGate.Set("PodShareProcessNamespace=true")
 	for dir, expected := range cases {
 		tested := 0
@@ -1489,7 +1489,7 @@ GitHub project has [instructions](https://github.com/quobyte/quobyte-csi#quobyte
 Quobyte 支持{{< glossary_tooltip text="容器存储接口" term_id="csi" >}}。
 推荐使用 CSI 插件以在 Kubernetes 中使用 Quobyte 卷。
-Quobyte 的 GitHub 项目具有[说明](https://github.com/quobyte/quobyte-csi#quobyte-csi)以及使用示例来部署 CSI 的 Quobyte。
+Quobyte 的 GitHub 项目具有[说明(https://github.com/quobyte/quobyte/quobyte-csi#quobyte-csi)以及使用示例来部署 CSI 的 Quobyte。
 ### rbd {#rbd}
@@ -31,14 +31,14 @@ This page assumes you have a working Juju deployed cluster.
 <!--
 ## Implementation
-The TLS and easyrsa implementations use the following [layers](https://old-docs.jujucharms.com/2.5/en/developer-layers).
+The TLS and easyrsa implementations use the following [layers](https://jujucharms.com/docs/2.2/developer-layers).
 [layer-tls-client](https://github.com/juju-solutions/layer-tls-client)
 [layer-easyrsa](https://github.com/juju-solutions/layer-easyrsa)
 -->
 ## 实现
-TLS 和 easyrsa 的实现使用以下 [layers](https://old-docs.jujucharms.com/2.5/en/developer-layers)。
+TLS 和 easyrsa 的实现使用以下 [layers](https://jujucharms.com/docs/2.2/developer-layers)。
 [layer-tls-client](https://github.com/juju-solutions/layer-tls-client)
 [layer-easyrsa](https://github.com/juju-solutions/layer-easyrsa)
@@ -52,7 +52,7 @@ By default the administrator can ssh to any deployed node in a cluster. You can
 Note: The Juju controller node will still have open ssh access in your cloud, and will be used as a jump host in this case.
-Refer to the [model management](https://old-docs.jujucharms.com/2.5/en/models) page in the Juju documentation for instructions on how to manage ssh keys.
+Refer to the [model management](https://jujucharms.com/docs/2.2/models) page in the Juju documentation for instructions on how to manage ssh keys.
 -->
 ## 限制 ssh 访问
@@ -62,7 +62,7 @@ Refer to the [model management](https://old-docs.jujucharms.com/2.5/en/models) p
 注意：Juju 控制器节点在您的云中仍然有开放的 ssh 访问权限，并且在这种情况下将被用作跳板机。
-有关如何管理 ssh 密钥的说明，请参阅 Juju 文档中的 [模型管理](https://old-docs.jujucharms.com/2.5/en/models) 页面。
+有关如何管理 ssh 密钥的说明，请参阅 Juju 文档中的 [模型管理](https://jujucharms.com/docs/2.2/models) 页面。
 {{% /capture %}}
@@ -62,13 +62,13 @@ policies using an example application.
 ## Deploying Cilium for Production Use
 For detailed instructions around deploying Cilium for production, see:
-[Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/gettingstarted/#installation)
+[Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
 This documentation includes detailed requirements, instructions and example
 production DaemonSet files.
 -->
 ## 部署 Cilium 用于生产用途
-关于部署 Cilium 用于生产的详细说明，请见[Cilium Kubernetes 安装指南](https://cilium.readthedocs.io/en/latest/gettingstarted/#installation)
+关于部署 Cilium 用于生产的详细说明，请见[Cilium Kubernetes 安装指南](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
 ，此文档包括详细的需求、说明和生产用途 DaemonSet 文件示例。
 {{% /capture %}}
@@ -103,7 +103,7 @@ There are two main components to be aware of:
 on the traffic to/from Pods on that node using Linux BPF.
 - For production deployments, Cilium should leverage the key-value store cluster
 (e.g., etcd) used by Kubernetes, which typically runs on the Kubernetes master nodes.
-The [Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/gettingstarted/#installation)
+The [Cilium Kubernetes Installation Guide](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
 includes an example DaemonSet which can be customized to point to this key-value
 store cluster. The simple ''all-in-one'' DaemonSet for minikube requires no such
 configuration because it automatically connects to the minikube's etcd instance.
@@ -112,7 +112,7 @@ configuration because it automatically connects to the minikube's etcd instance.
 - 在集群中的每个节点上都会运行一个 `cilium` Pod，并利用Linux BPF执行网络策略管理该节点上进出 Pod 的流量。
 - 对于生产部署，Cilium 应该复用 Kubernetes 所使用的键值存储集群（如 etcd），其通常在Kubernetes 的 master 节点上运行。
-[Cilium Kubernetes安装指南](https://cilium.readthedocs.io/en/latest/gettingstarted/#installation)
+[Cilium Kubernetes安装指南](https://cilium.readthedocs.io/en/latest/kubernetes/install/)
 包括了一个示例 DaemonSet，可以自定义指定此键值存储集群。
 简单的 minikube 的“一体化” DaemonSet 不需要这样的配置，因为它会自动连接到 minikube 的 etcd 实例。
@@ -35,7 +35,7 @@ content_template: templates/task
 * [Romana 网络策略](https://github.com/romana/romana/wiki/Romana-policies)
-    * [Romana 网络策略示例](https://github.com/romana/core/blob/master/doc/policy.md)
+    * [Romana 网络策略示例](https://github.com/romana/core/tree/master/policy)
 * NetworkPolicy API
@@ -604,7 +604,7 @@ Kubernetes 可能会在创建新的日志文件时删除旧的日志文件; 您
 [gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh#L735
 [kubeconfig]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/
 [fluentd]: http://www.fluentd.org/
-[fluentd_install_doc]: https://docs.fluentd.org/v/0.12/articles/quickstart#step1-installing-fluentd
+[fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
 [logstash]: https://www.elastic.co/products/logstash
 [logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
 [kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation
@@ -2016,6 +2016,12 @@ inspect them.</p>
 <td>If the requested object does not exist the command will return exit code 0. </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -2664,7 +2670,7 @@ inspect them.</p>
 </code></pre>
 <p>Delete resources by filenames, stdin, resources and names, or by resources and label selector.</p>
 <p> JSON and YAML formats are accepted. Only one type of the arguments may be specified: filenames, resources and names, or resources and label selector.</p>
-<p> Some resources, such as pods, support graceful deletion. These resources define a default period before they are forcibly terminated (the grace period) but you may override that value with the --grace-period flag, or pass --now to set a grace-period of 1. Because these resources often represent entities in the cluster, deletion may not be acknowledged immediately. If the node hosting a pod is down or cannot reach the API server, termination may take significantly longer than the grace period. To force delete a resource, you must pass a grace period of 0 and specify the --force flag. Note: only a subset of resources support graceful deletion. In absence of the support, --grace-period is ignored.</p>
+<p> Some resources, such as pods, support graceful deletion. These resources define a default period before they are forcibly terminated (the grace period) but you may override that value with the --grace-period flag, or pass --now to set a grace-period of 1. Because these resources often represent entities in the cluster, deletion may not be acknowledged immediately. If the node hosting a pod is down or cannot reach the API server, termination may take significantly longer than the grace period. To force delete a resource, you must pass a grace period of 0 and specify the --force flag.</p>
 <p> IMPORTANT: Force deleting pods does not wait for confirmation that the pod&#39;s processes have been terminated, which can leave those processes running until the node detects the deletion and completes graceful deletion. If your processes use shared storage or talk to a remote API and depend on the name of the pod to identify themselves, force deleting those pods may result in multiple processes running on different machines using the same identification which may lead to data corruption or inconsistency. Only force delete pods when you are sure the pod is terminated, or if your application can tolerate multiple copies of the same pod running at once. Also, if you force delete pods the scheduler may place new pods on those nodes before the node has released those resources and causing those pods to be evicted immediately.</p>
 <p> Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the rest of the resource.</p>
 <h3 id="usage">Usage</h3>
@@ -2729,6 +2735,12 @@ inspect them.</p>
 <td>Treat &quot;resource not found&quot; as a successful delete. Defaults to &quot;true&quot; when --all is specified. </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -2879,6 +2891,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion). </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -3013,6 +3031,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files to use to edit the resource </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -3264,6 +3288,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files identifying the resource to update the annotation </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -3535,7 +3565,7 @@ viewing your workloads in a Kubernetes cluster.</p>
 </code></pre>
 <p>Diff configurations specified by filename or stdin between the current online configuration, and the configuration as it would be if applied.</p>
 <p> Output is always YAML.</p>
-<p> KUBECTL_EXTERNAL_DIFF environment variable can be used to select your own diff command. By default, the &quot;diff&quot; command available in your path will be run with &quot;-u&quot; (unified diff) and &quot;-N&quot; (treat absent files as empty) options.</p>
+<p> KUBECTL_EXTERNAL_DIFF environment variable can be used to select your own diff command. By default, the &quot;diff&quot; command available in your path will be run with &quot;-u&quot; (unicode) and &quot;-N&quot; (treat new files as empty) options.</p>
 <h3 id="usage">Usage</h3>
 <p><code>$ diff -f FILENAME</code></p>
 <h3 id="flags">Flags</h3>
@@ -3641,6 +3671,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files to use to edit the resource </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -3779,6 +3815,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files identifying the resource to update the labels </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -4868,6 +4910,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files identifying the resource to get from a server. </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -4980,6 +5028,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files identifying the resource to get from a server. </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -5173,6 +5227,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Filename, directory, or URL to files identifying the resource to get from a server. </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -5272,6 +5332,12 @@ viewing your workloads in a Kubernetes cluster.</p>
 <td>Groups to bind to the role </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -5782,6 +5848,12 @@ applications.</p>
 <td>Filename, directory, or URL to files containing the resource to describe </td>
 </tr>
 <tr>
 <td>include-uninitialized</td>
 <td></td>
 <td>false</td>
 <td>If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., &quot;--all&quot;. Objects with empty metadata.initializers are regarded as initialized. </td>
 </tr>
 <tr>
 <td>kustomize</td>
 <td>k</td>
 <td></td>
@@ -5922,11 +5994,6 @@ applications.</p>
 <pre class="code-block example"><code class="lang-shell">kubectl logs <span class="hljs-attribute">--since</span>=1h nginx
 </code></pre>
 <blockquote class="code-block example">
 <p> Show logs from a kubelet with an expired serving certificate</p>
 </blockquote>
 <pre class="code-block example"><code class="lang-shell">kubectl logs <span class="hljs-comment">--insecure-skip-tls-verify-backend nginx</span>
 </code></pre>
 <blockquote class="code-block example">
 <p> Return snapshot logs from first container of a job named hello</p>
 </blockquote>
 <pre class="code-block example"><code class="lang-shell">kubectl logs <span class="hljs-keyword">job</span>/hello
@@ -5975,12 +6042,6 @@ applications.</p>
 <td>If watching / following pod logs, allow for any errors that occur to be non-fatal </td>
 </tr>
 <tr>
 <td>insecure-skip-tls-verify-backend</td>
 <td></td>
 <td>false</td>
 <td>Skip verifying the identity of the kubelet that logs are requested from.  In theory, an attacker could provide invalid log content back. You might want to use this if your kubelet serving certificates have expired. </td>
 </tr>
 <tr>
 <td>limit-bytes</td>
 <td></td>
 <td>0</td>
@@ -5999,12 +6060,6 @@ applications.</p>
 <td>The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one pod is running </td>
 </tr>
 <tr>
 <td>prefix</td>
 <td></td>
 <td>false</td>
 <td>Prefix each log line with the log source (pod name and container name) </td>
 </tr>
 <tr>
 <td>previous</td>
 <td>p</td>
 <td>false</td>
@@ -6233,7 +6288,7 @@ applications.</p>
 <h1 id="top">top</h1>
 <p>Display Resource (CPU/Memory/Storage) usage.</p>
 <p> The top command allows you to see the resource consumption for nodes or pods.</p>
-<p> This command requires Metrics Server to be correctly configured and working on the server.</p>
+<p> This command requires Heapster to be correctly configured and working on the server.</p>
 <h3 id="usage">Usage</h3>
 <p><code>$ top</code></p>
 <hr>
@@ -6893,11 +6948,6 @@ applications.</p>
 <pre class="code-block example"><code class="lang-shell"><span class="hljs-attribute">kubectl api-resources -o wide</span>
 </code></pre>
 <blockquote class="code-block example">
 <p> Print the supported API Resources sorted by a column</p>
 </blockquote>
 <pre class="code-block example"><code class="lang-shell">kubectl api-<span class="hljs-built_in">resources</span> --<span class="hljs-built_in">sort</span>-by=<span class="hljs-built_in">name</span>
 </code></pre>
 <blockquote class="code-block example">
 <p> Print the supported namespaced resources</p>
 </blockquote>
 <pre class="code-block example"><code class="lang-shell"><span class="hljs-attribute">kubectl</span> api-resources --namespaced=<span class="hljs-literal">true</span>
@@ -6957,12 +7007,6 @@ applications.</p>
 <td>Output format. One of: wide&#124;name. </td>
 </tr>
 <tr>
 <td>sort-by</td>
 <td></td>
 <td></td>
 <td>If non-empty, sort nodes list using specified field. The field can be either &#39;name&#39; or &#39;kind&#39;. </td>
 </tr>
 <tr>
 <td>verbs</td>
 <td></td>
 <td>[]</td>
@@ -3914,7 +3914,7 @@ image: nginx:1.10
 <TR><TD><CODE>readinessProbe</CODE><BR /><I><a href="#probe-v1-core">Probe</a></I></TD><TD>Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes</TD></TR>
 <TR><TD><CODE>resources</CODE><BR /><I><a href="#resourcerequirements-v1-core">ResourceRequirements</a></I></TD><TD>Compute Resources required by this container. Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/</TD></TR>
 <TR><TD><CODE>securityContext</CODE><BR /><I><a href="#securitycontext-v1-core">SecurityContext</a></I></TD><TD>Security options the pod should run with. More info: https://kubernetes.io/docs/concepts/policy/security-context/ More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/</TD></TR>
-<TR><TD><CODE>startupProbe</CODE><BR /><I><a href="#probe-v1-core">Probe</a></I></TD><TD>StartupProbe indicates that the Pod has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the Pod will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a Pod&#39;s lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation. This cannot be updated. This is an beta feature enabled by default that can be disabled using the StartupProbe feature flag. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes</TD></TR>
+<TR><TD><CODE>startupProbe</CODE><BR /><I><a href="#probe-v1-core">Probe</a></I></TD><TD>StartupProbe indicates that the Pod has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the Pod will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a Pod&#39;s lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation. This cannot be updated. This is an alpha feature enabled by the StartupProbe feature flag. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes</TD></TR>
 <TR><TD><CODE>stdin</CODE><BR /><I>boolean</I></TD><TD>Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF. Default is false.</TD></TR>
 <TR><TD><CODE>stdinOnce</CODE><BR /><I>boolean</I></TD><TD>Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF. Default is false</TD></TR>
 <TR><TD><CODE>terminationMessagePath</CODE><BR /><I>string</I></TD><TD>Optional: Path at which the file to which the container&#39;s termination message will be written is mounted into the container&#39;s filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated.</TD></TR>
@@ -1,228 +0,0 @@
 /*
 Kubernetes colors
 kubernetes blue - rgb(50, 109, 230)
 dark blue - rgb(51, 113, 227)
 dark grey - rgb(48, 48, 48)
 light grey - rgb(161, 160, 158)
 */
 /* User agent CSS overrides */
 #sidebar-wrapper ul, #sidebar-wrapper li {
    margin-left: 10px;
    padding-left: 0;
 }
 .body-content hr {
    margin: 2em 0;
    border-top: 2px solid dimgrey;
    border-bottom: 2px solid antiquewhite;
 }
 .body-content table {
    margin-bottom: 1em;
    width: 100%;
    overflow: auto;
 }
 .body-content table th, .body-content table td {
    text-align: left;
    vertical-align: top;
    line-height: 1.5;
 }
 .body-content table th {
    padding: 15px 20px;
    border-bottom: 1px solid lightsteelblue;
    vertical-align: bottom;
 }
 .body-content table td {
    padding: 10px;
 }
 .body-content table tr:last-child {
    border-bottom: 1px solid lightsteelblue;
 }
 .body-content table tr:nth-child(odd) > td {
    background-color: WhiteSmoke;
 }
 .body-content table tr:nth-child(even) > td {
    background-color: Gainsboro;
 }
 .body-content dt {
    font-weight: bold;
 }
 .body-content dd {
    margin-left: 15px;
 }
 .body-content p, .body-content li, .body-content dt, .body-content dd {
    line-height: 1.6;
    margin-top: 0;
 }
 /* Brodoc CSS */
 body > #wrapper {
    display: block;
    padding-bottom: 500px;
 }
 #sidebar-wrapper {
    display: block;
    height: 100%;
    width: 20%;
    position: fixed;
    z-index: 1;
    top: 0;
    left: 0;
    background-color: whitesmoke;
    border-right: 2px solid slategrey;
    overflow-x: hidden;
    padding-top: 60px;
 }
 #sidebar-wrapper a {
    text-decoration: none;
    white-space: nowrap;
    overflow: hidden;
    text-overflow: ellipsis;
    padding: 0 5px;
 }
 #sidebar-wrapper ul {
    list-style: none;
 }
 #sidebar-wrapper a.selected {
    font-style: bold;
    color: whitesmoke;
    border: 1px solid rgb(161, 160, 158);
    background-color: rgb(51, 113, 227);
    border-radius: 5px;
 }
 #sidebar-wrapper .strong-nav {
    font-family: monospace;
    font-weight: bold;
 }
 #sidebar-wrapper .nav-level-1.strong-nav {
    margin-top: 25px;
 }
 #sidebar-wrapper .copyright {
    padding-left: 10px;
    padding-top: 50px;
    padding-bottom: 50px;
    text-decoration: underline;
 }
 #page-content-wrapper {
    margin-left: 20%;
    padding-top: 60px;
 }
 .body-content h1, .body-content h2 {
    clear: both;
    border-bottom: 3px solid lightslategrey;
    padding-top: 20px;
 }
 .body-content > h3, .body-content > h4, .body-content > h5, .body-content > h6, .body-content > p, .body-content > aside, .body-content > ul > li, .body-content > ul > li {
    padding-top: 20px;
 }
 .body-content table tr td:not(:first-child) {
    overflow-wrap: break-word;
    word-wrap: break-word;
 }
 .body-content table tr td a {
    word-break: break-word;
 }
 .body-content p code {
    text-overflow: ellipsis;
    color: #802060;
    display: inline-block;
    font-size: smaller;
    word-break: break-word;
 }
 .body-content blockquote {
    border-left: 0;
    border-radius: 5px;
 }
 .body-content pre.code-block {
    margin-top: 5px;
    margin-bottom: 5px;
 }
 .body-content blockquote p, .body-content pre {
    color: black;
    font-size: 13px;
 }
 .body-content blockquote.code-block {
    background: Wheat;
 }
 .body-content pre.code-block code {
  word-wrap: normal;
  white-space: pre;
 }
 .body-content code {
  color: Brown !important;
 }
 .code-block {
    display: none;
    width: 60%;
    float: left;
    clear: right;
 }
 .code-block.active {
    display: initial;
 }
 #code-tabs-wrapper {
    width: 55%;
    height: 60px;
    /* position: fixed; */
    top: 0;
    right: 0;
 }
 #code-tabs-wrapper .code-tab-list {
    float: right;
    margin-top: 0;
    padding: 0 10px;
 }
 #code-tabs-wrapper .code-tab {
    color: white;
    /* display: inline-block; */
    padding: 0 30px;
    background: rgb(48, 48, 48);
    border: 1px solid rgb(161, 160, 158);
    border-radius: 5px;
 }
 #code-tabs-wrapper .tab-selected {
    background: rgb(51, 113, 227);
    font-style: bold;
    border-radius: 5px;
 }
 .side-nav a {
    color: black;
 }
@@ -1,7 +0,0 @@
 /**
 * Copyright (c) 2007-2015 Ariel Flesler - aflesler<a>gmail<d>com | http://flesler.blogspot.com
 * Licensed under MIT
 * @author Ariel Flesler
 * @version 2.1.2
 */
 ;(function(f){"use strict";"function"===typeof define&&define.amd?define(["jquery"],f):"undefined"!==typeof module&&module.exports?module.exports=f(require("jquery")):f(jQuery)})(function($){"use strict";function n(a){return!a.nodeName||-1!==$.inArray(a.nodeName.toLowerCase(),["iframe","#document","html","body"])}function h(a){return $.isFunction(a)||$.isPlainObject(a)?a:{top:a,left:a}}var p=$.scrollTo=function(a,d,b){return $(window).scrollTo(a,d,b)};p.defaults={axis:"xy",duration:0,limit:!0};$.fn.scrollTo=function(a,d,b){"object"=== typeof d&&(b=d,d=0);"function"===typeof b&&(b={onAfter:b});"max"===a&&(a=9E9);b=$.extend({},p.defaults,b);d=d||b.duration;var u=b.queue&&1<b.axis.length;u&&(d/=2);b.offset=h(b.offset);b.over=h(b.over);return this.each(function(){function k(a){var k=$.extend({},b,{queue:!0,duration:d,complete:a&&function(){a.call(q,e,b)}});r.animate(f,k)}if(null!==a){var l=n(this),q=l?this.contentWindow||window:this,r=$(q),e=a,f={},t;switch(typeof e){case "number":case "string":if(/^([+-]=?)?\d+(\.\d+)?(px|%)?$/.test(e)){e= h(e);break}e=l?$(e):$(e,q);case "object":if(e.length===0)return;if(e.is||e.style)t=(e=$(e)).offset()}var v=$.isFunction(b.offset)&&b.offset(q,e)||b.offset;$.each(b.axis.split(""),function(a,c){var d="x"===c?"Left":"Top",m=d.toLowerCase(),g="scroll"+d,h=r[g](),n=p.max(q,c);t?(f[g]=t[m]+(l?0:h-r.offset()[m]),b.margin&&(f[g]-=parseInt(e.css("margin"+d),10)||0,f[g]-=parseInt(e.css("border"+d+"Width"),10)||0),f[g]+=v[m]||0,b.over[m]&&(f[g]+=e["x"===c?"width":"height"]()*b.over[m])):(d=e[m],f[g]=d.slice&& "%"===d.slice(-1)?parseFloat(d)/100*n:d);b.limit&&/^\d+$/.test(f[g])&&(f[g]=0>=f[g]?0:Math.min(f[g],n));!a&&1<b.axis.length&&(h===f[g]?f={}:u&&(k(b.onAfterFirst),f={}))});k(b.onAfter)}})};p.max=function(a,d){var b="x"===d?"Width":"Height",h="scroll"+b;if(!n(a))return a[h]-$(a)[b.toLowerCase()]();var b="client"+b,k=a.ownerDocument||a.document,l=k.documentElement,k=k.body;return Math.max(l[h],k[h])-Math.min(l[b],k[b])};$.Tween.propHooks.scrollLeft=$.Tween.propHooks.scrollTop={get:function(a){return $(a.elem)[a.prop]()}, set:function(a){var d=this.get(a);if(a.options.interrupt&&a._last&&a._last!==d)return $(a.elem).stop();var b=Math.round(a.now);d!==b&&($(a.elem)[a.prop](b),a._last=this.get(a))}};return p});
@@ -1,196 +0,0 @@
 $(document).ready(function() {
    /**
     * TODO: Refactor with intent toward pure functions. Mutation of state can lead to bugs and difficult debugging.
     */
    var toc = navData.toc;
    var flatToc = navData.flatToc.reverse();
    function collectNodes(tocMap) {
        var tocNodes = {};
        tocMap.map(function(node, index) {
            var sectionNode = $('#' + node.section);
            var tocSubsections = {};
            tocItem = {section: sectionNode};
            var subsectionNodes;
            if (node.subsections) {
                subsectionNodes = (collectNodes(node.subsections));
                tocItem.subsections = subsectionNodes;
            }
            tocNodes[node.section] = tocItem;
        });
        return tocNodes;
    }
    var tocItems = collectNodes(toc);
    function collectNodesFlat(tocMap, obj) {
        var collect = obj || {};
        tocMap.map(function(node, index) {
            var sectionNode = $('#' + node.section);
            tocItem = {section: sectionNode};
            if (node.subsections) {
                subsectionNodes = (collectNodesFlat(node.subsections, collect));
            }
            collect[node.section] = sectionNode;
        });
        return collect;
    }
    var tocFlat = collectNodesFlat(toc);
    var prevSectionToken;
    var prevSubsectionToken;
    var activeTokensObj = {};
    function scrollActions(scrollPosition) {
        var activeSection = checkNodePositions(toc, tocFlat, scrollPosition);
        var activeSubSection,
            prevL1Nav,
            currL1Nav,
            prevL2Nav,
            currL2Nav;
        // No active section - return existing activeTokensObj (may be empty)
        if (!activeSection) {
            return activeTokensObj;
        }
        /**
         * This block deals with L1Nav sections
         */
        // If no previous token, set previous to current active and show L1Nav
        if (!prevSectionToken) {
            prevSectionToken = activeSection.token;
            currL1Nav = getNavNode(activeSection.token);
            currL1Nav.show('fast');
        } 
        // If active is not the same as previous, hide previous L1Nav and show current L1Nav; set previous to current
        else if (activeSection.token !== prevSectionToken) {
            prevL1Nav = getNavNode(prevSectionToken);
            currL1Nav = getNavNode(activeSection.token);
            prevL1Nav.hide('fast');
            currL1Nav.show('fast');
            prevSectionToken = activeSection.token;
        }
        /**
         * This block deals with L2Nav subsections
         */
        // If there is a subsections array and it has a non-zero length, set active subsection
        if (activeSection.subsections && activeSection.subsections.length !== 0) {
            activeSubSection = checkNodePositions(activeSection.subsections, tocFlat, scrollPosition);
            if (activeSubSection) {
                if (!prevSubsectionToken) {
                    prevSubsectionToken = activeSubSection.token;
                    currL2Nav = getNavNode(activeSubSection.token);
                    currL2Nav.show('fast');
                } else if (activeSubSection.token !== prevSubsectionToken) {
                    prevL2Nav = getNavNode(prevSubsectionToken);
                    currL2Nav = getNavNode(activeSubSection.token);
                    prevL2Nav.hide('fast');
                    currL2Nav.show('fast');
                    prevSubsectionToken = activeSubSection.token;
                }
            } else {
                prevL2Nav = getNavNode(prevSubsectionToken);
                prevL2Nav.hide('fast');
                prevSubsectionToken = null;
            }
        }
        activeTokensObj.L1 = prevSectionToken;
        activeTokensObj.L2 = prevSubsectionToken;
        return activeTokensObj;
    }
    /**
     * Checks for active elements by scroll position
     */
    var prevElemToken;
    var activeElemToken;
    function checkActiveElement(items, scrollPosition) {
        var offset = 50;
        var offsetScroll = scrollPosition + offset;
        var visibleNode;
        for (var i = 0; i < items.length; i++) {
            var token = items[i];
            var node = getHeadingNode(token);
            if (offsetScroll >= node.offset().top) {
                activeElemToken = token;
            }
        }
        if (!prevElemToken) {
            getNavElemNode(activeElemToken).addClass('selected');
            prevElemToken = activeElemToken;
            return;
        }
        if (activeElemToken !== prevElemToken) {
            getNavElemNode(prevElemToken).removeClass('selected');
            getNavElemNode(activeElemToken).addClass('selected');
            prevElemToken = activeElemToken;
        }
        return activeElemToken;
    }
    function getHeadingNode(token) {
        return $('#' + token);
    }
    function getNavNode(token) {
        return $('#' + token + '-nav');
    }
    function getNavElemNode(token) {
        return $('#sidebar-wrapper > ul a[href="#' + token + '"]');
    }
    function checkNodePositions(nodes, flatNodeMap, scrollPosition) {
        var activeNode;
        for (var i = 0; i < nodes.length; i++) {
            var item = nodes[i];
            var node = flatNodeMap[item.section];
            var nodeTop = node.offset().top - 50;
            if (scrollPosition >= nodeTop) {
                activeNode = {token: item.section, node: node};
                if (item.subsections) {
                    activeNode.subsections = item.subsections;
                }
                break;
            }
        }
        return activeNode;
    }
    function scrollToNav(token) {
        setTimeout(function() {
            var scrollPosition = $(window).scrollTop();
            var activeSectionTokens = scrollActions(scrollPosition);
            var activeElemToken = checkActiveElement(flatToc, scrollPosition);
            var navNode = $('#sidebar-wrapper > ul a[href="#' + token + '"]');
            $('#sidebar-wrapper').scrollTo(navNode, {duration: 'fast', axis: 'y'});
        }, 200);
    }
    $(window).on('hashchange', function(event) {
        var scrollPosition = $(window).scrollTop();
        var activeSectionTokens = scrollActions(scrollPosition);
        var activeElemToken = checkActiveElement(flatToc, scrollPosition);
        var scrollToken = activeSectionTokens.L2 ? activeSectionTokens.L2 : activeSectionTokens.L1;
        scrollToNav(scrollToken);
        var token = location.hash.slice(1);
    });
    var scrollPosition = $(window).scrollTop();
    scrollActions(scrollPosition);
    checkActiveElement(flatToc, scrollPosition);
    // TODO: prevent scroll on sidebar from propagating to window
    $(window).on('scroll', function(event) {
        var scrollPosition = $(window).scrollTop();
        var activeSectionTokens = scrollActions(scrollPosition);
        var activeElemToken = checkActiveElement(flatToc, scrollPosition);
    });
 });