Add comprehensive CRD versioning flow to documentation (#16026)

* Add comprehensive CRD versioning flow to documentation

* Break down CRD versioning upgrade processes into steps, apply review feedback

* Reference doc section about custom resource storage migration in workflow documentation

* Add step to find any clients still on old version during CRD version migration

* Fix indentation in list in conversion end-to-end flow doc

* Fix spacing before lists in docs about conversion webhooks

* Fix two typos

content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md

@@ -30,9 +30,69 @@ level of your CustomResourceDefinitions or advance your API to a new version wit
 
 {{< feature-state state="stable" for_kubernetes_version="1.16" >}}
 
-The CustomResourceDefinition API supports a `versions` field that you can use to
+The CustomResourceDefinition API provides a workflow for introducing and upgrading
-support multiple versions of custom resources that you have developed. Versions
+to new versions of a CustomResourceDefinition.
-can have different schemas with a conversion webhook to convert custom resources between versions.
+
+When a CustomResourceDefinition is created, the first version is set in the
+CustomResourceDefinition `spec.versions` list to an appropriate stability level
+and a version number. For example `v1beta1` would indicate that the first
+version is not yet stable. All custom resource objects will initially be stored
+at this version.
+
+Once the CustomResourceDefinition is created, clients may begin using the
+`v1beta1` API.
+
+Later it might be necessary to add new version such as `v1`.
+
+Adding a new version:
+
+1. Pick a conversion strategy. Since custom resource objects need to be able to
+   be served at both versions, that means they will sometimes be served at a
+   different version than their storage version. In order for this to be
+   possible, the custom resource objects must sometimes be converted between the
+   version they are stored at and the version they are served at. If the
+   conversion involves schema changes and requires custom logic, a conversion
+   webhook should be used. If there are no schema changes, the default `None`
+   conversion strategy may be used and only the `apiVersion` field will be
+   modified when serving different versions.
+1. If using conversion webhooks, create and deploy the conversion webhook. See
+   the [Webhook conversion](#webhook-conversion) for more details.
+1. Update the CustomResourceDefinition to include the new version in the
+   `spec.versions` list with `served:true`.  Also, set `spec.conversion` field
+   to the selected conversion strategy. If using a conversion webhook, configure
+   `spec.conversion.webhookClientConfig` field to call the webhook.
+
+Once the new version is added, clients may incrementally migrate to the new
+version. It is perfectly safe for some clients to use the old version while
+others use the new version.
+
+Migrate stored objects to the new version:
+
+1. See the [upgrade existing objects to a new stored version](#upgrade-existing-objects-to-a-new-stored-version) section.
+
+It is safe for clients to use both the old and new version before, during and
+after upgrading the objects to a new stored version.
+
+Removing an old version:
+
+1. Ensure all clients are fully migrated to the new version. The kube-apiserver
+   logs can reviewed to help identify any clients that are still accessing via
+   the old version.
+1. Set `served` to `false` for the old version in the `spec.versions` list. If
+   any clients are still unexpectedly using the old version they may begin reporting
+   errors attempting to access the custom resource objects at the old version.
+   If this occurs, switch back to using `served:true` on the old version, migrate the 
+   remaining clients to the new version and repeat this step.
+1. Ensure the [upgrade of existing objects to the new stored version](#upgrade-existing-objects-to-a-new-stored-version) step has been completed.
+    1. Verify that the `stored` is set to `true` for the new version in the `spec.versions` list in the CustomResourceDefinition.
+    1. Verify that the old version is no longer listed in the CustomResourceDefinition `status.storedVersions`.
+1. Remove the old version from the CustomResourceDefinition `spec.versions` list.
+1. Drop conversion support for the old version in conversion webhooks.
+
+## Specify multiple versions
+
+The CustomResourceDefinition API `versions` field can be used to support multiple versions of custom resources that you
+have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions.
 Webhook conversions should follow the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md) wherever applicable.
 Specifically, See the [API change documentation](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) for a set of useful gotchas and suggestions.
 
@@ -42,8 +102,6 @@ In `apiextensions.k8s.io/v1beta1`, there was a `version` field instead of `versi
 match the first item in the `versions` field.
 {{< /note >}}
 
-## Specify multiple versions
-
 This example shows a CustomResourceDefinition with two versions. For the first
 example, the assumption is all versions share the same schema with no conversion
 between them. The comments in the YAML provide more context.
@@ -885,16 +943,23 @@ can exist in storage at a version that has never been a storage version.
 
 ## Upgrade existing objects to a new stored version
 
-When deprecating versions and dropping support, devise a storage upgrade
+When deprecating versions and dropping support, select a storage upgrade
-procedure. The following is an example procedure to upgrade from `v1beta1`
+procedure. 
-to `v1`.
+
+*Option 1:* Use the Storage Version Migrator
+
+1. Run the [storage Version migrator](https://github.com/kubernetes-sigs/kube-storage-version-migrator)
+2. Remove the old version from the CustomResourceDefinition `status.storedVersions` field.
+
+*Option 2:* Manually upgrade the existing objects to a new stored version
+
+The following is an example procedure to upgrade from `v1beta1` to `v1`.
 
 1.  Set `v1` as the storage in the CustomResourceDefinition file and apply it
     using kubectl. The `storedVersions` is now `v1beta1, v1`.
 2.  Write an upgrade procedure to list all existing objects and write them with
     the same content. This forces the backend to write objects in the current
     storage version, which is `v1`.
-3.  Update the CustomResourceDefinition `Status` by removing `v1beta1` from
+2. Remove `v1beta1` from the CustomResourceDefinition `status.storedVersions` field.
-    `storedVersions` field.
 
 {{% /capture %}}