CRD versioning Public Documentation (#8834)

* CRD versioning Public Documentation

* Copyedit

Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com>

* Address feedback

* More rewrites

* Address feedback

* Update main CRD page in light of versioning

* Reorg CRD docs

* Further reorg

* Tweak title

content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md

@@ -8,7 +8,11 @@ weight: 20
 ---
 
 {{% capture overview %}}
-This page explains [*custom resources*](/docs/concepts/api-extension/custom-resources/), which are extensions of the Kubernetes API. This page explains when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.
+
+This page explains *custom resources*, which are extensions of the Kubernetes
+API, including when to add a custom resource to your Kubernetes cluster and when
+to use a standalone service. It describes the two methods for adding custom
+resources and how to choose between them.
 
 {{% /capture %}}
 
@@ -103,20 +107,20 @@ Use a custom resource (CRD or Aggregated API) if most of the following apply:
 
 Kubernetes provides two ways to add custom resources to your cluster:
 
-- [Custom Resource Definitions](/docs/concepts/api-extension/custom-resources/) (CRDs) are easier to use: they do not require any programming in some cases.
+- CRDs are simple and do not always require programming.
 - [API Aggregation](/docs/concepts/api-extension/apiserver-aggregation/) requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.
 
 Kubernetes provides these two options to meet the needs of different users, so that neither ease of use nor flexibility are compromised.
 
 Aggregated APIs are subordinate APIServers that sit behind the primary API server, which acts as a proxy. This arrangement is called [API Aggregation](/docs/concepts/api-extension/apiserver-aggregation/) (AA). To users, it simply appears that the Kubernetes API is extended.
 
-Custom Resource Definitions (CRDS) allow users to create new types of resources without adding another APIserver. You do not need to understand API Aggregation to use CRDs.
+CRDs allow users to create new types of resources without adding another APIserver. You do not need to understand API Aggregation to use CRDs.
 
-Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods).
+Regardless of how they are installed, the new resources are referred to as Custom Resources to distinguish them from built-in Kubernetes resources (like pods).
 
 ## CustomResourceDefinitions
 
-The [CustomResourceDefinition](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/) (CRD) API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.
+The [CustomResourceDefinition](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/) API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.
 
 This frees you from writing your own API server to handle the custom resource,
 but the generic nature of the implementation means you have less flexibility than with
@@ -132,7 +136,7 @@ and setup a controller to handle events.
 
 ## API server aggregation
 
-Usually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like *pods* and *services*, and can also handle custom resources in a generic way through [CustomResourceDefinitions](#customresourcedefinitions).
+Usually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like *pods* and *services*, and can also handle custom resources in a generic way through [CRDs](#customresourcedefinitions).
 
 The [aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/) allows you to provide specialized
 implementations for your custom resources by writing and deploying your own standalone API server.
@@ -152,7 +156,7 @@ Typically, CRDs are a good fit if:
 
 CRDs are easier to create than Aggregated APIs.
 
-| Custom Resource Definitions | Aggregated API |
+| CRDs                        | Aggregated API |
 | --------------------------- | -------------- |
 | Do not require programming. Users can choose any language for a CRD controller. | Requires programming in Go and building binary and image. Users can choose any language for a CRD controller. |
 | No additional service to run; CRs are handled by API Server. | An additional service to create and that could fail. |

content/en/docs/tasks/access-kubernetes-api/configure-aggregation-layer.md

@@ -5,6 +5,7 @@ reviewers:
 - cheftako
 - chenopis
 content_template: templates/task
+weight: 10
 ---
 
 {{% capture overview %}}

content/en/docs/tasks/access-kubernetes-api/custom-resources/_index.md

@@ -0,0 +1,5 @@
+---
+title: "Use Custom Resources"
+weight: 10
+---
+

content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md

@@ -0,0 +1,201 @@
+---
+title: Versions of CustomResourceDefinitions
+reviewers:
+- mbohlool
+- sttts
+- liggitt
+content_template: templates/task
+weight: 30
+---
+
+{{% capture overview %}}
+This page explains how to add versioning information to
+[CustomResourceDefinitions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions), to indicate the stability
+level of your CustomResourceDefinitions. It also describes how to upgrade an
+object from one version to another.
+
+{{< note >}}
+**Note**: All specified versions must use the same schema. The is no schema
+conversion between versions.
+{{< /note >}}
+
+{{% /capture %}}
+
+{{% capture prerequisites %}}
+
+{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
+
+* Make sure your Kubernetes cluster has a master version of 1.11.0 or higher.
+
+* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
+
+{{% /capture %}}
+
+{{% capture steps %}}
+
+## Overview
+
+The CustomResourceDefinition API supports a `versions` field that you can use to
+support multiple versions of custom resources that you have developed, and
+indicate the stability of a given custom resource. All versions must currently
+use the same schema, so if you need to add a field, you must add it to all
+versions.
+
+{{< note >}}
+Earlier iterations included a `version` field instead of `versions`. The
+`version` field is deprecated and optional, but if it is not empty, it must
+match the first item in the `versions` field.
+{{< /note >}}
+
+## Specify multiple versions
+
+This example shows a CustomResourceDefinition with two versions. The comments in
+the YAML provide more context.
+
+```yaml
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  # name must match the spec fields below, and be in the form: <plural>.<group>
+  name: crontabs.example.com
+spec:
+  # group name to use for REST API: /apis/<group>/<version>
+  group: example.com
+  # list of versions supported by this CustomResourceDefinition
+  versions:
+  - name: v1beta1
+    # Each version can be enabled/disabled by Served flag.
+    served: true
+    # One and only one version must be marked as the storage version.
+    storage: true
+  - name: v1
+    served: true
+    storage: false
+  # either Namespaced or Cluster
+  scope: Namespaced
+  names:
+    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
+    plural: crontabs
+    # singular name to be used as an alias on the CLI and for display
+    singular: crontab
+    # kind is normally the CamelCased singular type. Your resource manifests use this.
+    kind: CronTab
+    # shortNames allow shorter string to match your resource on the CLI
+    shortNames:
+    - ct
+```
+
+You can save the CustomResourceDefinition in a YAML file, then use
+`kubectl create` to create it.
+
+```shell
+kubectl create -f my-versioned-crontab.yaml
+```
+
+After creation, the API server starts to serve each enabled version at an HTTP
+REST endpoint. In the above example, the API versions are available at
+`/apis/example.com/v1beta1` and `/apis/example.com/v1`.
+
+### Version priority
+
+Regardless of the order in which versions are defined in a
+CustomResourceDefinition, the version with the highest priority is used by
+kubectl as the default version to access objects. The priority is determined
+by parsing the _name_ field to determine the version number, the stability
+(GA, Beta, or Alpha), and the sequence within that stability level.
+
+The algorithm used for sorting the versions is designed to sort versions in the
+same way that the Kubernetes project sorts Kubernetes versions. Versions start with a
+`v` followed by a number, an optional `beta` or `alpha` designation, and
+optional additional versioning information. Broadly, a version string might look
+like `v2` or `v2beta1`. Versions are sorted using the following algorithm:
+  
+- Entries that follow Kubernetes version patterns are sorted before those that
+  do not.
+- For entries that follow Kubernetes version patterns, the numeric portions of
+  the version string is sorted largest to smallest.
+- If the strings `beta` or `alpha` follow the first numeric portion, they sorted
+  in that order, after the equivalent string without the `beta` or `alpha`
+  suffix (which is presumed to be the GA version).
+- If another number follows the `beta`, or `alpha`, those numbers are also
+  sorted from largest to smallest.
+- Strings that don't fit the above format are sorted alphabetically and the
+  numeric portions are not treated specially. Notice that in the example below,
+  `foo1` is sorted above `foo10`. This is different from the sorting of the
+  numeric portion of entries that do follow the Kubernetes version patterns.
+
+This might make sense if you look at the following sorted version list:
+
+```none
+- v10
+- v2
+- v1
+- v11beta2
+- v10beta3
+- v3beta1
+- v12alpha1
+- v11alpha2
+- foo1
+- foo10
+```
+
+For the example in [Specify multiple versions](#specify-multiple-versions), the
+version sort order is `v1`, followed by `v1beta1`. This causes the kubectl
+command to use `v1` as the default version unless the provided object specifies
+the version.
+
+## Writing, reading, and updating versioned CustomResourceDefinition objects
+
+When an object is written, it is persisted at the version designated as the
+storage version at the time of the write. If the storage version changes,
+existing objects are never converted automatically. However, newly-created
+or updated objects are created at the new storage version. It is possible for an
+object to have been written at a version that is no longer served.
+
+When you read an object, you specify the version as part of the path. If you
+specify a version that is different from the object's persisted version,
+Kubernetes returns the object to you at the version you requested, but does not
+modify the persisted object. You can request an object at any version that is
+currently served.
+
+If you update an existing object, it is rewritten at the version that is
+currently the storage version. This is the only way that objects can change from
+one version to another.
+
+To illustrate this, consider the following hypothetical series of events:
+
+1.  The storage version is `v1beta`. You create an object. It is persisted in
+    storage at version `v1beta1`
+2.  You add version `v1` to your CustomResourceDefinition and designate it as
+    the storage version.
+3.  You read your object at version `v1beta`, then you read the object again at
+    version `v1`. Both returned objects are identical except for the apiVersion
+    field.
+4.  You create a new object. It is persisted in storage at version `v1`. You now
+    have two objects, one of which is at `v1beta`, and the other of which is at
+    `v1`.
+5.  You update the first object. It is now persisted at version `v1` since that
+    is the current storage version.
+
+### Previous storage versions
+
+The API server records each version which has ever been marked as the storage
+version in the status field `storedVersions`. Objects may have been persisted
+at any version that has ever been designated as a storage version. No objects
+can exist in storage at a version that has never been a storage version.
+
+## Upgrade existing objects to a new stored version
+
+When deprecating versions and dropping support, devise a storage upgrade
+procedure. The following is an example procedure to upgrade from `v1beta1`
+to `v1`.
+
+1.  Set `v1` as the storage in the CustomResourceDefinition file and apply it
+    using kubectl. The `storedVersions` is now `v1beta1, v1`.
+2.  Write an upgrade procedure to list all existing objects and write them with
+    the same content. This forces the backend to write objects in the current
+    storage version, which is `v1`.
+3.  Update the CustomResourceDefinition `Status` by removing `v1beta1` from
+    `storedVersions` field.
+
+{{% /capture %}}

content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md

@@ -4,11 +4,12 @@ reviewers:
 - deads2k
 - enisoc
 content_template: templates/task
+weight: 20
 ---
 
 {{% capture overview %}}
 This page shows how to install a
-[custom resource](/docs/concepts/api-extension/custom-resources/)
+[custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
 into the Kubernetes API by creating a
 [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
 {{% /capture %}}
@@ -27,10 +28,11 @@ into the Kubernetes API by creating a
 ## Create a CustomResourceDefinition
 
 When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server
-reacts by creating a new RESTful resource path, either namespaced or cluster-scoped,
+creates a new RESTful resource path for each version you specify. The CRD can be
-as specified in the CRD's `scope` field. As with existing built-in objects, deleting a
+either namespaced or cluster-scoped, as specified in the CRD's `scope` field. As
-namespace deletes all custom objects in that namespace.
+with existing built-in objects, deleting a namespace deletes all custom objects
-CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.
+in that namespace. CustomResourceDefinitions themselves are non-namespaced and
+are available to all namespaces.
 
 For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`:
 
@@ -43,8 +45,13 @@ metadata:
 spec:
   # group name to use for REST API: /apis/<group>/<version>
   group: stable.example.com
-  # version name to use for REST API: /apis/<group>/<version>
+  # list of versions supported by this CustomResourceDefinition
-  version: v1
+  versions:
+    - name: v1
+      # Each version can be enabled/disabled by Served flag.
+      served: true
+      # One and only one version must be marked as the storage version.
+      storage: true
   # either Namespaced or Cluster
   scope: Namespaced
   names:
@@ -75,7 +82,7 @@ This endpoint URL can then be used to create and manage custom objects.
 The `kind` of these objects will be `CronTab` from the spec of the
 CustomResourceDefinition object you created above.
 
-Please note that it might take a few seconds for the endpoint to be created.
+It might take a few seconds for the endpoint to be created.
 You can watch the `Established` condition of your CustomResourceDefinition
 to be true or watch the discovery information of the API server for your
 resource to show up.
@@ -120,9 +127,8 @@ NAME                 AGE
 my-new-cron-object   6s
 ```
 
-Note that resource names are not case-sensitive when using kubectl,
+Resource names are not case-sensitive when using kubectl, and you can use either
-and you can use either the singular or plural forms defined in the CRD,
+the singular or plural forms defined in the CRD, as well as any short names.
-as well as any short names.
 
 You can also view the raw YAML data:
 
@@ -173,6 +179,12 @@ Error from server (NotFound): Unable to list "crontabs": the server could not fi
 
 If you later recreate the same CustomResourceDefinition, it will start out empty.
 
+## Serving multiple versions of a CRD
+
+See [Custom resource definition versioning](custom-resource-definition-versioning)
+for more information about serving multiple versions of your
+CustomResourceDefinition and migrating your objects from one version to another.
+
 {{% /capture %}}
 
 {{% capture discussion %}}
@@ -245,6 +257,10 @@ metadata:
   name: crontabs.stable.example.com
 spec:
   group: stable.example.com
+  versions:
+    - name: v1
+      served: true
+      storage: true
   version: v1
   scope: Namespaced
   names:
@@ -396,7 +412,10 @@ metadata:
   name: crontabs.stable.example.com
 spec:
   group: stable.example.com
-  version: v1
+  versions:
+    - name: v1
+      served: true
+      storage: true
   scope: Namespaced
   names:
     plural: crontabs
@@ -487,7 +506,10 @@ metadata:
   name: crontabs.stable.example.com
 spec:
   group: stable.example.com
-  version: v1
+  versions:
+    - name: v1
+      served: true
+      storage: true
   scope: Namespaced
   names:
     plural: crontabs
@@ -544,6 +566,8 @@ crontabs/my-new-cron-object   3s
 {{% capture whatsnext %}}
 * Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
 * See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions-k8s-io).
+* Serve [multiple versions](custom-resource-definitions-versioning) of a
+  CustomResourceDefinition
 {{% /capture %}}
 
 

content/en/docs/tasks/access-kubernetes-api/custom-resources/migrate-third-party-resource.md

@@ -4,6 +4,7 @@ reviewers:
 - enisoc
 - deads2k
 content_template: templates/task
+weight: 50
 ---
 
 {{% capture overview %}}
@@ -71,7 +72,10 @@ you **on a best-effort basis**.
     spec:
       scope: Namespaced
       group: stable.example.com
-      version: v1
+      versions:
+        - name: v1
+          served: true
+          storage: true
       names:
         kind: CronTab
         plural: crontabs
@@ -163,7 +167,7 @@ you **on a best-effort basis**.
 
 {{% capture whatsnext %}}
 * Learn more about [custom resources](/docs/concepts/api-extension/custom-resources/).
-* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
+* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/).
 * See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
 {{% /capture %}}
 

content/en/docs/tasks/access-kubernetes-api/http-proxy-access-api.md

@@ -1,6 +1,7 @@
 ---
 title: Use an HTTP Proxy to Access the Kubernetes API
 content_template: templates/task
+weight: 40
 ---
 
 {{% capture overview %}}

content/en/docs/tasks/access-kubernetes-api/setup-extension-api-server.md

@@ -5,6 +5,7 @@ reviewers:
 - cheftako
 - chenopis
 content_template: templates/task
+weight: 15
 ---
 
 {{% capture overview %}}

static/_redirects

@@ -224,6 +224,8 @@
 /docs/tasks/access-application-cluster/access-cluster.md     /docs/tasks/access-application-cluster/access-cluster/ 301
 /docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/     /docs/tasks/access-application-cluster/configure-access-multiple-clusters/ 301
 /docs/tasks/access-kubernetes-api/access-kubernetes-api/http-proxy-access-api/     /docs/tasks/access-kubernetes-api/http-proxy-access-api/ 301
+/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/     /docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/ 301
+/docs/tasks/access-kubernetes-api/migrate-third-party-resource/    /docs/tasks/access-kubernetes-api/custom-resources/migrate-third-party-resource/ 301
 
 /docs/tasks/administer-cluster/apply-resource-quota-limit/     /docs/tasks/administer-cluster/quota-api-object/ 301
 /docs/tasks/administer-cluster/assign-pods-nodes/     /docs/tasks/configure-pod-container/assign-pods-nodes/ 301