From d29e8ebcf411cee02ecdc6b7f00be95c7d9462d4 Mon Sep 17 00:00:00 2001 From: Joe Betz Date: Sun, 8 Sep 2019 21:09:17 -0700 Subject: [PATCH] Add comprehensive CRD versioning flow to documentation (#16026) * Add comprehensive CRD versioning flow to documentation * Break down CRD versioning upgrade processes into steps, apply review feedback * Reference doc section about custom resource storage migration in workflow documentation * Add step to find any clients still on old version during CRD version migration * Fix indentation in list in conversion end-to-end flow doc * Fix spacing before lists in docs about conversion webhooks * Fix two typos --- .../custom-resource-definition-versioning.md | 85 ++++++++++++++++--- 1 file changed, 75 insertions(+), 10 deletions(-) diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md index 45cabeefd4..35d0e2bb60 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md @@ -30,9 +30,69 @@ level of your CustomResourceDefinitions or advance your API to a new version wit {{< feature-state state="stable" for_kubernetes_version="1.16" >}} -The CustomResourceDefinition API supports a `versions` field that you can use to -support multiple versions of custom resources that you have developed. Versions -can have different schemas with a conversion webhook to convert custom resources between versions. +The CustomResourceDefinition API provides a workflow for introducing and upgrading +to new versions of a CustomResourceDefinition. + +When a CustomResourceDefinition is created, the first version is set in the +CustomResourceDefinition `spec.versions` list to an appropriate stability level +and a version number. For example `v1beta1` would indicate that the first +version is not yet stable. All custom resource objects will initially be stored +at this version. + +Once the CustomResourceDefinition is created, clients may begin using the +`v1beta1` API. + +Later it might be necessary to add new version such as `v1`. + +Adding a new version: + +1. Pick a conversion strategy. Since custom resource objects need to be able to + be served at both versions, that means they will sometimes be served at a + different version than their storage version. In order for this to be + possible, the custom resource objects must sometimes be converted between the + version they are stored at and the version they are served at. If the + conversion involves schema changes and requires custom logic, a conversion + webhook should be used. If there are no schema changes, the default `None` + conversion strategy may be used and only the `apiVersion` field will be + modified when serving different versions. +1. If using conversion webhooks, create and deploy the conversion webhook. See + the [Webhook conversion](#webhook-conversion) for more details. +1. Update the CustomResourceDefinition to include the new version in the + `spec.versions` list with `served:true`. Also, set `spec.conversion` field + to the selected conversion strategy. If using a conversion webhook, configure + `spec.conversion.webhookClientConfig` field to call the webhook. + +Once the new version is added, clients may incrementally migrate to the new +version. It is perfectly safe for some clients to use the old version while +others use the new version. + +Migrate stored objects to the new version: + +1. See the [upgrade existing objects to a new stored version](#upgrade-existing-objects-to-a-new-stored-version) section. + +It is safe for clients to use both the old and new version before, during and +after upgrading the objects to a new stored version. + +Removing an old version: + +1. Ensure all clients are fully migrated to the new version. The kube-apiserver + logs can reviewed to help identify any clients that are still accessing via + the old version. +1. Set `served` to `false` for the old version in the `spec.versions` list. If + any clients are still unexpectedly using the old version they may begin reporting + errors attempting to access the custom resource objects at the old version. + If this occurs, switch back to using `served:true` on the old version, migrate the + remaining clients to the new version and repeat this step. +1. Ensure the [upgrade of existing objects to the new stored version](#upgrade-existing-objects-to-a-new-stored-version) step has been completed. + 1. Verify that the `stored` is set to `true` for the new version in the `spec.versions` list in the CustomResourceDefinition. + 1. Verify that the old version is no longer listed in the CustomResourceDefinition `status.storedVersions`. +1. Remove the old version from the CustomResourceDefinition `spec.versions` list. +1. Drop conversion support for the old version in conversion webhooks. + +## Specify multiple versions + +The CustomResourceDefinition API `versions` field can be used to support multiple versions of custom resources that you +have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions. Webhook conversions should follow the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md) wherever applicable. Specifically, See the [API change documentation](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) for a set of useful gotchas and suggestions. @@ -42,8 +102,6 @@ In `apiextensions.k8s.io/v1beta1`, there was a `version` field instead of `versi match the first item in the `versions` field. {{< /note >}} -## Specify multiple versions - This example shows a CustomResourceDefinition with two versions. For the first example, the assumption is all versions share the same schema with no conversion between them. The comments in the YAML provide more context. @@ -885,16 +943,23 @@ can exist in storage at a version that has never been a storage version. ## Upgrade existing objects to a new stored version -When deprecating versions and dropping support, devise a storage upgrade -procedure. The following is an example procedure to upgrade from `v1beta1` -to `v1`. +When deprecating versions and dropping support, select a storage upgrade +procedure. + +*Option 1:* Use the Storage Version Migrator + +1. Run the [storage Version migrator](https://github.com/kubernetes-sigs/kube-storage-version-migrator) +2. Remove the old version from the CustomResourceDefinition `status.storedVersions` field. + +*Option 2:* Manually upgrade the existing objects to a new stored version + +The following is an example procedure to upgrade from `v1beta1` to `v1`. 1. Set `v1` as the storage in the CustomResourceDefinition file and apply it using kubectl. The `storedVersions` is now `v1beta1, v1`. 2. Write an upgrade procedure to list all existing objects and write them with the same content. This forces the backend to write objects in the current storage version, which is `v1`. -3. Update the CustomResourceDefinition `Status` by removing `v1beta1` from - `storedVersions` field. +2. Remove `v1beta1` from the CustomResourceDefinition `status.storedVersions` field. {{% /capture %}}