Add comprehensive CRD versioning flow to documentation (#16026)
* Add comprehensive CRD versioning flow to documentation * Break down CRD versioning upgrade processes into steps, apply review feedback * Reference doc section about custom resource storage migration in workflow documentation * Add step to find any clients still on old version during CRD version migration * Fix indentation in list in conversion end-to-end flow doc * Fix spacing before lists in docs about conversion webhooks * Fix two typos
This commit is contained in:
Joe Betz
committed by
Kubernetes Prow Robot
Kubernetes Prow Robot
parent
bc113745fa
commit
d29e8ebcf4
+75
-10
@@ -30,9 +30,69 @@ level of your CustomResourceDefinitions or advance your API to a new version wit
|
||||
|
||||
{{< feature-state state="stable" for_kubernetes_version="1.16" >}}
|
||||
|
||||
The CustomResourceDefinition API supports a `versions` field that you can use to
|
||||
support multiple versions of custom resources that you have developed. Versions
|
||||
can have different schemas with a conversion webhook to convert custom resources between versions.
|
||||
The CustomResourceDefinition API provides a workflow for introducing and upgrading
|
||||
to new versions of a CustomResourceDefinition.
|
||||
|
||||
When a CustomResourceDefinition is created, the first version is set in the
|
||||
CustomResourceDefinition `spec.versions` list to an appropriate stability level
|
||||
and a version number. For example `v1beta1` would indicate that the first
|
||||
version is not yet stable. All custom resource objects will initially be stored
|
||||
at this version.
|
||||
|
||||
Once the CustomResourceDefinition is created, clients may begin using the
|
||||
`v1beta1` API.
|
||||
|
||||
Later it might be necessary to add new version such as `v1`.
|
||||
|
||||
Adding a new version:
|
||||
|
||||
1. Pick a conversion strategy. Since custom resource objects need to be able to
|
||||
be served at both versions, that means they will sometimes be served at a
|
||||
different version than their storage version. In order for this to be
|
||||
possible, the custom resource objects must sometimes be converted between the
|
||||
version they are stored at and the version they are served at. If the
|
||||
conversion involves schema changes and requires custom logic, a conversion
|
||||
webhook should be used. If there are no schema changes, the default `None`
|
||||
conversion strategy may be used and only the `apiVersion` field will be
|
||||
modified when serving different versions.
|
||||
1. If using conversion webhooks, create and deploy the conversion webhook. See
|
||||
the [Webhook conversion](#webhook-conversion) for more details.
|
||||
1. Update the CustomResourceDefinition to include the new version in the
|
||||
`spec.versions` list with `served:true`. Also, set `spec.conversion` field
|
||||
to the selected conversion strategy. If using a conversion webhook, configure
|
||||
`spec.conversion.webhookClientConfig` field to call the webhook.
|
||||
|
||||
Once the new version is added, clients may incrementally migrate to the new
|
||||
version. It is perfectly safe for some clients to use the old version while
|
||||
others use the new version.
|
||||
|
||||
Migrate stored objects to the new version:
|
||||
|
||||
1. See the [upgrade existing objects to a new stored version](#upgrade-existing-objects-to-a-new-stored-version) section.
|
||||
|
||||
It is safe for clients to use both the old and new version before, during and
|
||||
after upgrading the objects to a new stored version.
|
||||
|
||||
Removing an old version:
|
||||
|
||||
1. Ensure all clients are fully migrated to the new version. The kube-apiserver
|
||||
logs can reviewed to help identify any clients that are still accessing via
|
||||
the old version.
|
||||
1. Set `served` to `false` for the old version in the `spec.versions` list. If
|
||||
any clients are still unexpectedly using the old version they may begin reporting
|
||||
errors attempting to access the custom resource objects at the old version.
|
||||
If this occurs, switch back to using `served:true` on the old version, migrate the
|
||||
remaining clients to the new version and repeat this step.
|
||||
1. Ensure the [upgrade of existing objects to the new stored version](#upgrade-existing-objects-to-a-new-stored-version) step has been completed.
|
||||
1. Verify that the `stored` is set to `true` for the new version in the `spec.versions` list in the CustomResourceDefinition.
|
||||
1. Verify that the old version is no longer listed in the CustomResourceDefinition `status.storedVersions`.
|
||||
1. Remove the old version from the CustomResourceDefinition `spec.versions` list.
|
||||
1. Drop conversion support for the old version in conversion webhooks.
|
||||
|
||||
## Specify multiple versions
|
||||
|
||||
The CustomResourceDefinition API `versions` field can be used to support multiple versions of custom resources that you
|
||||
have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions.
|
||||
Webhook conversions should follow the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md) wherever applicable.
|
||||
Specifically, See the [API change documentation](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) for a set of useful gotchas and suggestions.
|
||||
|
||||
@@ -42,8 +102,6 @@ In `apiextensions.k8s.io/v1beta1`, there was a `version` field instead of `versi
|
||||
match the first item in the `versions` field.
|
||||
{{< /note >}}
|
||||
|
||||
## Specify multiple versions
|
||||
|
||||
This example shows a CustomResourceDefinition with two versions. For the first
|
||||
example, the assumption is all versions share the same schema with no conversion
|
||||
between them. The comments in the YAML provide more context.
|
||||
@@ -885,16 +943,23 @@ can exist in storage at a version that has never been a storage version.
|
||||
|
||||
## Upgrade existing objects to a new stored version
|
||||
|
||||
When deprecating versions and dropping support, devise a storage upgrade
|
||||
procedure. The following is an example procedure to upgrade from `v1beta1`
|
||||
to `v1`.
|
||||
When deprecating versions and dropping support, select a storage upgrade
|
||||
procedure.
|
||||
|
||||
*Option 1:* Use the Storage Version Migrator
|
||||
|
||||
1. Run the [storage Version migrator](https://github.com/kubernetes-sigs/kube-storage-version-migrator)
|
||||
2. Remove the old version from the CustomResourceDefinition `status.storedVersions` field.
|
||||
|
||||
*Option 2:* Manually upgrade the existing objects to a new stored version
|
||||
|
||||
The following is an example procedure to upgrade from `v1beta1` to `v1`.
|
||||
|
||||
1. Set `v1` as the storage in the CustomResourceDefinition file and apply it
|
||||
using kubectl. The `storedVersions` is now `v1beta1, v1`.
|
||||
2. Write an upgrade procedure to list all existing objects and write them with
|
||||
the same content. This forces the backend to write objects in the current
|
||||
storage version, which is `v1`.
|
||||
3. Update the CustomResourceDefinition `Status` by removing `v1beta1` from
|
||||
`storedVersions` field.
|
||||
2. Remove `v1beta1` from the CustomResourceDefinition `status.storedVersions` field.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
Reference in New Issue
Block a user