whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CRD versioning Public Documentation (#8834)

Browse Source 
* CRD versioning Public Documentation

* Copyedit

Signed-off-by: Misty Stanley-Jones <mistyhacks@google.com>

* Address feedback

* More rewrites

* Address feedback

* Update main CRD page in light of versioning

* Reorg CRD docs

* Further reorg

* Tweak title
This commit is contained in:
Mehdy Bohlool
2018-06-18 10:18:33 -07:00
committed by Misty Linville
parent c4c4064194
commit 00b66727d5
9 changed files with 265 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/access-kubernetes-api/custom-resources/_index.md
Executable
+5
View File
@@ -0,0 +1,5 @@
---
title: "Use Custom Resources"
weight: 10
---
content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md
+201
View File
@@ -0,0 +1,201 @@
---
title: Versions of CustomResourceDefinitions
reviewers:
- mbohlool
- sttts
- liggitt
content_template: templates/task
weight: 30
---
{{% capture overview %}}
This page explains how to add versioning information to
[CustomResourceDefinitions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions), to indicate the stability
level of your CustomResourceDefinitions. It also describes how to upgrade an
object from one version to another.
{{< note >}}
**Note**: All specified versions must use the same schema. The is no schema
conversion between versions.
{{< /note >}}
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Make sure your Kubernetes cluster has a master version of 1.11.0 or higher.
* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
{{% /capture %}}
{{% capture steps %}}
## Overview
The CustomResourceDefinition API supports a `versions` field that you can use to
support multiple versions of custom resources that you have developed, and
indicate the stability of a given custom resource. All versions must currently
use the same schema, so if you need to add a field, you must add it to all
versions.
{{< note >}}
Earlier iterations included a `version` field instead of `versions`. The
`version` field is deprecated and optional, but if it is not empty, it must
match the first item in the `versions` field.
{{< /note >}}
## Specify multiple versions
This example shows a CustomResourceDefinition with two versions. The comments in
the YAML provide more context.
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1beta1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
- name: v1
served: true
storage: false
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
```
You can save the CustomResourceDefinition in a YAML file, then use
`kubectl create` to create it.
```shell
kubectl create -f my-versioned-crontab.yaml
```
After creation, the API server starts to serve each enabled version at an HTTP
REST endpoint. In the above example, the API versions are available at
`/apis/example.com/v1beta1` and `/apis/example.com/v1`.
### Version priority
Regardless of the order in which versions are defined in a
CustomResourceDefinition, the version with the highest priority is used by
kubectl as the default version to access objects. The priority is determined
by parsing the _name_ field to determine the version number, the stability
(GA, Beta, or Alpha), and the sequence within that stability level.
The algorithm used for sorting the versions is designed to sort versions in the
same way that the Kubernetes project sorts Kubernetes versions. Versions start with a
`v` followed by a number, an optional `beta` or `alpha` designation, and
optional additional versioning information. Broadly, a version string might look
like `v2` or `v2beta1`. Versions are sorted using the following algorithm:
- Entries that follow Kubernetes version patterns are sorted before those that
do not.
- For entries that follow Kubernetes version patterns, the numeric portions of
the version string is sorted largest to smallest.
- If the strings `beta` or `alpha` follow the first numeric portion, they sorted
in that order, after the equivalent string without the `beta` or `alpha`
suffix (which is presumed to be the GA version).
- If another number follows the `beta`, or `alpha`, those numbers are also
sorted from largest to smallest.
- Strings that don't fit the above format are sorted alphabetically and the
numeric portions are not treated specially. Notice that in the example below,
`foo1` is sorted above `foo10`. This is different from the sorting of the
numeric portion of entries that do follow the Kubernetes version patterns.
This might make sense if you look at the following sorted version list:
```none
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- v12alpha1
- v11alpha2
- foo1
- foo10
```
For the example in [Specify multiple versions](#specify-multiple-versions), the
version sort order is `v1`, followed by `v1beta1`. This causes the kubectl
command to use `v1` as the default version unless the provided object specifies
the version.
## Writing, reading, and updating versioned CustomResourceDefinition objects
When an object is written, it is persisted at the version designated as the
storage version at the time of the write. If the storage version changes,
existing objects are never converted automatically. However, newly-created
or updated objects are created at the new storage version. It is possible for an
object to have been written at a version that is no longer served.
When you read an object, you specify the version as part of the path. If you
specify a version that is different from the object's persisted version,
Kubernetes returns the object to you at the version you requested, but does not
modify the persisted object. You can request an object at any version that is
currently served.
If you update an existing object, it is rewritten at the version that is
currently the storage version. This is the only way that objects can change from
one version to another.
To illustrate this, consider the following hypothetical series of events:
1. The storage version is `v1beta`. You create an object. It is persisted in
storage at version `v1beta1`
2. You add version `v1` to your CustomResourceDefinition and designate it as
the storage version.
3. You read your object at version `v1beta`, then you read the object again at
version `v1`. Both returned objects are identical except for the apiVersion
field.
4. You create a new object. It is persisted in storage at version `v1`. You now
have two objects, one of which is at `v1beta`, and the other of which is at
`v1`.
5. You update the first object. It is now persisted at version `v1` since that
is the current storage version.
### Previous storage versions
The API server records each version which has ever been marked as the storage
version in the status field `storedVersions`. Objects may have been persisted
at any version that has ever been designated as a storage version. No objects
can exist in storage at a version that has never been a storage version.
## Upgrade existing objects to a new stored version
When deprecating versions and dropping support, devise a storage upgrade
procedure. The following is an example procedure to upgrade from `v1beta1`
to `v1`.
1. Set `v1` as the storage in the CustomResourceDefinition file and apply it
using kubectl. The `storedVersions` is now `v1beta1, v1`.
2. Write an upgrade procedure to list all existing objects and write them with
the same content. This forces the backend to write objects in the current
storage version, which is `v1`.
3. Update the CustomResourceDefinition `Status` by removing `v1beta1` from
`storedVersions` field.
{{% /capture %}}
content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md
+573
View File
@@ -0,0 +1,573 @@
---
title: Extend the Kubernetes API with CustomResourceDefinitions
reviewers:
- deads2k
- enisoc
content_template: templates/task
weight: 20
---
{{% capture overview %}}
This page shows how to install a
[custom resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
into the Kubernetes API by creating a
[CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Make sure your Kubernetes cluster has a master version of 1.7.0 or higher.
* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
{{% /capture %}}
{{% capture steps %}}
## Create a CustomResourceDefinition
When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server
creates a new RESTful resource path for each version you specify. The CRD can be
either namespaced or cluster-scoped, as specified in the CRD's `scope` field. As
with existing built-in objects, deleting a namespace deletes all custom objects
in that namespace. CustomResourceDefinitions themselves are non-namespaced and
are available to all namespaces.
For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
```
And create it:
```shell
kubectl create -f resourcedefinition.yaml
```
Then a new namespaced RESTful API endpoint is created at:
```
/apis/stable.example.com/v1/namespaces/*/crontabs/...
```
This endpoint URL can then be used to create and manage custom objects.
The `kind` of these objects will be `CronTab` from the spec of the
CustomResourceDefinition object you created above.
It might take a few seconds for the endpoint to be created.
You can watch the `Established` condition of your CustomResourceDefinition
to be true or watch the discovery information of the API server for your
resource to show up.
## Create custom objects
After the CustomResourceDefinition object has been created, you can create
custom objects. Custom objects can contain custom fields. These fields can
contain arbitrary JSON.
In the following example, the `cronSpec` and `image` custom fields are set in a
custom object of kind `CronTab`. The kind `CronTab` comes from the spec of the
CustomResourceDefinition object you created above.
If you save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
```
and create it:
```shell
kubectl create -f my-crontab.yaml
```
You can then manage your CronTab objects using kubectl. For example:
```shell
kubectl get crontab
```
Should print a list like this:
```console
NAME AGE
my-new-cron-object 6s
```
Resource names are not case-sensitive when using kubectl, and you can use either
the singular or plural forms defined in the CRD, as well as any short names.
You can also view the raw YAML data:
```shell
kubectl get ct -o yaml
```
You should see that it contains the custom `cronSpec` and `image` fields
from the yaml you used to create it:
```console
apiVersion: v1
items:
- apiVersion: stable.example.com/v1
kind: CronTab
metadata:
clusterName: ""
creationTimestamp: 2017-05-31T12:56:35Z
deletionGracePeriodSeconds: null
deletionTimestamp: null
name: my-new-cron-object
namespace: default
resourceVersion: "285"
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
uid: 9423255b-4600-11e7-af6a-28d2447dc82b
spec:
cronSpec: '* * * * */5'
image: my-awesome-cron-image
kind: List
metadata:
resourceVersion: ""
selfLink: ""
```
## Delete a CustomResourceDefinition
When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint
and **delete all custom objects stored in it**.
```shell
kubectl delete -f resourcedefinition.yaml
kubectl get crontabs
```
```console
Error from server (NotFound): Unable to list "crontabs": the server could not find the requested resource (get crontabs.stable.example.com)
```
If you later recreate the same CustomResourceDefinition, it will start out empty.
## Serving multiple versions of a CRD
See [Custom resource definition versioning](custom-resource-definition-versioning)
for more information about serving multiple versions of your
CustomResourceDefinition and migrating your objects from one version to another.
{{% /capture %}}
{{% capture discussion %}}
## Advanced topics
### Finalizers
*Finalizers* allow controllers to implement asynchronous pre-delete hooks.
Custom objects support finalizers just like built-in objects.
You can add a finalizer to a custom object like this:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
finalizers:
- finalizer.stable.example.com
```
Finalizers are arbitrary string values, that when present ensure that a hard delete
of a resource is not possible while they exist.
The first delete request on an object with finalizers merely sets a value for the
`metadata.deletionTimestamp` field instead of deleting it. Once this value is set,
entries in the `finalizer` list can only be removed.
This triggers controllers watching the object to execute any finalizers they handle.
This will be represented via polling update requests for that
object, until all finalizers have been removed and the resource is deleted.
The time period of polling update can be controlled by `metadata.deletionGracePeriodSeconds`.
It is the responsibility of each controller to removes its finalizer from the list.
Kubernetes will only finally delete the object if the list of finalizers is empty,
meaning all finalizers are done.
### Validation
Validation of custom objects is possible via
[OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject).
Additionally, the following restrictions are applied to the schema:
- The fields `default`, `nullable`, `discriminator`, `readOnly`, `writeOnly`, `xml`,
`deprecated` and `$ref` cannot be set.
- The field `uniqueItems` cannot be set to true.
- The field `additionalProperties` cannot be set to false.
This feature is __beta__ in v1.9.
You can disable this feature using the `CustomResourceValidation` feature gate on
the [kube-apiserver](/docs/admin/kube-apiserver):
```
--feature-gates=CustomResourceValidation=false
```
The schema is defined in the CustomResourceDefinition. In the following example, the
CustomResourceDefinition applies the following validations on the custom object:
- `spec.cronSpec` must be a string and must be of the form described by the regular expression.
- `spec.replicas` must be an integer and must have a minimum value of 1 and a maximum value of 10.
Save the CustomResourceDefinition to `resourcedefinition.yaml`:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
version: v1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
validation:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
properties:
spec:
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
replicas:
type: integer
minimum: 1
maximum: 10
```
And create it:
```shell
kubectl create -f resourcedefinition.yaml
```
A request to create a custom object of kind `CronTab` will be rejected if there are invalid values in its fields.
In the following example, the custom object contains fields with invalid values:
- `spec.cronSpec` does not match the regular expression.
- `spec.replicas` is greater than 10.
If you save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * *"
image: my-awesome-cron-image
replicas: 15
```
and create it:
```shell
kubectl create -f my-crontab.yaml
```
you will get an error:
```console
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10
```
If the fields contain valid values, the object creation request is accepted.
Save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 5
```
And create it:
```shell
kubectl create -f my-crontab.yaml
crontab "my-new-cron-object" created
```
### Subresources
Custom resources support `/status` and `/scale` subresources.
This feature is __beta__ in v1.11 and enabled by default.
You can disable this feature using the `CustomResourceSubresources` feature gate on
the [kube-apiserver](/docs/admin/kube-apiserver):
```
--feature-gates=CustomResourceSubresources=false
```
The status and scale subresources can be optionally enabled by
defining them in the CustomResourceDefinition.
#### Status subresource
When the status subresource is enabled, the `/status` subresource for the custom resource is exposed.
- The status and the spec stanzas are represented by the `.status` and `.spec` JSONPaths respectively inside of a custom resource.
- `PUT` requests to the `/status` subresource take a custom resource object and ignore changes to anything except the status stanza.
- `PUT` requests to the `/status` subresource only validate the status stanza of the custom resource.
- `PUT`/`POST`/`PATCH` requests to the custom resource ignore changes to the status stanza.
- Any changes to the spec stanza increments the value at `.metadata.generation`.
- `properties`, `required` and `description` are the only constructs allowed in the root of the CRD OpenAPI validation schema.
#### Scale subresource
When the scale subresource is enabled, the `/scale` subresource for the custom resource is exposed.
The `autoscaling/v1.Scale` object is sent as the payload for `/scale`.
To enable the scale subresource, the following values are defined in the CustomResourceDefinition.
- `SpecReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Spec.Replicas`.
- It is a required value.
- Only JSONPaths under `.spec` and with the dot notation are allowed.
- If there is no value under the `SpecReplicasPath` in the custom resource,
the `/scale` subresource will return an error on GET.
- `StatusReplicasPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Status.Replicas`.
- It is a required value.
- Only JSONPaths under `.status` and with the dotation are allowed.
- If there is no value under the `StatusReplicasPath` in the custom resource,
the status replica value in the `/scale` subresource will default to 0.
- `LabelSelectorPath` defines the JSONPath inside of a custom resource that corresponds to `Scale.Status.Selector`.
- It is an optional value.
- It must be set to work with HPA.
- Only JSONPaths under `.status` and with the dotation are allowed.
- If there is no value under the `LabelSelectorPath` in the custom resource,
the status selector value in the `/scale` subresource will default to the empty string.
In the following example, both status and scale subresources are enabled.
Save the CustomResourceDefinition to `resourcedefinition.yaml`:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
# subresources describes the subresources for custom resources.
subresources:
# status enables the status subresource.
status: {}
# scale enables the scale subresource.
scale:
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
specReplicasPath: .spec.replicas
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
statusReplicasPath: .status.replicas
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
labelSelectorPath: .status.labelSelector
```
And create it:
```shell
kubectl create -f resourcedefinition.yaml
```
After the CustomResourceDefinition object has been created, you can create custom objects.
If you save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 3
```
and create it:
```shell
kubectl create -f my-crontab.yaml
```
Then new namespaced RESTful API endpoints are created at:
```
/apis/stable.example.com/v1/namespaces/*/crontabs/status
```
and
```
/apis/stable.example.com/v1/namespaces/*/crontabs/scale
```
A custom resource can be scaled using the `kubectl scale` command.
For example, the following command sets `.spec.replicas` of the
custom resource created above to 5:
```shell
kubectl scale --replicas=5 crontabs/my-new-cron-object
crontabs "my-new-cron-object" scaled
kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
5
```
### Categories
Categories is a list of grouped resources the custom resource belongs to (eg. `all`).
You can use `kubectl get <category-name>` to list the resources belonging to the category.
This feature is __beta__ and available for custom resources from v1.10.
The following example adds `all` in the list of categories in the CustomResourceDefinition
and illustrates how to output the custom resource using `kubectl get all`.
Save the following CustomResourceDefinition to `resourcedefinition.yaml`:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
# categories is a list of grouped resources the custom resource belongs to.
categories:
- all
```
And create it:
```shell
kubectl create -f resourcedefinition.yaml
```
After the CustomResourceDefinition object has been created, you can create custom objects.
Save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
```
and create it:
```shell
kubectl create -f my-crontab.yaml
```
You can specify the category using `kubectl get`:
```
kubectl get all
```
and it will include the custom resources of kind `CronTab`:
```console
NAME AGE
crontabs/my-new-cron-object 3s
```
{{% /capture %}}
{{% capture whatsnext %}}
* Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions-k8s-io).
* Serve [multiple versions](custom-resource-definitions-versioning) of a
CustomResourceDefinition
{{% /capture %}}
content/en/docs/tasks/access-kubernetes-api/custom-resources/migrate-third-party-resource.md
+174
View File
@@ -0,0 +1,174 @@
---
title: Migrate a ThirdPartyResource to CustomResourceDefinition
reviewers:
- enisoc
- deads2k
content_template: templates/task
weight: 50
---
{{% capture overview %}}
This page shows how to migrate data stored in a ThirdPartyResource (TPR) to a
[CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions) (CRD).
Kubernetes does not automatically migrate existing TPRs.
This is due to API changes introduced as part of
[graduating to beta](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/thirdpartyresources.md)
under a new name and API group.
Instead, both TPR and CRD are available and operate independently in Kubernetes 1.7.
Users must migrate each TPR one by one to preserve their data before upgrading to Kubernetes 1.8.
The simplest way to migrate is to stop all clients that use a given TPR, then delete the TPR and
start from scratch with a CRD.
This page describes an optional process that eases the transition by migrating existing TPR data for
you **on a best-effort basis**.
{{% /capture %}}
{{% capture prerequisites %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Make sure your Kubernetes cluster has a **master version of exactly 1.7.x** (any patch release),
as this is the only version that supports both TPR and CRD.
* If you use a TPR-based custom controller, check with the author of the controller first.
Some or all of these steps may be unnecessary if the custom controller handles the migration for
you.
* Be familiar with the concept of [custom resources](/docs/concepts/api-extension/custom-resources/),
which were known as *third-party resources* until Kubernetes 1.7.
* Be familiar with [CustomResourceDefinitions](/docs/concepts/api-extension/custom-resources/#customresourcedefinitions),
which are a simple way to implement custom resources.
* **Before performing a migration on real data, conduct a dry run by going through these steps in a test cluster.**
{{% /capture %}}
{{% capture steps %}}
## Migrate TPR data
1. **Rewrite the TPR definition**
Clients that access the REST API for your custom resource should not need any changes.
However, you will need to rewrite your TPR definition as a CRD.
Make sure you specify values for the CRD fields that match what the server used to fill in for
you with TPR.
For example, if your ThirdPartyResource looks like this:
```yaml
apiVersion: extensions/v1beta1
kind: ThirdPartyResource
metadata:
name: cron-tab.stable.example.com
description: "A specification of a Pod to run on a cron style schedule"
versions:
- name: v1
```
A matching CustomResourceDefinition could look like this:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
scope: Namespaced
group: stable.example.com
versions:
- name: v1
served: true
storage: true
names:
kind: CronTab
plural: crontabs
singular: crontab
```
1. **Install the CustomResourceDefinition**
While the source TPR is still active, install the matching CRD with `kubectl create`.
Existing TPR data remains accessible because TPRs take precedence over CRDs when both try
to serve the same resource.
After you create the CRD, make sure the *Established* condition goes to True.
You can check it with a command like this:
```shell
kubectl get crd -o 'custom-columns=NAME:{.metadata.name},ESTABLISHED:{.status.conditions[?(@.type=="Established")].status}'
```
The output should look like this:
```console
NAME ESTABLISHED
crontabs.stable.example.com True
```
1. **Stop all clients that use the TPR**
The API server attempts to prevent TPR data for the resource from changing while it
copies objects to the CRD, but it can't guarantee consistency in all cases, such as with
[multiple masters](/docs/admin/high-availability/).
Stopping clients, such as TPR-based custom controllers, helps to avoid inconsistencies in
the copied data.
In addition, clients that watch TPR data do not receive any more events once the migration
begins.
You must restart them after the migration completes so they start watching CRD data instead.
1. **Back up TPR data**
In case the data migration fails, save a copy of existing data for the resource:
```shell
kubectl get crontabs --all-namespaces -o yaml > crontabs.yaml
```
You should also save a copy of the TPR definition if you don't have one already:
```shell
kubectl get thirdpartyresource cron-tab.stable.example.com -o yaml --export > tpr.yaml
```
1. **Delete the TPR definition**
Normally, when you delete a TPR definition, the API server tries to clean up any objects stored
in that resource.
Because a matching CRD exists, the server copies objects to the CRD instead of deleting them.
```shell
kubectl delete thirdpartyresource cron-tab.stable.example.com
```
1. **Verify the new CRD data**
It can take up to 10 seconds for the TPR controller to notice when you delete the TPR definition
and to initiate the migration. The TPR data remains accessible during this time.
Once the migration completes, the resource begins serving through the CRD.
Check that all your objects were correctly copied:
```shell
kubectl get crontabs --all-namespaces -o yaml
```
If the copy failed, you can quickly revert to the set of objects that existed just before the
migration by recreating the TPR definition:
```shell
kubectl create -f tpr.yaml
```
1. **Restart clients**
After verifying the CRD data, restart any clients you stopped before the migration, such as
custom controllers and other watchers.
These clients now access CRD data when they make requests on the same API endpoints
that the TPR previously served.
{{% /capture %}}
{{% capture whatsnext %}}
* Learn more about [custom resources](/docs/concepts/api-extension/custom-resources/).
* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/).
* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions).
{{% /capture %}}