Update task for DaemonSet history and rollback (#4098)
* Update task for DaemonSet history and rollback Also remove mentions of templateGeneration field because it's deprecated * Address comments
This commit is contained in:
@@ -148,6 +148,7 @@ toc:
|
|||||||
- title: Manage Cluster Daemons
|
- title: Manage Cluster Daemons
|
||||||
section:
|
section:
|
||||||
- docs/tasks/manage-daemon/update-daemon-set.md
|
- docs/tasks/manage-daemon/update-daemon-set.md
|
||||||
|
- docs/tasks/manage-daemon/rollback-daemon-set.md
|
||||||
|
|
||||||
- title: Manage GPUs
|
- title: Manage GPUs
|
||||||
section:
|
section:
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
---
|
||||||
|
assignees:
|
||||||
|
- janetkuo
|
||||||
|
title: Performing a Rollback on a DaemonSet
|
||||||
|
---
|
||||||
|
|
||||||
|
{% capture overview %}
|
||||||
|
|
||||||
|
This page shows how to perform a rollback on a DaemonSet.
|
||||||
|
|
||||||
|
{% endcapture %}
|
||||||
|
|
||||||
|
|
||||||
|
{% capture prerequisites %}
|
||||||
|
|
||||||
|
* The DaemonSet rollout history and DaemonSet rollback features are only
|
||||||
|
supported in `kubectl` in Kubernetes version 1.7 or later.
|
||||||
|
* Make sure you know how to [perform a rolling update on a
|
||||||
|
DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/)
|
||||||
|
|
||||||
|
{% endcapture %}
|
||||||
|
|
||||||
|
|
||||||
|
{% capture steps %}
|
||||||
|
|
||||||
|
## Performing a Rollback on a DaemonSet
|
||||||
|
|
||||||
|
### Step 1: Find the DaemonSet revision you want to roll back to
|
||||||
|
|
||||||
|
You can skip this step if you just want to roll back to the last revision.
|
||||||
|
|
||||||
|
List all revisions of a DaemonSet:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl rollout history daemonset <daemonset-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns a list of DaemonSet revisions:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
daemonsets "<daemonset-name>"
|
||||||
|
REVISION CHANGE-CAUSE
|
||||||
|
1 ...
|
||||||
|
2 ...
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
* Change cause is copied from DaemonSet annotation `kubernetes.io/change-cause`
|
||||||
|
to its revisions upon creation. You may specify `--record=true` in `kubectl`
|
||||||
|
to record the command executed in the change cause annotation.
|
||||||
|
|
||||||
|
To see the details of a specific revision:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl rollout history daemonset <daemonset-name> --revision=1
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns the details of that revision:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
daemonsets "<daemonset-name>" with revision #1
|
||||||
|
Pod Template:
|
||||||
|
Labels: foo=bar
|
||||||
|
Containers:
|
||||||
|
app:
|
||||||
|
Image: ...
|
||||||
|
Port: ...
|
||||||
|
Environment: ...
|
||||||
|
Mounts: ...
|
||||||
|
Volumes: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Roll back to a specific revision
|
||||||
|
|
||||||
|
```shell
|
||||||
|
# Specify the revision number you get from Step 1 in --to-revision
|
||||||
|
kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision>
|
||||||
|
```
|
||||||
|
|
||||||
|
If it succeeds, the command returns:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
daemonset "<daemonset-name>" rolled back
|
||||||
|
```
|
||||||
|
|
||||||
|
If `--to-revision` flag is not specified, the last revision will be picked.
|
||||||
|
|
||||||
|
### Step 3: Watch the progress of the DaemonSet rollback
|
||||||
|
|
||||||
|
`kubectl rollout undo daemonset` tells the server to start rolling back the
|
||||||
|
DaemonSet. The real rollback is done asynchronously on the server side.
|
||||||
|
|
||||||
|
To watch the progress of the rollback:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl rollout status ds/<daemonset-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
When the rollback is complete, the output is similar to this:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
daemon set "<daemonset-name>" successfully rolled out
|
||||||
|
```
|
||||||
|
|
||||||
|
{% endcapture %}
|
||||||
|
|
||||||
|
|
||||||
|
{% capture discussion %}
|
||||||
|
|
||||||
|
## Understanding DaemonSet Revisions
|
||||||
|
|
||||||
|
In the previous `kubectl rollout history` step, you got a list of DaemonSet
|
||||||
|
revisions. Each revision is stored in a resource named `ControllerRevision`.
|
||||||
|
`ControllerRevision` is a resource only available in Kubernetes release 1.7 or
|
||||||
|
later.
|
||||||
|
|
||||||
|
To see what is stored in each revision, find the DaemonSet revision raw
|
||||||
|
resources:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
kubectl get controllerrevision -l <daemonset-selector-key>=<daemonset-selector-value>
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns a list of `ControllerRevisions`:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
NAME CONTROLLER REVISION AGE
|
||||||
|
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 1 1h
|
||||||
|
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 2 1h
|
||||||
|
```
|
||||||
|
|
||||||
|
Each `ControllerRevision` stores the annotations and template of a DaemonSet
|
||||||
|
revision.
|
||||||
|
|
||||||
|
`kubectl rollout undo` takes a specific `ControllerRevision` and replaces
|
||||||
|
DaemonSet template with the template stored in the `ControllerRevision`.
|
||||||
|
`kubectl rollout undo` is equivalent to updating DaemonSet template to a
|
||||||
|
previous revision through other commands, such as `kubectl edit` or `kubectl
|
||||||
|
apply`.
|
||||||
|
|
||||||
|
Note that DaemonSet revisions only roll forward. That is to say, after a
|
||||||
|
rollback is complete, the revision number (`.revision` field) of the
|
||||||
|
`ControllerRevision` being rolled back to will advance. For example, if you
|
||||||
|
have revision 1 and 2 in the system, and roll back from revision 2 to revision
|
||||||
|
1, the `ControllerRevision` with `.revision: 1` will become `.revision: 3`.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
* See [troubleshooting DaemonSet rolling
|
||||||
|
update](/docs/tasks/manage-daemon/update-daemon-set/#troubleshooting)
|
||||||
|
|
||||||
|
{% endcapture %}
|
||||||
|
|
||||||
|
{% include templates/task.md %}
|
||||||
@@ -33,46 +33,18 @@ DaemonSet has two update strategy types :
|
|||||||
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
|
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
|
||||||
will be created automatically, in a controlled fashion.
|
will be created automatically, in a controlled fashion.
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
* DaemonSet rollout history is not supported yet.
|
|
||||||
* DaemonSet rollback is not directly supported in `kubectl` yet. You can rollback
|
|
||||||
by updating DaemonSet template to match the previous version.
|
|
||||||
|
|
||||||
## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before
|
## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before
|
||||||
|
|
||||||
If you try to rolling update a DaemonSet that was created from Kubernetes
|
If you try a rolling update on a DaemonSet that was created from Kubernetes
|
||||||
version 1.5 or before, a rollout will be triggered when you *first* change the
|
version 1.5 or before, a rollout will be triggered when you *first* change the
|
||||||
DaemonSet update strategy to `RollingUpdate`, even when DaemonSet template isn't
|
DaemonSet update strategy to `RollingUpdate`, no matter if DaemonSet template is
|
||||||
modified. All existing DaemonSet pods will be restarted.
|
modified or not. If the DaemonSet template is not changed, all existing DaemonSet
|
||||||
|
pods will be restarted (deleted and created).
|
||||||
|
|
||||||
To avoid this restart, first find the DaemonSet's current
|
Therefore, make sure you want to trigger a rollout before you first switch the
|
||||||
`.spec.templateGeneration`:
|
strategy to `RollingUpdate`.
|
||||||
|
|
||||||
```shell{% raw %}
|
## Performing a Rolling Update
|
||||||
kubectl get ds/<daemonset-name> -o go-template='{{.spec.templateGeneration}}{{"\n"}}'
|
|
||||||
{% endraw %}```
|
|
||||||
|
|
||||||
The output should be a number *N*. If the output shows `<no value>`, N = 0.
|
|
||||||
|
|
||||||
Then, simply label the existing DaemonSet pods with `pod-template-generation=<N>`
|
|
||||||
before changing DaemonSet `.spec.updateStrategy` to `RollingUpdate`:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# Replace N with DaemonSet `.spec.templateGeneration`
|
|
||||||
# Only run this on pods created from current DaemonSet template
|
|
||||||
kubectl label pods -l <daemonset-selector-key>=<daemonset-selector-value> pod-template-generation=<N>
|
|
||||||
```
|
|
||||||
|
|
||||||
This tells the DaemonSet that the labeled DaemonSet pods are created from current
|
|
||||||
DaemonSet template. Therefore, you should only run this command to existing DaemonSet
|
|
||||||
pods that are generated from current DaemonSet template.
|
|
||||||
|
|
||||||
Note that you only need to do this when you first change the update strategy of
|
|
||||||
a DaemonSet created from Kubernetes version 1.5 or before to `RollingUpdate`,
|
|
||||||
but don't want to update its template and start a new rollout yet.
|
|
||||||
|
|
||||||
## Setting DaemonSet update strategy for rolling update
|
|
||||||
|
|
||||||
To enable the rolling update feature of a DaemonSet, you must set its
|
To enable the rolling update feature of a DaemonSet, you must set its
|
||||||
`.spec.updateStrategy.type` to `RollingUpdate`.
|
`.spec.updateStrategy.type` to `RollingUpdate`.
|
||||||
@@ -87,15 +59,15 @@ First, check the update strategy of your DaemonSet, and make sure it's set to
|
|||||||
RollingUpdate:
|
RollingUpdate:
|
||||||
|
|
||||||
```shell{% raw %}
|
```shell{% raw %}
|
||||||
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'{% endraw %}
|
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
|
||||||
```
|
``` {% endraw %}
|
||||||
|
|
||||||
If you haven't created the DaemonSet in the system, check your DaemonSet
|
If you haven't created the DaemonSet in the system, check your DaemonSet
|
||||||
manifest with the following command instead:
|
manifest with the following command instead:
|
||||||
|
|
||||||
```shell{% raw %}
|
```shell{% raw %}
|
||||||
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
|
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
|
||||||
{% endraw %}```
|
``` {% endraw %}
|
||||||
|
|
||||||
The output from both commands should be:
|
The output from both commands should be:
|
||||||
|
|
||||||
@@ -224,7 +196,9 @@ progress.
|
|||||||
|
|
||||||
{% capture whatsnext %}
|
{% capture whatsnext %}
|
||||||
|
|
||||||
*TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
|
* See [Task: Performing a rollback on a
|
||||||
|
DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/)
|
||||||
|
* *TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
|
||||||
|
|
||||||
{% endcapture %}
|
{% endcapture %}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user