Add warnings blog post (#23592)
This commit is contained in:
Jordan Liggitt
@@ -0,0 +1,278 @@
|
||||
---
|
||||
layout: blog
|
||||
title: "Warning: Helpful Warnings Ahead"
|
||||
date: 2020-09-03
|
||||
slug: warnings
|
||||
---
|
||||
|
||||
**Author**: Jordan Liggitt (Google)
|
||||
|
||||
As Kubernetes maintainers, we're always looking for ways to improve usability while preserving compatibility.
|
||||
As we develop features, triage bugs, and answer support questions, we accumulate information that would be helpful for Kubernetes users to know.
|
||||
In the past, sharing that information was limited to out-of-band methods like release notes, announcement emails, documentation, and blog posts.
|
||||
Unless someone knew to seek out that information and managed to find it, they would not benefit from it.
|
||||
|
||||
In Kubernetes v1.19, we added a feature that allows the Kubernetes API server to
|
||||
[send warnings to API clients](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1693-warnings).
|
||||
The warning is sent using a [standard `Warning` response header](https://tools.ietf.org/html/rfc7234#section-5.5),
|
||||
so it does not change the status code or response body in any way.
|
||||
This allows the server to send warnings easily readable by any API client, while remaining compatible with previous client versions.
|
||||
|
||||
Warnings are surfaced by `kubectl` v1.19+ in `stderr` output, and by the `k8s.io/client-go` client library v0.19.0+ in log output.
|
||||
The `k8s.io/client-go` behavior can be overridden [per-process](https://godoc.org/k8s.io/client-go/rest#SetDefaultWarningHandler)
|
||||
or [per-client](https://godoc.org/k8s.io/client-go/rest#Config).
|
||||
|
||||
## Deprecation Warnings
|
||||
|
||||
The first way we are using this new capability is to send warnings for use of deprecated APIs.
|
||||
|
||||
Kubernetes is a [big, fast-moving project](https://www.cncf.io/cncf-kubernetes-project-journey/#development-velocity).
|
||||
Keeping up with the [changes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#changelog-since-v1180)
|
||||
in each release can be daunting, even for people who work on the project full-time. One important type of change is API deprecations.
|
||||
As APIs in Kubernetes graduate to GA versions, pre-release API versions are deprecated and eventually removed.
|
||||
|
||||
Even though there is an [extended deprecation period](/docs/reference/using-api/deprecation-policy/),
|
||||
and deprecations are [included in release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#deprecation),
|
||||
they can still be hard to track. During the deprecation period, the pre-release API remains functional,
|
||||
allowing several releases to transition to the stable API version. However, we have found that users often don't even realize
|
||||
they are depending on a deprecated API version until they upgrade to the release that stops serving it.
|
||||
|
||||
Starting in v1.19, whenever a request is made to a deprecated REST API, a warning is returned along with the API response.
|
||||
This warning includes details about the release in which the API will no longer be available, and the replacement API version.
|
||||
|
||||
Because the warning originates at the server, and is intercepted at the client level, it works for all kubectl commands,
|
||||
including high-level commands like `kubectl apply`, and low-level commands like `kubectl get --raw`:
|
||||
|
||||
<img alt="kubectl applying a manifest file, then displaying a warning message 'networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress'."
|
||||
src="kubectl-warnings.png"
|
||||
style="width:637px;max-width:100%;">
|
||||
|
||||
This helps people affected by the deprecation to know the request they are making is deprecated,
|
||||
how long they have to address the issue, and what API they should use instead.
|
||||
This is especially helpful when the user is applying a manifest they didn't create,
|
||||
so they have time to reach out to the authors to ask for an updated version.
|
||||
|
||||
We also realized that the person *using* a deprecated API is often not the same person responsible for upgrading the cluster,
|
||||
so we added two administrator-facing tools to help track use of deprecated APIs and determine when upgrades are safe.
|
||||
|
||||
### Metrics
|
||||
|
||||
Starting in Kubernetes v1.19, when a request is made to a deprecated REST API endpoint,
|
||||
an `apiserver_requested_deprecated_apis` gauge metric is set to `1` in the kube-apiserver process.
|
||||
This metric has labels for the API `group`, `version`, `resource`, and `subresource`,
|
||||
and a `removed_version` label that indicates the Kubernetes release in which the API will no longer be served.
|
||||
|
||||
This is an example query using `kubectl`, [prom2json](https://github.com/prometheus/prom2json),
|
||||
and [jq](https://stedolan.github.io/jq/) to determine which deprecated APIs have been requested
|
||||
from the current instance of the API server:
|
||||
|
||||
```sh
|
||||
kubectl get --raw /metrics | prom2json | jq '
|
||||
.[] | select(.name=="apiserver_requested_deprecated_apis").metrics[].labels
|
||||
'
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```json
|
||||
{
|
||||
"group": "extensions",
|
||||
"removed_release": "1.22",
|
||||
"resource": "ingresses",
|
||||
"subresource": "",
|
||||
"version": "v1beta1"
|
||||
}
|
||||
{
|
||||
"group": "rbac.authorization.k8s.io",
|
||||
"removed_release": "1.22",
|
||||
"resource": "clusterroles",
|
||||
"subresource": "",
|
||||
"version": "v1beta1"
|
||||
}
|
||||
```
|
||||
|
||||
This shows the deprecated `extensions/v1beta1` Ingress and `rbac.authorization.k8s.io/v1beta1` ClusterRole APIs
|
||||
have been requested on this server, and will be removed in v1.22.
|
||||
|
||||
We can join that information with the `apiserver_request_total` metrics to get more details about the requests being made to these APIs:
|
||||
|
||||
```sh
|
||||
kubectl get --raw /metrics | prom2json | jq '
|
||||
# set $deprecated to a list of deprecated APIs
|
||||
[
|
||||
.[] |
|
||||
select(.name=="apiserver_requested_deprecated_apis").metrics[].labels |
|
||||
{group,version,resource}
|
||||
] as $deprecated
|
||||
|
||||
|
|
||||
|
||||
# select apiserver_request_total metrics which are deprecated
|
||||
.[] | select(.name=="apiserver_request_total").metrics[] |
|
||||
select(.labels | {group,version,resource} as $key | $deprecated | index($key))
|
||||
'
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```json
|
||||
{
|
||||
"labels": {
|
||||
"code": "0",
|
||||
"component": "apiserver",
|
||||
"contentType": "application/vnd.kubernetes.protobuf;stream=watch",
|
||||
"dry_run": "",
|
||||
"group": "extensions",
|
||||
"resource": "ingresses",
|
||||
"scope": "cluster",
|
||||
"subresource": "",
|
||||
"verb": "WATCH",
|
||||
"version": "v1beta1"
|
||||
},
|
||||
"value": "21"
|
||||
}
|
||||
{
|
||||
"labels": {
|
||||
"code": "200",
|
||||
"component": "apiserver",
|
||||
"contentType": "application/vnd.kubernetes.protobuf",
|
||||
"dry_run": "",
|
||||
"group": "extensions",
|
||||
"resource": "ingresses",
|
||||
"scope": "cluster",
|
||||
"subresource": "",
|
||||
"verb": "LIST",
|
||||
"version": "v1beta1"
|
||||
},
|
||||
"value": "1"
|
||||
}
|
||||
{
|
||||
"labels": {
|
||||
"code": "200",
|
||||
"component": "apiserver",
|
||||
"contentType": "application/json",
|
||||
"dry_run": "",
|
||||
"group": "rbac.authorization.k8s.io",
|
||||
"resource": "clusterroles",
|
||||
"scope": "cluster",
|
||||
"subresource": "",
|
||||
"verb": "LIST",
|
||||
"version": "v1beta1"
|
||||
},
|
||||
"value": "1"
|
||||
}
|
||||
```
|
||||
|
||||
The output shows that only read requests are being made to these APIs, and the most requests have been made to watch the deprecated Ingress API.
|
||||
|
||||
You can also find that information through the following Prometheus query,
|
||||
which returns information about requests made to deprecated APIs which will be removed in v1.22:
|
||||
|
||||
```promql
|
||||
apiserver_requested_deprecated_apis{removed_version="1.22"} * on(group,version,resource,subresource)
|
||||
group_right() apiserver_request_total
|
||||
```
|
||||
|
||||
### Audit annotations
|
||||
|
||||
Metrics are a fast way to check whether deprecated APIs are being used, and at what rate,
|
||||
but they don't include enough information to identify particular clients or API objects.
|
||||
Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/)
|
||||
for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`.
|
||||
Administrators can use those audit events to identify specific clients or objects that need to be updated.
|
||||
|
||||
## Custom Resource Definitions
|
||||
|
||||
Along with the API server ability to warn about deprecated API use, starting in v1.19, a CustomResourceDefinition can indicate a
|
||||
[particular version of the resource it defines is deprecated](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation).
|
||||
When API requests to a deprecated version of a custom resource are made, a warning message is returned, matching the behavior of built-in APIs.
|
||||
|
||||
The author of the CustomResourceDefinition can also customize the warning for each version if they want to.
|
||||
This allows them to give a pointer to a migration guide or other information if needed.
|
||||
|
||||
```yaml
|
||||
apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
name: crontabs.example.com
|
||||
spec:
|
||||
versions:
|
||||
- name: v1alpha1
|
||||
# This indicates the v1alpha1 version of the custom resource is deprecated.
|
||||
# API requests to this version receive a warning in the server response.
|
||||
deprecated: true
|
||||
# This overrides the default warning returned to clients making v1alpha1 API requests.
|
||||
deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; use example.com/v1 CronTab (see http://example.com/v1alpha1-v1)"
|
||||
...
|
||||
|
||||
- name: v1beta1
|
||||
# This indicates the v1beta1 version of the custom resource is deprecated.
|
||||
# API requests to this version receive a warning in the server response.
|
||||
# A default warning message is returned for this version.
|
||||
deprecated: true
|
||||
...
|
||||
|
||||
- name: v1
|
||||
...
|
||||
```
|
||||
|
||||
## Admission Webhooks
|
||||
|
||||
[Admission webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers)
|
||||
are the primary way to integrate custom policies or validation with Kubernetes.
|
||||
Starting in v1.19, admission webhooks can [return warning messages](/docs/reference/access-authn-authz/extensible-admission-controllers/#response)
|
||||
that are passed along to the requesting API client. Warnings can be returned with allowed or rejected admission responses.
|
||||
|
||||
As an example, to allow a request but warn about a configuration known not to work well, an admission webhook could send this response:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "admission.k8s.io/v1",
|
||||
"kind": "AdmissionReview",
|
||||
"response": {
|
||||
"uid": "<value from request.uid>",
|
||||
"allowed": true,
|
||||
"warnings": [
|
||||
".spec.memory: requests >1GB do not work on Fridays"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If you are implementing a webhook that returns a warning message, here are some tips:
|
||||
|
||||
* Don't include a "Warning:" prefix in the message (that is added by clients on output)
|
||||
* Use warning messages to describe problems the client making the API request should correct or be aware of
|
||||
* Be brief; limit warnings to 120 characters if possible
|
||||
|
||||
There are many ways admission webhooks could use this new feature, and I'm looking forward to seeing what people come up with.
|
||||
Here are a couple ideas to get you started:
|
||||
|
||||
* webhook implementations adding a "complain" mode, where they return warnings instead of rejections,
|
||||
to allow trying out a policy to verify it is working as expected before starting to enforce it
|
||||
* "lint" or "vet"-style webhooks, inspecting objects and surfacing warnings when best practices are not followed
|
||||
|
||||
## Kubectl strict mode
|
||||
|
||||
If you want to be sure you notice deprecations as soon as possible and get a jump start on addressing them,
|
||||
`kubectl` added a `--warnings-as-errors` option in v1.19. When invoked with this option,
|
||||
`kubectl` treats any warnings it receives from the server as errors and exits with a non-zero exit code:
|
||||
|
||||
<img alt="kubectl applying a manifest file with a --warnings-as-errors flag, displaying a warning message and exiting with a non-zero exit code."
|
||||
src="kubectl-warnings-as-errors.png"
|
||||
style="width:637px;max-width:100%;">
|
||||
|
||||
This could be used in a CI job to apply manifests to a current server,
|
||||
and required to pass with a zero exit code in order for the CI job to succeed.
|
||||
|
||||
## Future Possibilities
|
||||
|
||||
Now that we have a way to communicate helpful information to users in context,
|
||||
we're already considering other ways we can use this to improve people's experience with Kubernetes.
|
||||
A couple areas we're looking at next are warning about [known problematic values](http://issue.k8s.io/64841#issuecomment-395141013)
|
||||
we cannot reject outright for compatibility reasons, and warning about use of deprecated fields or field values
|
||||
(like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/kubernetes-api/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)).
|
||||
I'm excited to see progress in this area, continuing to make it easier to use Kubernetes.
|
||||
|
||||
---
|
||||
|
||||
_[Jordan Liggitt](https://twitter.com/liggitt) is a software engineer at Google, and helps lead Kubernetes authentication, authorization, and API efforts._
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 296 KiB |
Reference in New Issue
Block a user