diff --git a/content/en/blog/_posts/2020-09-03-warnings/index.md b/content/en/blog/_posts/2020-09-03-warnings/index.md new file mode 100644 index 0000000000..2e93d5a573 --- /dev/null +++ b/content/en/blog/_posts/2020-09-03-warnings/index.md @@ -0,0 +1,278 @@ +--- +layout: blog +title: "Warning: Helpful Warnings Ahead" +date: 2020-09-03 +slug: warnings +--- + +**Author**: Jordan Liggitt (Google) + +As Kubernetes maintainers, we're always looking for ways to improve usability while preserving compatibility. +As we develop features, triage bugs, and answer support questions, we accumulate information that would be helpful for Kubernetes users to know. +In the past, sharing that information was limited to out-of-band methods like release notes, announcement emails, documentation, and blog posts. +Unless someone knew to seek out that information and managed to find it, they would not benefit from it. + +In Kubernetes v1.19, we added a feature that allows the Kubernetes API server to +[send warnings to API clients](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1693-warnings). +The warning is sent using a [standard `Warning` response header](https://tools.ietf.org/html/rfc7234#section-5.5), +so it does not change the status code or response body in any way. +This allows the server to send warnings easily readable by any API client, while remaining compatible with previous client versions. + +Warnings are surfaced by `kubectl` v1.19+ in `stderr` output, and by the `k8s.io/client-go` client library v0.19.0+ in log output. +The `k8s.io/client-go` behavior can be overridden [per-process](https://godoc.org/k8s.io/client-go/rest#SetDefaultWarningHandler) +or [per-client](https://godoc.org/k8s.io/client-go/rest#Config). + +## Deprecation Warnings + +The first way we are using this new capability is to send warnings for use of deprecated APIs. + +Kubernetes is a [big, fast-moving project](https://www.cncf.io/cncf-kubernetes-project-journey/#development-velocity). +Keeping up with the [changes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#changelog-since-v1180) +in each release can be daunting, even for people who work on the project full-time. One important type of change is API deprecations. +As APIs in Kubernetes graduate to GA versions, pre-release API versions are deprecated and eventually removed. + +Even though there is an [extended deprecation period](/docs/reference/using-api/deprecation-policy/), +and deprecations are [included in release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#deprecation), +they can still be hard to track. During the deprecation period, the pre-release API remains functional, +allowing several releases to transition to the stable API version. However, we have found that users often don't even realize +they are depending on a deprecated API version until they upgrade to the release that stops serving it. + +Starting in v1.19, whenever a request is made to a deprecated REST API, a warning is returned along with the API response. +This warning includes details about the release in which the API will no longer be available, and the replacement API version. + +Because the warning originates at the server, and is intercepted at the client level, it works for all kubectl commands, +including high-level commands like `kubectl apply`, and low-level commands like `kubectl get --raw`: + +kubectl applying a manifest file, then displaying a warning message 'networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress'. + +This helps people affected by the deprecation to know the request they are making is deprecated, +how long they have to address the issue, and what API they should use instead. +This is especially helpful when the user is applying a manifest they didn't create, +so they have time to reach out to the authors to ask for an updated version. + +We also realized that the person *using* a deprecated API is often not the same person responsible for upgrading the cluster, +so we added two administrator-facing tools to help track use of deprecated APIs and determine when upgrades are safe. + +### Metrics + +Starting in Kubernetes v1.19, when a request is made to a deprecated REST API endpoint, +an `apiserver_requested_deprecated_apis` gauge metric is set to `1` in the kube-apiserver process. +This metric has labels for the API `group`, `version`, `resource`, and `subresource`, +and a `removed_version` label that indicates the Kubernetes release in which the API will no longer be served. + +This is an example query using `kubectl`, [prom2json](https://github.com/prometheus/prom2json), +and [jq](https://stedolan.github.io/jq/) to determine which deprecated APIs have been requested +from the current instance of the API server: + +```sh +kubectl get --raw /metrics | prom2json | jq ' + .[] | select(.name=="apiserver_requested_deprecated_apis").metrics[].labels +' +``` + +Output: + +```json +{ + "group": "extensions", + "removed_release": "1.22", + "resource": "ingresses", + "subresource": "", + "version": "v1beta1" +} +{ + "group": "rbac.authorization.k8s.io", + "removed_release": "1.22", + "resource": "clusterroles", + "subresource": "", + "version": "v1beta1" +} +``` + +This shows the deprecated `extensions/v1beta1` Ingress and `rbac.authorization.k8s.io/v1beta1` ClusterRole APIs +have been requested on this server, and will be removed in v1.22. + +We can join that information with the `apiserver_request_total` metrics to get more details about the requests being made to these APIs: + +```sh +kubectl get --raw /metrics | prom2json | jq ' + # set $deprecated to a list of deprecated APIs + [ + .[] | + select(.name=="apiserver_requested_deprecated_apis").metrics[].labels | + {group,version,resource} + ] as $deprecated + + | + + # select apiserver_request_total metrics which are deprecated + .[] | select(.name=="apiserver_request_total").metrics[] | + select(.labels | {group,version,resource} as $key | $deprecated | index($key)) +' +``` + +Output: + +```json +{ + "labels": { + "code": "0", + "component": "apiserver", + "contentType": "application/vnd.kubernetes.protobuf;stream=watch", + "dry_run": "", + "group": "extensions", + "resource": "ingresses", + "scope": "cluster", + "subresource": "", + "verb": "WATCH", + "version": "v1beta1" + }, + "value": "21" +} +{ + "labels": { + "code": "200", + "component": "apiserver", + "contentType": "application/vnd.kubernetes.protobuf", + "dry_run": "", + "group": "extensions", + "resource": "ingresses", + "scope": "cluster", + "subresource": "", + "verb": "LIST", + "version": "v1beta1" + }, + "value": "1" +} +{ + "labels": { + "code": "200", + "component": "apiserver", + "contentType": "application/json", + "dry_run": "", + "group": "rbac.authorization.k8s.io", + "resource": "clusterroles", + "scope": "cluster", + "subresource": "", + "verb": "LIST", + "version": "v1beta1" + }, + "value": "1" +} +``` + +The output shows that only read requests are being made to these APIs, and the most requests have been made to watch the deprecated Ingress API. + +You can also find that information through the following Prometheus query, +which returns information about requests made to deprecated APIs which will be removed in v1.22: + +```promql +apiserver_requested_deprecated_apis{removed_version="1.22"} * on(group,version,resource,subresource) +group_right() apiserver_request_total +``` + +### Audit annotations + +Metrics are a fast way to check whether deprecated APIs are being used, and at what rate, +but they don't include enough information to identify particular clients or API objects. +Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/) +for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`. +Administrators can use those audit events to identify specific clients or objects that need to be updated. + +## Custom Resource Definitions + +Along with the API server ability to warn about deprecated API use, starting in v1.19, a CustomResourceDefinition can indicate a +[particular version of the resource it defines is deprecated](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation). +When API requests to a deprecated version of a custom resource are made, a warning message is returned, matching the behavior of built-in APIs. + +The author of the CustomResourceDefinition can also customize the warning for each version if they want to. +This allows them to give a pointer to a migration guide or other information if needed. + +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition + name: crontabs.example.com +spec: + versions: + - name: v1alpha1 + # This indicates the v1alpha1 version of the custom resource is deprecated. + # API requests to this version receive a warning in the server response. + deprecated: true + # This overrides the default warning returned to clients making v1alpha1 API requests. + deprecationWarning: "example.com/v1alpha1 CronTab is deprecated; use example.com/v1 CronTab (see http://example.com/v1alpha1-v1)" + ... + + - name: v1beta1 + # This indicates the v1beta1 version of the custom resource is deprecated. + # API requests to this version receive a warning in the server response. + # A default warning message is returned for this version. + deprecated: true + ... + + - name: v1 + ... +``` + +## Admission Webhooks + +[Admission webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers) +are the primary way to integrate custom policies or validation with Kubernetes. +Starting in v1.19, admission webhooks can [return warning messages](/docs/reference/access-authn-authz/extensible-admission-controllers/#response) +that are passed along to the requesting API client. Warnings can be returned with allowed or rejected admission responses. + +As an example, to allow a request but warn about a configuration known not to work well, an admission webhook could send this response: + +```json +{ + "apiVersion": "admission.k8s.io/v1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": true, + "warnings": [ + ".spec.memory: requests >1GB do not work on Fridays" + ] + } +} +``` + +If you are implementing a webhook that returns a warning message, here are some tips: + +* Don't include a "Warning:" prefix in the message (that is added by clients on output) +* Use warning messages to describe problems the client making the API request should correct or be aware of +* Be brief; limit warnings to 120 characters if possible + +There are many ways admission webhooks could use this new feature, and I'm looking forward to seeing what people come up with. +Here are a couple ideas to get you started: + +* webhook implementations adding a "complain" mode, where they return warnings instead of rejections, + to allow trying out a policy to verify it is working as expected before starting to enforce it +* "lint" or "vet"-style webhooks, inspecting objects and surfacing warnings when best practices are not followed + +## Kubectl strict mode + +If you want to be sure you notice deprecations as soon as possible and get a jump start on addressing them, +`kubectl` added a `--warnings-as-errors` option in v1.19. When invoked with this option, +`kubectl` treats any warnings it receives from the server as errors and exits with a non-zero exit code: + +kubectl applying a manifest file with a --warnings-as-errors flag, displaying a warning message and exiting with a non-zero exit code. + +This could be used in a CI job to apply manifests to a current server, +and required to pass with a zero exit code in order for the CI job to succeed. + +## Future Possibilities + +Now that we have a way to communicate helpful information to users in context, +we're already considering other ways we can use this to improve people's experience with Kubernetes. +A couple areas we're looking at next are warning about [known problematic values](http://issue.k8s.io/64841#issuecomment-395141013) +we cannot reject outright for compatibility reasons, and warning about use of deprecated fields or field values +(like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/kubernetes-api/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)). +I'm excited to see progress in this area, continuing to make it easier to use Kubernetes. + +--- + +_[Jordan Liggitt](https://twitter.com/liggitt) is a software engineer at Google, and helps lead Kubernetes authentication, authorization, and API efforts._ \ No newline at end of file diff --git a/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings-as-errors.png b/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings-as-errors.png new file mode 100644 index 0000000000..5171eca6bc Binary files /dev/null and b/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings-as-errors.png differ diff --git a/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings.png b/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings.png new file mode 100644 index 0000000000..967bc591bf Binary files /dev/null and b/content/en/blog/_posts/2020-09-03-warnings/kubectl-warnings.png differ