From 19b386d06ea640317f4fc90c893a853431a23628 Mon Sep 17 00:00:00 2001 From: Chao Xu Date: Wed, 28 Jun 2017 16:38:44 -0700 Subject: [PATCH] Extensible admission controller (#4092) * extensible-admission-controllers * Update extensible-admission-controllers.md * more on initializers * fixes * Expand external admission webhooks documentation * wrap at 80 chars * more * add reference --- _data/reference.yml | 1 + .../admin/extensible-admission-controllers.md | 286 ++++++++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 docs/admin/extensible-admission-controllers.md diff --git a/_data/reference.yml b/_data/reference.yml index a6caf1af65..a755e8ebb5 100644 --- a/_data/reference.yml +++ b/_data/reference.yml @@ -12,6 +12,7 @@ toc: - docs/admin/authentication.md - docs/admin/bootstrap-tokens.md - docs/admin/admission-controllers.md + - docs/admin/extensible-admission-controllers.md - docs/admin/service-accounts-admin.md - title: Authorization section: diff --git a/docs/admin/extensible-admission-controllers.md b/docs/admin/extensible-admission-controllers.md new file mode 100644 index 0000000000..e57d512c8b --- /dev/null +++ b/docs/admin/extensible-admission-controllers.md @@ -0,0 +1,286 @@ +--- +assignees: +- smarterclayton +- lavalamp +- whitlockjc +- caesrxuchao +title: Dynamic Admission Control +--- + +* TOC +{:toc} + +## Overview + +The [admission controllers documentation](/doc/admin/admission-controllers.md) +introduces how to use standard, plugin-style admission controllers. However, +plugin admission controllers are not flexible enough for all use cases, due to +the following: + +* They need to be compiled into kube-apiserver. + +* They are only configurable when the apiserver starts up. + +1.7 introduces two alpha features, *Initializers* and *External Admission +Webhooks*, that address these limitations. These features allow admission +controllers to be developed out-of-tree and configured at runtime. + +This page describes how to use Initializers and External Admission Webhooks. + +## Initializers + +### What are initializers? + +Two meanings: + +* A list of pending pre-initialization tasks, stored in every object's metadata + (e.g., "AddMyCorporatePolicySidecar"). + +* The controllers which actually perform those tasks. The name of the task + corresponds to the controller which performs the task. For clarity, we call + them "initializer controllers" in this page. + +Once the controller has performed its assigned task, it removes its name from +the list. For example, it may send a PATCH that inserts a container in a pod and +also removes its name from `metadata.initalizers`. Initializers may make +mutations to objects. + +Objects which have a non-empty initializer list are considered uninitialized, +and are not visible in the API unless specifically requested +(`?includeUninitialized=true`). + +### When to use initializers? + +Initializers are useful for admins to force policies (e.g., the +[AlwaysPullImages](https://kubernetes.io/docs/admin/admission-controllers/#alwayspullimages) +admission controller), or to inject defaults (e.g., the +[DefaultStorageClass](https://kubernetes.io/docs/admin/admission-controllers/#defaultstorageclass) +admission controller), etc. + +Note that if your use case does not involve mutating objects, consider using +external admission webhooks as they have better performance. + +### How are initializers triggered? + +When an object is POSTed, it is checked against all existing +`initializerConfiguration` objects (explained below). For all that it matches, +all `spec.initializers[].name`s are appended to the new object's +`metadata.initializers` field. + +An initializer controller should list and watch for uninitialized objects, by +using the query parameter `?includeUninitialized=true`. If using client-go, just +set the +[listOptions.includeUninitialized](https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L315) +to true. + +For the observed uninitialized objects, an initializer controller should first +check if its name matches `metadata.initializers[0]`, if so, it should then +perform its assigned task and remove its name from the list. + +### Enable initializers alpha feature + +Initializers are an alpha feature, which is disabled by default. To turn it on, +you need to: + +* Include "Initializer" in the `--admission-control` flag when starting + `kube-apiserver`. If you have multiple `kube-apiserver` replicas, all should + have the same flag setting. + +* Enable the dynamic admission controller registration API by adding + `admissionregistration.k8s.io/v1alpha1` to the `--runtime-config` flag passed + to `kube-apiserver`, e.g. + `--runtime-config=admissionregistration.k8s.io/v1alpha1`. Again, all replicas + should have the same flag setting. + +### Deploy an initializer controller + +We suggest that deploying an initializer controller via the [deployment +API](https://kubernetes.io/docs/api-reference/v1.6/#deployment-v1beta1-apps). + +### Configure initializers on the fly + +You can configure what initializers are enabled and what resources are subject +to the initializers by creating `initializerconfigurations`. + +We suggest that you first deploy the initializer controller and make sure it is +working properly before creating the `initializerconfigurations`, otherwise any +newly created resources will be stuck in an uninitialized state. + +The following is an example `initiallizerConfiguration`. + +```yaml +apiVersion: admissionregistration.k8s.io/v1alpha1 +kind: InitializerConfiguration +metadata: + name: example-config +spec: + initializers: + # the name needs to be fully qualified, i.e., containing at least two "." + - name: podimage.example.com + rules: + # apiGroups, apiVersion, resources all support wildcard "*". + # "*" cannot be mixed with non-wildcard. + - apiGroups: + - "" + apiVersions: + - v1 + resources: + - pods +``` + +Make sure that all expansions of the `` tuple +in a `rule` are valid; if they are not, separate them in different `rules`. + +After you create the `initializerConfiguration`, the system will take a few +seconds to honor the new configuration. + +## External Admission Webhooks + +### What are external admission webhooks? + +External admission webhooks are HTTP callbacks that are intended to receive +admission requests and do something with them. What the external admission +webhook does is up to you but there is an +[interface](https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/pkg/apis/admission/v1alpha1/types.go) +the external admission webhook must follow and that is to respond back with +whether or not the admission request should be allowed. + +Unlike initializers or the plugin-style admission controllers, external +admission webhooks are not allowed to mutate the admission request in any way. + +Because admission is a high security operation, the external admission webhooks +must support TLS. + +### When to use admission webhooks? + +A good example use of an external admission webhook is to do semantic validation +of Kubernetes resources. Imagine your infrastructure requires that all `Pod` +resources have a common set of labels and you do not want any `Pod` to be +persisted to Kubernetes if those needs are not met. You could write your +external admission webhook to do this validation and respond accordingly. Of +course this is a very simple case but you get the idea. + +### How are external admission webhooks triggered? + +Whenever a request comes in, the `GenericAdmissionWebhook` admission plugin will +get the list of interested external admission webhooks from +`externalAdmissionHookConfiguration` objects (explained below) and call them in +parallel. If **all** of the external admission webhooks approve the admission +request, the admission chain continues. If **any** of the external admission +webhooks deny the admission request, the admission request will be denied and +the reason for doing so will be based on the _first_ external admission webhook +denial reason. _(This means if there is more than one external admission webhook +that denied the admission request, only the first will be returned to the +user.)_ If there is an error encountered when calling an external admission +webhook, that request is ignored and will not be used to approve/deny the +admission request. + +**Note:** The admission chain depends solely on the order of the +`--admission-control` option passed to `kube-apiserver`. + +### Enable external admission webhooks + +External Admission Webhooks is an alpha feature, which is disabled by default. +To turn it on, you need to + +* Include "GenericAdmissionWebhook" in the `--admission-control` flag when + starting the apiserver. If you have multiple `kube-apiserver` replicas, all + should have the same flag setting. + +* Enable the dynamic admission controller registration API by adding + `admissionregistration.k8s.io/v1alpha1` to the `--runtime-config` flag passed + to `kube-apiserver`, e.g. + `--runtime-config=admissionregistration.k8s.io/v1alpha1`. Again, all replicas + should have the same flag setting. + +### Write a webhook admission controller + +See [here](https://github.com/caesarxuchao/example-webhook-admission-controller) +for an example webhook admission controller. + +The communication between the webhook admission controller and the apiserver, or +more precisely, the GenericAdmissionWebhook admission controller, needs to be +TLS secured. You need to generate a CA cert and use it to sign the server cert +used by your webhook admission controller. The pem formatted CA cert is supplied +to the apiserver via the dynamic registration API +`externaladmissionhookconfigurations.clientConfig.caBundle`. + +For each request received by the apiserver, the GenericAdmissionWebhook +admission controller sends an +[admissionReview](https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/pkg/apis/admission/v1alpha1/types.go#L27) +to the relevant webhook admission controller. The webhook admission controller +gathers information like `object`, `oldobject`, and `userInfo`, from +`admissionReview.spec`, sends back a response with the body also being the +`admissionReview`, whose `status` field is filled with the admission decision. + +### Deploy the webhook admission controller + +See [here](https://github.com/caesarxuchao/example-webhook-admission-controller/tree/master/deployment) +for an example deployment. + +We suggest that deploying the webhook admission controller via the [deployment +API](https://kubernetes.io/docs/api-reference/v1.6/#deployment-v1beta1-apps). +You also need to create a +[service](https://kubernetes.io/docs/api-reference/v1.6/#service-v1-core) as the +front-end of the deployment. + +### Configure webhook admission controller on the fly + +You can configure what webhook admission controller are enabled and what +resources are subject to the admission controller via creating +externaladmissionhookconfigurations. + +We suggest that you first deploy the webhook admission controller and make sure +it is working properly before creating the externaladmissionhookconfigurations, +otherwise depending whether the webhook is configured as fail open or fail +closed, operations will be unconditionally accepted or rejected. + +The following is an example externaladmissionhookconfiguration. + +```yaml +apiVersion: admissionregistration.k8s.io/v1alpha1 +kind: ExternalAdmissionHookConfiguration +metadata: + name: example-config +externalAdmissionHooks: +- name: pod-image.k8s.io + rules: + - apiGroups: + - "" + apiVersions: + - v1 + operations: + - CREATE + resources: + - pods + failurePolicy: Ignore + clientConfig: + caBundle: + service: + name: + namespace: +``` + +For a request received by the apiserver, if the request matches any of the +`rules` of an `externalAdmissionHook`, the `GenericAdmissionWebhook` admission +controller will send an `admissionReview` request to the `externalAdmissionHook` +to ask for admission decision. + +The `rule` is similar to the `rule` in `initializerConfiguration`, with two +differences: + +* The addition of the `operations` field, specifying what operations the webhook + is interested in; + +* The `resources` field accepts subresources in the form or resource/subresource. + +Make sure that all expansions of the `` tuple +in a `rule` are valid; if they are not, separate them to different `rules`. + +You can also specify the `failurePolicy`. In 1.7, the system supports `Ignore` +and `Fail` policies, meaning upon an communication error with the webhook +admission controller, if the `GenericAdmissionWebhook` will admit or reject the +operation. + +After you create the `initializerConfiguration`, the system will take a few +seconds to honor the new configuration.