Arrange the Extending Kubernetes section in the Concepts TOC (#8569)
* Left nav for Extending Kubernetes * Redirect moved topics. Remove extraneous directory.
This commit is contained in:
Steve Perry
committed by
k8s-ci-robot
k8s-ci-robot
parent
fbfb8c4087
commit
169aadbbe1
@@ -0,0 +1,3 @@
|
||||
---
|
||||
title: Extending Kubernetes
|
||||
---
|
||||
@@ -0,0 +1,4 @@
|
||||
---
|
||||
title: Extending the Kubernetes API
|
||||
weight: 20
|
||||
---
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: Extending the Kubernetes API with the aggregation layer
|
||||
reviewers:
|
||||
- lavalamp
|
||||
- cheftako
|
||||
- chenopis
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
The aggregation layer allows Kubernetes to be extended with additional APIs, beyond what is offered by the core Kubernetes APIs.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Overview
|
||||
|
||||
The aggregation layer enables installing additional Kubernetes-style APIs in your cluster. These can either be pre-built, existing 3rd party solutions, such as [service-catalog](https://github.com/kubernetes-incubator/service-catalog/blob/master/README.md), or user-created APIs like [apiserver-builder](https://github.com/kubernetes-incubator/apiserver-builder/blob/master/README.md), which can get you started.
|
||||
|
||||
In 1.7 the aggregation layer runs in-process with the kube-apiserver. Until an extension resource is registered, the aggregation layer will do nothing. To register an API, users must add an APIService object, which "claims" the URL path in the Kubernetes API. At that point, the aggregation layer will proxy anything sent to that API path (e.g. /apis/myextension.mycompany.io/v1/…) to the registered APIService.
|
||||
|
||||
Ordinarily, the APIService will be implemented by an *extension-apiserver* in a pod running in the cluster. This extension-apiserver will normally need to be paired with one or more controllers if active management of the added resources is needed. As a result, the apiserver-builder will actually provide a skeleton for both. As another example, when the service-catalog is installed, it provides both the extension-apiserver and controller for the services it provides.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
* To get the aggregator working in your environment, [configure the aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/).
|
||||
* Then, [setup an extension api-server](/docs/tasks/access-kubernetes-api/setup-extension-api-server/) to work with the aggregation layer.
|
||||
* Also, learn how to [extend the Kubernetes API using Custom Resource Definitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,242 @@
|
||||
---
|
||||
title: Custom Resources
|
||||
reviewers:
|
||||
- enisoc
|
||||
- deads2k
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page explains [*custom resources*](/docs/concepts/api-extension/custom-resources/), which are extensions of the Kubernetes API. This page explains when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
## Custom resources
|
||||
|
||||
A *resource* is an endpoint in the [Kubernetes API](/docs/reference/api-overview/) that stores a collection of [API objects](/docs/concepts/overview/working-with-objects/kubernetes-objects/) of a certain kind. For example, the built-in *pods* resource contains a collection of Pod objects.
|
||||
|
||||
A *custom resource* is an extension of the Kubernetes API that is not necessarily available on every
|
||||
Kubernetes cluster.
|
||||
In other words, it represents a customization of a particular Kubernetes installation.
|
||||
|
||||
Custom resources can appear and disappear in a running cluster through dynamic registration,
|
||||
and cluster admins can update custom resources independently of the cluster itself.
|
||||
Once a custom resource is installed, users can create and access its objects with
|
||||
[kubectl](/docs/user-guide/kubectl-overview/), just as they do for built-in resources like *pods*.
|
||||
|
||||
### Custom controllers
|
||||
|
||||
On their own, custom resources simply let you store and retrieve structured data.
|
||||
It is only when combined with a *controller* that they become a true declarative API.
|
||||
A [declarative API](/docs/concepts/overview/working-with-objects/kubernetes-objects/#understanding-kubernetes-objects)
|
||||
allows you to _declare_ or specify the desired state of your resource and tries
|
||||
to match the actual state to this desired state.
|
||||
Here, the controller interprets the structured data as a record of the user's
|
||||
desired state, and continually takes action to achieve and maintain this state.
|
||||
|
||||
A *custom controller* is a controller that users can deploy and update on a running cluster, independently of the cluster's own lifecycle. Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources. The [Operator](https://coreos.com/blog/introducing-operators.html) pattern is one example of such a combination. It allows developers to encode domain knowledge for specific applications into an extension of the Kubernetes API.
|
||||
|
||||
### Should I add a custom resource to my Kubernetes Cluster?
|
||||
|
||||
When creating a new API, consider whether to [aggregate your API with the Kubernetes cluster APIs](/docs/concepts/api-extension/apiserver-aggregation/) or let your API stand alone.
|
||||
|
||||
| Consider API aggregation if: | Prefer a stand-alone API if: |
|
||||
| ---------------------------- | ---------------------------- |
|
||||
| Your API is [Declarative](#declarative-apis). | Your API does not fit the [Declarative](#declarative-apis) model. |
|
||||
| You want your new types to be readable and writable using `kubectl`.| `kubectl` support is not required |
|
||||
| You want to view your new types in a Kubernetes UI, such as dashboard, alongside built-in types. | Kubernetes UI support is not required. |
|
||||
| You are developing a new API. | You already have a program that serves your API and works well. |
|
||||
| You are willing to accept the format restriction that Kubernetes puts on REST resource paths, such as API Groups and Namespaces. (See the [API Overview](/docs/concepts/overview/kubernetes-api/).) | You need to have specific REST paths to be compatible with an already defined REST API. |
|
||||
| Your resources are naturally scoped to a cluster or to namespaces of a cluster. | Cluster or namespace scoped resources are a poor fit; you need control over the specifics of resource paths. |
|
||||
| You want to reuse [Kubernetes API support features](#common-features). | You don't need those features. |
|
||||
|
||||
#### Declarative APIs
|
||||
|
||||
In a Declarative API, typically:
|
||||
|
||||
- Your API consists of a relatively small number of relatively small objects (resources).
|
||||
- The objects define configuration of applications or infrastructure.
|
||||
- The objects are updated relatively infrequently.
|
||||
- Humans often need to read and write the objects.
|
||||
- The main operations on the objects are CRUD-y (creating, reading, updating and deleting).
|
||||
- Transactions across objects are not required: the API represents a desired state, not an exact state.
|
||||
|
||||
Imperative APIs are not declarative.
|
||||
Signs that your API might not be declarative include:
|
||||
|
||||
- The client says "do this", and then gets a synchronous response back when it is done.
|
||||
- The client says "do this", and then gets an operation ID back, and has to check a separate Operation objects to determine completion of the request.
|
||||
- You talk about Remote Procedure Calls (RPCs).
|
||||
- Directly storing large amounts of data (e.g. > a few kB per object, or >1000s of objects).
|
||||
- High bandwidth access (10s of requests per second sustained) needed.
|
||||
- Store end-user data (such as images, PII, etc) or other large-scale data processed by applications.
|
||||
- The natural operations on the objects are not CRUD-y.
|
||||
- The API is not easily modeled as objects.
|
||||
- You chose to represent pending operations with an operation ID or operation object.
|
||||
|
||||
### Should I use a configMap or a custom resource?
|
||||
|
||||
Use a ConfigMap if any of the following apply:
|
||||
|
||||
* There is an existing, well-documented config file format, such as a `mysql.cnf` or `pom.xml`.
|
||||
* You want to put the entire config file into one key of a configMap.
|
||||
* The main use of the config file is for a program running in a Pod on your cluster to consume the file to configure itself.
|
||||
* Consumers of the file prefer to consume via file in a Pod or environment variable in a pod, rather than the Kubernetes API.
|
||||
* You want to perform rolling updates via Deployment, etc, when the file is updated.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** Use a [secret](/docs/concepts/configuration/secret/) for sensitive data, which is similar to a configMap but more secure.
|
||||
{{< /note >}}
|
||||
|
||||
Use a custom resource (CRD or Aggregated API) if most of the following apply:
|
||||
|
||||
* You want to use Kubernetes client libraries and CLIs to create and update the new resource.
|
||||
* You want top-level support from kubectl (for example: `kubectl get my-object object-name`).
|
||||
* You want to build new automation that watches for updates on the new object, and then CRUD other objects, or vice versa.
|
||||
* You want to write automation that handles updates to the object.
|
||||
* You want to use Kubernetes API conventions like `.spec`, `.status`, and `.metadata`.
|
||||
* You want the object to be an abstraction over a collection of controlled resources, or a summarization of other resources.
|
||||
|
||||
## Adding custom resources
|
||||
|
||||
Kubernetes provides two ways to add custom resources to your cluster:
|
||||
|
||||
- [Custom Resource Definitions](/docs/concepts/api-extension/custom-resources/) (CRDs) are easier to use: they do not require any programming in some cases.
|
||||
- [API Aggregation](/docs/concepts/api-extension/apiserver-aggregation/) requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.
|
||||
|
||||
Kubernetes provides these two options to meet the needs of different users, so that neither ease of use nor flexibility are compromised.
|
||||
|
||||
Aggregated APIs are subordinate APIServers that sit behind the primary API server, which acts as a proxy. This arrangement is called [API Aggregation](/docs/concepts/api-extension/apiserver-aggregation/) (AA). To users, it simply appears that the Kubernetes API is extended.
|
||||
|
||||
Custom Resource Definitions (CRDS) allow users to create new types of resources without adding another APIserver. You do not need to understand API Aggregation to use CRDs.
|
||||
|
||||
Regardless of whether they are installed via CRDs or AA, the new resources are called Custom Resources to distinguish them from built-in Kubernetes resources (like pods).
|
||||
|
||||
## CustomResourceDefinitions
|
||||
|
||||
The [CustomResourceDefinition](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/) (CRD) API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.
|
||||
|
||||
This frees you from writing your own API server to handle the custom resource,
|
||||
but the generic nature of the implementation means you have less flexibility than with
|
||||
[API server aggregation](#api-server-aggregation).
|
||||
|
||||
Refer to the [Custom Controller example, which uses Custom Resources](https://github.com/kubernetes/sample-controller)
|
||||
for a demonstration of how to register a new custom resource, work with instances of your new resource type,
|
||||
and setup a controller to handle events.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** CRD is the successor to the deprecated *ThirdPartyResource* (TPR) API, and is available as of Kubernetes 1.7.
|
||||
{{< /note >}}
|
||||
|
||||
## API server aggregation
|
||||
|
||||
Usually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like *pods* and *services*, and can also handle custom resources in a generic way through [CustomResourceDefinitions](#customresourcedefinitions).
|
||||
|
||||
The [aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/) allows you to provide specialized
|
||||
implementations for your custom resources by writing and deploying your own standalone API server.
|
||||
The main API server delegates requests to you for the custom resources that you handle,
|
||||
making them available to all of its clients.
|
||||
|
||||
### Choosing a method for adding custom resources
|
||||
|
||||
CRDs are easier to use. Aggregated APIs are more flexible. Choose the method that best meets your needs.
|
||||
|
||||
Typically, CRDs are a good fit if:
|
||||
|
||||
* You have a handful of fields
|
||||
* You are using the resource within your company, or as part of a small open-source project (as opposed to a commercial product)
|
||||
|
||||
#### Comparing ease of use
|
||||
|
||||
CRDs are easier to create than Aggregated APIs.
|
||||
|
||||
| Custom Resource Definitions | Aggregated API |
|
||||
| --------------------------- | -------------- |
|
||||
| Do not require programming. Users can choose any language for a CRD controller. | Requires programming in Go and building binary and image. Users can choose any language for a CRD controller. |
|
||||
| No additional service to run; CRs are handled by API Server. | An additional service to create and that could fail. |
|
||||
| No ongoing support once the CRD is created. Any bug fixes are picked up as part of normal Kubernetes Master upgrades. | May need to periodically pickup bug fixes from upstream and rebuild and update the Aggregated APIserver. |
|
||||
| No need to handle multiple versions of your API. For example: when you control the client for this resource, you can upgrade it in sync with the API. | You need to handle multiple versions of your API, for example: when developing an extension to share with the world. |
|
||||
|
||||
### Advanced features and flexibility
|
||||
|
||||
Aggregated APIs offer more advanced API features and customization of other features, for example: the storage layer.
|
||||
|
||||
| Feature | Description | CRDs | Aggregated API |
|
||||
| ------- | ----------- | ---- | -------------- |
|
||||
| Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Beta feature of CRDs in v1.9. Checks limited to what is supported by OpenAPI v3.0. | Yes, arbitrary validation checks |
|
||||
| Defaulting | See above | No, but can achieve the same effect with an Initializer (requires programming) | Yes |
|
||||
| Multi-versioning | Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions. | No | Yes |
|
||||
| Custom Storage | If you need storage with a different performance mode (for example, time-series database instead of key-value store) or isolation for security (for example, encryption secrets or different | No | Yes |
|
||||
| Custom Business Logic | Perform arbitrary checks or actions when creating, reading, updating or deleting an object | No, but can get some of the same effects with Initializers or Finalizers (requires programming) | Yes |
|
||||
| Subresources | <ul><li>Add extra operations other than CRUD, such as "scale" or "exec"</li><li>Allows systems like HorizontalPodAutoscaler and PodDisruptionBudget interact with your new resource</li><li>Finer-grained access control: user writes spec section, controller writes status section.</li><li>Allows incrementing object Generation on custom resource data mutation (requires separate spec and status sections in the resource)</li></ul> | No but planned | Yes, any Subresource |
|
||||
| strategic-merge-patch | The new endpoints support PATCH with `Content-Type: application/strategic-merge-patch+json`. Useful for updating objects that may be modified both locally, and by the server. For more information, see ["Update API Objects in Place Using kubectl patch"](/docs/tasks/run-application/update-api-object-kubectl-patch/) | No | Yes |
|
||||
| Protocol Buffers | The new resource supports clients that want to use Protocol Buffers | No | Yes |
|
||||
| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | No but planned | Yes |
|
||||
|
||||
#### Common Features
|
||||
|
||||
When you create a custom resource, either via a CRDs or an AA, you get many features for your API, compared to implementing it outside the Kubernetes platform:
|
||||
|
||||
| Feature | What it does |
|
||||
| ------- | ------------ |
|
||||
| CRUD | The new endpoints support CRUD basic operations via HTTP and `kubectl` |
|
||||
| Watch | The new endpoints support Kubernetes Watch operations via HTTP |
|
||||
| Discovery | Clients like kubectl and dashboard automatically offer list, display, and field edit operations on your resources |
|
||||
| json-patch | The new endpoints support PATCH with `Content-Type: application/json-patch+json` |
|
||||
| merge-patch | The new endpoints support PATCH with `Content-Type: application/merge-patch+json` |
|
||||
| HTTPS | The new endpoints uses HTTPS |
|
||||
| Built-in Authentication | Access to the extension uses the core apiserver (aggregation layer) for authentication |
|
||||
| Built-in Authorization | Access to the extension can reuse the authorization used by the core apiserver (e.g. RBAC) |
|
||||
| Finalizers | Block deletion of extension resources until external cleanup happens. |
|
||||
| Admission Webhooks | Set default values and validate extension resources during any create/update/delete operation. |
|
||||
| UI/CLI Display | Kubectl, dashboard can display extension resources. |
|
||||
| Unset vs Empty | Clients can distinguish unset fields from zero-valued fields. |
|
||||
| Client Libraries Generation | Kubernetes provides generic client libraries, as well as tools to generate type-specific client libraries. |
|
||||
| Labels and annotations | Common metadata across objects that tools know how to edit for core and custom resources. |
|
||||
|
||||
## Preparing to install a custom resource
|
||||
|
||||
There are several points to be aware of before adding a custom resource to your cluster.
|
||||
|
||||
### Third party code and new points of failure
|
||||
|
||||
While creating a CRD does not automatically add any new points of failure (for example, by causing third party code to run on your API server), packages (for example, Charts) or other installation bundles often include CRDs as well as a Deployment of third-party code that implements the business logic for a new custom resource.
|
||||
|
||||
Installing an Aggregated APIserver always involves running a new Deployment.
|
||||
|
||||
### Storage
|
||||
|
||||
Custom resources consume storage space in the same way that ConfigMaps do. Creating too many custom resources may overload your API server's storage space.
|
||||
|
||||
Aggregated API servers may use the same storage as the main API server, in which case the same warning applies.
|
||||
|
||||
### Authentication, authorization, and auditing
|
||||
|
||||
CRDs always use the same authentication, authorization, and audit logging as the built-in resources of your API Server.
|
||||
|
||||
If you use RBAC for authorization, most RBAC roles will not grant access to the new resources (except the cluster-admin role or any role created with wildcard rules). You'll need to explicitly grant access to the new resources. CRDs and Aggregated APIs often come bundled with new role definitions for the types they add.
|
||||
|
||||
Aggregated API servers may or may not use the same authentication, authorization, and auditing as the primary API server.
|
||||
|
||||
## Accessing a custom resource
|
||||
|
||||
Kubernetes [client libraries](/docs/reference/client-libraries/) can be used to access custom resources. Not all client libraries support custom resources. The go and python client libraries do.
|
||||
|
||||
When you add a custom resource, you can access it using:
|
||||
|
||||
- kubectl
|
||||
- The kubernetes dynamic client.
|
||||
- A REST client that you write.
|
||||
- A client generated using [Kubernetes client generation tools](https://github.com/kubernetes/code-generator) (generating one is an advanced undertaking, but some projects may provide a client along with the CRD or AA).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
* Learn how to [Extend the Kubernetes API with the aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/).
|
||||
* Learn how to [Extend the Kubernetes API with CustomResourceDefinition](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
* Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
---
|
||||
title: Compute, Storage, and Networking Extensions
|
||||
weight: 30
|
||||
---
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
reviewers:
|
||||
title: Device Plugins
|
||||
description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup.
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
{{% capture overview %}}
|
||||
Starting in version 1.8, Kubernetes provides a
|
||||
[device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md)
|
||||
for vendors to advertise their resources to the kubelet without changing Kubernetes core code.
|
||||
Instead of writing custom Kubernetes code, vendors can implement a device plugin that can
|
||||
be deployed manually or as a DaemonSet. The targeted devices include GPUs,
|
||||
High-performance NICs, FPGAs, InfiniBand, and other similar computing resources
|
||||
that may require vendor specific initialization and setup.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Device plugin registration
|
||||
|
||||
The device plugins feature is gated by the `DevicePlugins` feature gate which
|
||||
is disabled by default before 1.10. When the device plugins feature is enabled,
|
||||
the kubelet exports a `Registration` gRPC service:
|
||||
|
||||
```gRPC
|
||||
service Registration {
|
||||
rpc Register(RegisterRequest) returns (Empty) {}
|
||||
}
|
||||
```
|
||||
A device plugin can register itself with the kubelet through this gRPC service.
|
||||
During the registration, the device plugin needs to send:
|
||||
|
||||
* The name of its Unix socket.
|
||||
* The Device Plugin API version against which it was built.
|
||||
* The `ResourceName` it wants to advertise. Here `ResourceName` needs to follow the
|
||||
[extended resource naming scheme](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#extended-resources)
|
||||
as `vendor-domain/resource`.
|
||||
For example, an Nvidia GPU is advertised as `nvidia.com/gpu`.
|
||||
|
||||
Following a successful registration, the device plugin sends the kubelet the
|
||||
list of devices it manages, and the kubelet is then in charge of advertising those
|
||||
resources to the API server as part of the kubelet node status update.
|
||||
For example, after a device plugin registers `vendor-domain/foo` with the kubelet
|
||||
and reports two healthy devices on a node, the node status is updated
|
||||
to advertise 2 `vendor-domain/foo`.
|
||||
|
||||
Then, users can request devices in a
|
||||
[Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core)
|
||||
specification as they request other types of resources, with the following limitations:
|
||||
* Extended resources are only supported as integer resources and cannot be overcommitted.
|
||||
* Devices cannot be shared among Containers.
|
||||
|
||||
Suppose a Kubernetes cluster is running a device plugin that advertises resource `vendor-domain/resource`
|
||||
on certain nodes, here is an example user pod requesting this resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: demo-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: demo-container-1
|
||||
image: k8s.gcr.io/pause:2.0
|
||||
resources:
|
||||
limits:
|
||||
vendor-domain/resource: 2 # requesting 2 vendor-domain/resource
|
||||
```
|
||||
|
||||
## Device plugin implementation
|
||||
|
||||
The general workflow of a device plugin includes the following steps:
|
||||
|
||||
* Initialization. During this phase, the device plugin performs vendor specific
|
||||
initialization and setup to make sure the devices are in a ready state.
|
||||
|
||||
* The plugin starts a gRPC service, with a Unix socket under host path
|
||||
`/var/lib/kubelet/device-plugins/`, that implements the following interfaces:
|
||||
|
||||
```gRPC
|
||||
service DevicePlugin {
|
||||
// ListAndWatch returns a stream of List of Devices
|
||||
// Whenever a Device state change or a Device disappears, ListAndWatch
|
||||
// returns the new list
|
||||
rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
|
||||
|
||||
// Allocate is called during container creation so that the Device
|
||||
// Plugin can run device specific operations and instruct Kubelet
|
||||
// of the steps to make the Device available in the container
|
||||
rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
|
||||
}
|
||||
```
|
||||
|
||||
* The plugin registers itself with the kubelet through the Unix socket at host
|
||||
path `/var/lib/kubelet/device-plugins/kubelet.sock`.
|
||||
|
||||
* After successfully registering itself, the device plugin runs in serving mode, during which it keeps
|
||||
monitoring device health and reports back to the kubelet upon any device state changes.
|
||||
It is also responsible for serving `Allocate` gRPC requests. During `Allocate`, the device plugin may
|
||||
do device-specific preparation; for example, GPU cleanup or QRNG initialization.
|
||||
If the operations succeed, the device plugin returns an `AllocateResponse` that contains container
|
||||
runtime configurations for accessing the allocated devices. The kubelet passes this information
|
||||
to the container runtime.
|
||||
|
||||
A device plugin is expected to detect kubelet restarts and re-register itself with the new
|
||||
kubelet instance. In the current implementation, a new kubelet instance deletes all the existing Unix sockets
|
||||
under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion
|
||||
of its Unix socket and re-register itself upon such an event.
|
||||
|
||||
## Device plugin deployment
|
||||
|
||||
A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has
|
||||
the benefit that Kubernetes can restart the device plugin if it fails.
|
||||
Otherwise, an extra mechanism is needed to recover from device plugin failures.
|
||||
The canonical directory `/var/lib/kubelet/device-plugins` requires privileged access,
|
||||
so a device plugin must run in a privileged security context.
|
||||
If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins`
|
||||
must be mounted as a
|
||||
[Volume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core)
|
||||
in the plugin's
|
||||
[PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core).
|
||||
|
||||
Kubernetes device plugin support is still in alpha. As development continues, its API version can
|
||||
change in incompatible ways. We recommend that device plugin developers do the following:
|
||||
* Watch for changes in future releases.
|
||||
* Support multiple versions of the device plugin API for backward/forward compatibility.
|
||||
|
||||
If you enable the DevicePlugins feature and run device plugins on nodes that need to be upgraded to
|
||||
a Kubernetes release with a newer device plugin API version, upgrade your device plugins
|
||||
to support both versions before upgrading these nodes to
|
||||
ensure the continuous functioning of the device allocations during the upgrade.
|
||||
|
||||
## Examples
|
||||
|
||||
For examples of device plugin implementations, see:
|
||||
* The official [NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin)
|
||||
* it requires using [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) which allows you to run GPU enabled docker containers
|
||||
* The [NVIDIA GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu).
|
||||
* The [RDMA device plugin](https://github.com/hustcat/k8s-rdma-device-plugin)
|
||||
* The [Solarflare device plugin](https://github.com/vikaschoudhary16/sfc-device-plugin)
|
||||
* The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin)
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
reviewers:
|
||||
- dcbw
|
||||
- freehan
|
||||
- thockin
|
||||
title: Network Plugins
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
__Disclaimer__: Network plugins are in alpha. Its contents will change rapidly.
|
||||
|
||||
Network plugins in Kubernetes come in a few flavors:
|
||||
|
||||
* CNI plugins: adhere to the appc/CNI specification, designed for interoperability.
|
||||
* Kubenet plugin: implements basic `cbr0` using the `bridge` and `host-local` CNI plugins
|
||||
|
||||
## Installation
|
||||
|
||||
The kubelet has a single default network plugin, and a default network common to the entire cluster. It probes for plugins when it starts up, remembers what it found, and executes the selected plugin at appropriate times in the pod lifecycle (this is only true for Docker, as rkt manages its own CNI plugins). There are two Kubelet command line parameters to keep in mind when using plugins:
|
||||
|
||||
* `cni-bin-dir`: Kubelet probes this directory for plugins on startup
|
||||
* `network-plugin`: The network plugin to use from `cni-bin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is simply "cni".
|
||||
|
||||
## Network Plugin Requirements
|
||||
|
||||
Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy.
|
||||
|
||||
By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy.
|
||||
|
||||
### CNI
|
||||
|
||||
The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`).
|
||||
|
||||
If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used.
|
||||
|
||||
In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0
|
||||
|
||||
Limitation: Due to [#31307](https://github.com/kubernetes/kubernetes/issues/31307), `HostPort` won't work with CNI networking plugin at the moment. That means all `hostPort` attribute in pod would be simply ignored.
|
||||
|
||||
### kubenet
|
||||
|
||||
Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.
|
||||
|
||||
Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host.
|
||||
|
||||
The plugin requires a few things:
|
||||
|
||||
* The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `cni-bin-dir` to supply additional search path. The first found match will take effect.
|
||||
* Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin
|
||||
* Kubelet should also be run with the `--non-masquerade-cidr=<clusterCidr>` argument to ensure traffic to IPs outside this range will use IP masquerade.
|
||||
* The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=<cidr>` controller-manager command-line options.
|
||||
|
||||
### Customizing the MTU (with kubenet)
|
||||
|
||||
The MTU should always be configured correctly to get the best networking performance. Network plugins will usually try
|
||||
to infer a sensible MTU, but sometimes the logic will not result in an optimal MTU. For example, if the
|
||||
Docker bridge or another interface has a small MTU, kubenet will currently select that MTU. Or if you are
|
||||
using IPSEC encapsulation, the MTU must be reduced, and this calculation is out-of-scope for
|
||||
most network plugins.
|
||||
|
||||
Where needed, you can specify the MTU explicitly with the `network-plugin-mtu` kubelet option. For example,
|
||||
on AWS the `eth0` MTU is typically 9001, so you might specify `--network-plugin-mtu=9001`. If you're using IPSEC you
|
||||
might reduce it to allow for encapsulation overhead e.g. `--network-plugin-mtu=8873`.
|
||||
|
||||
This option is provided to the network-plugin; currently **only kubenet supports `network-plugin-mtu`**.
|
||||
|
||||
## Usage Summary
|
||||
|
||||
* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`).
|
||||
* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge` and `host-local` plugins placed in `/opt/cni/bin` or `cni-bin-dir`.
|
||||
* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin.
|
||||
@@ -0,0 +1,216 @@
|
||||
---
|
||||
title: Extending your Kubernetes Cluster
|
||||
reviewers:
|
||||
- erictune
|
||||
- lavalamp
|
||||
- cheftako
|
||||
- chenopis
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
Kubernetes is highly configurable and extensible. As a result,
|
||||
there is rarely a need to fork or submit patches to the Kubernetes
|
||||
project code.
|
||||
|
||||
This guide describes the options for customizing a Kubernetes
|
||||
cluster. It is aimed at {{< glossary_tooltip text="Cluster Operators" term_id="cluster-operator" >}} who want to
|
||||
understand how to adapt their Kubernetes cluster to the needs of
|
||||
their work environment. Developers who are prospective {{< glossary_tooltip text="Platform Developers" term_id="platform-developer" >}} or Kubernetes Project {{< glossary_tooltip text="Contributors" term_id="contributor" >}} will also find it
|
||||
useful as an introduction to what extension points and patterns
|
||||
exist, and their trade-offs and limitations.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Overview
|
||||
|
||||
Customization approaches can be broadly divided into *configuration*, which only involves changing flags, local configuration files, or API resources; and *extensions*, which involve running additional programs or services. This document is primarily about extensions.
|
||||
|
||||
## Configuration
|
||||
|
||||
*Configuration files* and *flags* are documented in the Reference section of the online documentation, under each binary:
|
||||
|
||||
* [kubelet](/docs/admin/kubelet/)
|
||||
* [kube-apiserver](/docs/admin/kube-apiserver/)
|
||||
* [kube-controller-manager](/docs/admin/kube-controller-manager/)
|
||||
* [kube-scheduler](/docs/admin/kube-scheduler/).
|
||||
|
||||
Flags and configuration files may not always be changeable in a hosted Kubernetes service or a distribution with managed installation. When they are changeable, they are usually only changeable by the cluster administrator. Also, they are subject to change in future Kubernetes versions, and setting them may require restarting processes. For those reasons, they should be used only when there are no other options.
|
||||
|
||||
*Built-in Policy APIs*, such as [ResourceQuota](/docs/concepts/policy/resource-quotas/), [PodSecurityPolicies](/docs/concepts/policy/pod-security-policy/), [NetworkPolicy](/docs/concepts/services-networking/network-policies/) and Role-based Access Control ([RBAC](/docs/admin/authorization/rbac/)), are built-in Kubernetes APIs. APIs are typically used with hosted Kubernetes services and with managed Kubernetes installations. They are declarative and use the same conventions as other Kubernetes resources like pods, so new cluster configuration can be repeatable and be managed the same way as applications. And, where they are stable, they enjoy a [defined support policy](/docs/reference/deprecation-policy/) like other Kubernetes APIs. For these reasons, they are preferred over *configuration files* and *flags* where suitable.
|
||||
|
||||
## Extensions
|
||||
|
||||
Extensions are software components that extend and deeply integrate with Kubernetes.
|
||||
They adapt it to support new types and new kinds of hardware.
|
||||
|
||||
Most cluster administrators will use a hosted or distribution
|
||||
instance of Kubernetes. As a result, most Kubernetes users will need to
|
||||
install extensions and fewer will need to author new ones.
|
||||
|
||||
## Extension Patterns
|
||||
|
||||
Kubernetes is designed to be automated by writing client programs. Any
|
||||
program that reads and/or writes to the Kubernetes API can provide useful
|
||||
automation. *Automation* can run on the cluster or off it. By following
|
||||
the guidance in this doc you can write highly available and robust automation.
|
||||
Automation generally works with any Kubernetes cluster, including hosted
|
||||
clusters and managed installations.
|
||||
|
||||
There is a specific pattern for writing client programs that work well with
|
||||
Kubernetes called the *Controller* pattern. Controllers typically read an
|
||||
object's `.spec`, possibly do things, and then update the object's `.status`.
|
||||
|
||||
A controller is a client of Kubernetes. When Kubernetes is the client and
|
||||
calls out to a remote service, it is called a *Webhook*. The remote service
|
||||
is called a *Webhook Backend*. Like Controllers, Webhooks do add a point of
|
||||
failure.
|
||||
|
||||
In the webhook model, Kubernetes makes a network request to a remote service.
|
||||
In the *Binary Plugin* model, Kubernetes executes a binary (program).
|
||||
Binary plugins are used by the kubelet (e.g. [Flex Volume
|
||||
Plugins](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md)
|
||||
and [Network
|
||||
Plugins](/docs/concepts/cluster-administration/network-plugins/))
|
||||
and by kubectl.
|
||||
|
||||
Below is a diagram showing how the extensions points interact with the
|
||||
Kubernetes control plane.
|
||||
|
||||
<img src="https://docs.google.com/drawings/d/e/2PACX-1vQBRWyXLVUlQPlp7BvxvV9S1mxyXSM6rAc_cbLANvKlu6kCCf-kGTporTMIeG5GZtUdxXz1xowN7RmL/pub?w=960&h=720">
|
||||
|
||||
<!-- image source drawing https://docs.google.com/drawings/d/1muJ7Oxuj_7Gtv7HV9-2zJbOnkQJnjxq-v1ym_kZfB-4/edit?ts=5a01e054 -->
|
||||
|
||||
|
||||
## Extension Points
|
||||
|
||||
This diagram shows the extension points in a Kubernetes system.
|
||||
|
||||
<img src="https://docs.google.com/drawings/d/e/2PACX-1vSH5ZWUO2jH9f34YHenhnCd14baEb4vT-pzfxeFC7NzdNqRDgdz4DDAVqArtH4onOGqh0bhwMX0zGBb/pub?w=425&h=809">
|
||||
|
||||
<!-- image source diagrams: https://docs.google.com/drawings/d/1k2YdJgNTtNfW7_A8moIIkij-DmVgEhNrn3y2OODwqQQ/view -->
|
||||
|
||||
1. Users often interact with the Kubernetes API using `kubectl`. [Kubectl plugins](/docs/tasks/extend-kubectl/kubectl-plugins/) extend the kubectl binary. They only affect the individual user's local environment, and so cannot enforce site-wide policies.
|
||||
2. The apiserver handles all requests. Several types of extension points in the apiserver allow authenticating requests, or blocking them based on their content, editing content, and handling deletion. These are described in the [API Access Extensions](/docs/concepts/overview/extending#api-access-extensions) section.
|
||||
3. The apiserver serves various kinds of *resources*. *Built-in resource kinds*, like `pods`, are defined by the Kubernetes project and can't be changed. You can also add resources that you define, or that other projects have defined, called *Custom Resources*, as explained in the [Custom Resources](/docs/concepts/overview/extending#user-defined-types) section. Custom Resources are often used with API Access Extensions.
|
||||
4. The Kubernetes scheduler decides which nodes to place pods on. There are several ways to extend scheduling. These are described in the [Scheduler Extensions](/docs/concepts/overview/extending#scheduler-extensions) section.
|
||||
5. Much of the behavior of Kubernetes is implemented by programs called Controllers which are clients of the API-Server. Controllers are often used in conjunction with Custom Resources.
|
||||
6. The kubelet runs on servers, and helps pods appear like virtual servers with their own IPs on the cluster network. [Network Plugins](/docs/concepts/overview/extending#network-plugins) allow for different implementations of pod networking.
|
||||
7. The kubelet also mounts and unmounts volumes for containers. New types of storage can be supported via [Storage Plugins](/docs/concepts/overview/extending#storage-plugins).
|
||||
|
||||
If you are unsure where to start, this flowchart can help. Note that some solutions may involve several types of extensions.
|
||||
|
||||
|
||||
<img src="https://docs.google.com/drawings/d/e/2PACX-1vRWXNNIVWFDqzDY0CsKZJY3AR8sDeFDXItdc5awYxVH8s0OLherMlEPVUpxPIB1CSUu7GPk7B2fEnzM/pub?w=1440&h=1080">
|
||||
|
||||
<!-- image source drawing: https://docs.google.com/drawings/d/1sdviU6lDz4BpnzJNHfNpQrqI9F19QZ07KnhnxVrp2yg/edit -->
|
||||
|
||||
## API Extensions
|
||||
### User-Defined Types
|
||||
|
||||
Consider adding a Custom Resource to Kubernetes if you want to define new controllers, application configuration objects or other declarative APIs, and to manage them using Kubernetes tools, such as `kubectl`.
|
||||
|
||||
Do not use a Custom Resource as data storage for application, user, or monitoring data.
|
||||
|
||||
For more about Custom Resources, see the [Custom Resources concept guide](/docs/concepts/api-extension/custom-resources/).
|
||||
|
||||
|
||||
### Combining New APIs with Automation
|
||||
|
||||
Often, when you add a new API, you also add a control loop that reads and/or writes the new APIs. When the combination of a Custom API and a control loop is used to manage a specific, usually stateful, application, this is called the *Operator* pattern. Custom APIs and control loops can also be used to control other resources, such as storage, policies, and so on.
|
||||
|
||||
### Changing Built-in Resources
|
||||
|
||||
When you extend the Kubernetes API by adding custom resources, the added resources always fall into a new API Groups. You cannot replace or change existing API groups.
|
||||
Adding an API does not directly let you affect the behavior of existing APIs (e.g. Pods), but API Access Extensions do.
|
||||
|
||||
|
||||
### API Access Extensions
|
||||
|
||||
When a request reaches the Kubernetes API Server, it is first Authenticated, then Authorized, then subject to various types of Admission Control. See [[Accessing the API](/docs/admin/accessing-the-api/)] for more on this flow.
|
||||
|
||||
Each of these steps offers extension points.
|
||||
|
||||
Kubernetes has several built-in authentication methods that it supports. It can also sit behind an authenticating proxy, and it can send a token from an Authorization header to a remote service for verification (a webhook). All of these methods are covered in the [Authentication documentation](/docs/admin/authentication/).
|
||||
|
||||
### Authentication
|
||||
|
||||
[Authentication](/docs/admin/authentication) maps headers or certificates in all requests to a username for the client making the request.
|
||||
|
||||
Kubernetes provides several built-in authentication methods, and an [Authentication webhook](/docs/admin/authentication/#webhook-token-authentication) method if those don't meet your needs.
|
||||
|
||||
|
||||
### Authorization
|
||||
|
||||
[Authorization](/docs/admin/authorization/webhook/) determines whether specific users can read, write, and do other operations on API resources. It just works at the level of whole resources -- it doesn't discriminate based on arbitrary object fields. If the built-in authorization options don't meet your needs, and [Authorization webhook](/docs/admin/authorization/webhook/) allows calling out to user-provided code to make an authorization decision.
|
||||
|
||||
|
||||
### Dynamic Admission Control
|
||||
|
||||
After a request is authorized, if it is a write operation, it also goes through [Admission Control](/docs/admin/admission-controllers/) steps. In addition to the built-in steps, there are several extensions:
|
||||
|
||||
* The [Image Policy webhook](/docs/admin/admission-controllers/#imagepolicywebhook) restricts what images can be run in containers.
|
||||
* To make arbitrary admission control decisions, a general [Admission webhook](/docs/admin/extensible-admission-controllers/#external-admission-webhooks) can be used. Admission Webhooks can reject creations or updates.
|
||||
* [Initializers](/docs/admin/extensible-admission-controllers/#initializers) are controllers that can modify objects before they are created. Initializers can modify initial object creations but cannot affect updates to objects. Initializers can also reject objects.
|
||||
|
||||
## Infrastructure Extensions
|
||||
|
||||
|
||||
### Storage Plugins
|
||||
|
||||
[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/flexvolume-deployment.md
|
||||
) allow users to mount volume types without built-in support by having the
|
||||
Kubelet call a Binary Plugin to mount the volume.
|
||||
|
||||
|
||||
### Device Plugins
|
||||
|
||||
Device plugins allow a node to discover new Node resources (in addition to the
|
||||
builtin ones like cpu and memory) via a [Device
|
||||
Plugin](/docs/concepts/cluster-administration/device-plugins/).
|
||||
|
||||
|
||||
### Network Plugins
|
||||
|
||||
Different networking fabrics can be supported via node-level [Network Plugins](/docs/admin/network-plugins/).
|
||||
|
||||
### Scheduler Extensions
|
||||
|
||||
The scheduler is a special type of controller that watches pods, and assigns
|
||||
pods to nodes. The default scheduler can be replaced entirely, while
|
||||
continuing to use other Kubernetes components, or [multiple
|
||||
schedulers](/docs/tasks/administer-cluster/configure-multiple-schedulers/)
|
||||
can run at the same time.
|
||||
|
||||
This is a significant undertaking, and almost all Kubernetes users find they
|
||||
do not need to modify the scheduler.
|
||||
|
||||
The scheduler also supports a
|
||||
[webhook](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md)
|
||||
that permits a webhook backend (scheduler extension) to filter and prioritize
|
||||
the nodes chosen for a pod.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
* Learn more about [Custom Resources](/docs/concepts/api-extension/custom-resources/)
|
||||
* Learn about [Dynamic admission control](/docs/admin/extensible-admission-controllers/)
|
||||
* Learn more about Infrastructure extensions
|
||||
* [Network Plugins](/docs/concepts/cluster-administration/network-plugins/)
|
||||
* [Device Plugins](/docs/concepts/cluster-administration/device-plugins/)
|
||||
* Learn about [kubectl plugins](/docs/tasks/extend-kubectl/kubectl-plugins/)
|
||||
* See examples of Automation
|
||||
* [List of Operators](https://github.com/coreos/awesome-kubernetes-extensions)
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
---
|
||||
title: Service Catalog
|
||||
reviewers:
|
||||
- chenopis
|
||||
content_template: templates/concept
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
{{< glossary_definition term_id="service-catalog" length="all" prepend="Service Catalog is" >}}
|
||||
|
||||
A service broker, as defined by the [Open service broker API spec](https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md), is an endpoint for a set of managed services offered and maintained by a third-party, which could be a cloud provider such as AWS, GCP, or Azure.
|
||||
Some examples of managed services are Microsoft Azure Cloud Queue, Amazon Simple Queue Service, and Google Cloud Pub/Sub, but they can be any software offering that can be used by an application.
|
||||
|
||||
Using Service Catalog, a {{< glossary_tooltip text="cluster operator" term_id="cluster-operator" >}} can browse the list of managed services offered by a service broker, provision an instance of a managed service, and bind with it to make it available to an application in the Kubernetes cluster.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture body %}}
|
||||
## Example use case
|
||||
|
||||
An {{< glossary_tooltip text="application developer" term_id="application-developer" >}} wants to use message queuing as part of their application running in a Kubernetes cluster.
|
||||
However, they do not want to deal with the overhead of setting such a service up and administering it themselves.
|
||||
Fortunately, there is a cloud provider that offers message queuing as a managed service through its service broker.
|
||||
|
||||
A cluster operator can setup Service Catalog and use it to communicate with the cloud provider's service broker to provision an instance of the message queuing service and make it available to the application within the Kubernetes cluster.
|
||||
The application developer therefore does not need to be concerned with the implementation details or management of the message queue.
|
||||
The application can simply use it as a service.
|
||||
|
||||
## Architecture
|
||||
|
||||
Service Catalog uses the [Open service broker API](https://github.com/openservicebrokerapi/servicebroker) to communicate with service brokers, acting as an intermediary for the Kubernetes API Server to negotiate the initial provisioning and retrieve the credentials necessary for the application to use a managed service.
|
||||
|
||||
It is implemented as an extension API server and a controller, using etcd for storage. It also uses the [aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/) available in Kubernetes 1.7+ to present its API.
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
|
||||
|
||||
### API Resources
|
||||
|
||||
Service Catalog installs the `servicecatalog.k8s.io` API and provides the following Kubernetes resources:
|
||||
|
||||
* `ClusterServiceBroker`: An in-cluster representation of a service broker, encapsulating its server connection details.
|
||||
These are created and managed by cluster operators who wish to use that broker server to make new types of managed services available within their cluster.
|
||||
* `ClusterServiceClass`: A managed service offered by a particular service broker.
|
||||
When a new `ClusterServiceBroker` resource is added to the cluster, the Service Catalog controller connects to the service broker to obtain a list of available managed services. It then creates a new `ClusterServiceClass` resource corresponding to each managed service.
|
||||
* `ClusterServicePlan`: A specific offering of a managed service. For example, a managed service may have different plans available, such as a free tier or paid tier, or it may have different configuration options, such as using SSD storage or having more resources. Similar to `ClusterServiceClass`, when a new `ClusterServiceBroker` is added to the cluster, Service Catalog creates a new `ClusterServicePlan` resource corresponding to each Service Plan available for each managed service.
|
||||
* `ServiceInstance`: A provisioned instance of a `ClusterServiceClass`.
|
||||
These are created by cluster operators to make a specific instance of a managed service available for use by one or more in-cluster applications.
|
||||
When a new `ServiceInstance` resource is created, the Service Catalog controller connects to the appropriate service broker and instruct it to provision the service instance.
|
||||
* `ServiceBinding`: Access credentials to a `ServiceInstance`.
|
||||
These are created by cluster operators who want their applications to make use of a `ServiceInstance`.
|
||||
Upon creation, the Service Catalog controller creates a Kubernetes `Secret` containing connection details and credentials for the Service Instance, which can be mounted into Pods.
|
||||
|
||||
### Authentication
|
||||
|
||||
Service Catalog supports these methods of authentication:
|
||||
|
||||
* Basic (username/password)
|
||||
* [OAuth 2.0 Bearer Token](https://tools.ietf.org/html/rfc6750)
|
||||
|
||||
## Usage
|
||||
|
||||
A cluster operator can use Service Catalog API Resources to provision managed services and make them available within a Kubernetes cluster. The steps involved are:
|
||||
|
||||
1. Listing the managed services and Service Plans available from a service broker.
|
||||
1. Provisioning a new instance of the managed service.
|
||||
1. Binding to the managed service, which returns the connection credentials.
|
||||
1. Mapping the connection credentials into the application.
|
||||
|
||||
### Listing managed services and Service Plans
|
||||
|
||||
First, a cluster operator must create a `ClusterServiceBroker` resource within the `servicecatalog.k8s.io` group. This resource contains the URL and connection details necessary to access a service broker endpoint.
|
||||
|
||||
This is an example of a `ClusterServiceBroker` resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: servicecatalog.k8s.io/v1beta1
|
||||
kind: ClusterServiceBroker
|
||||
metadata:
|
||||
name: cloud-broker
|
||||
spec:
|
||||
# Points to the endpoint of a service broker. (This example is not a working URL.)
|
||||
url: https://servicebroker.somecloudprovider.com/v1alpha1/projects/service-catalog/brokers/default
|
||||
#####
|
||||
# Additional values can be added here, which may be used to communicate
|
||||
# with the service broker, such as bearer token info or a caBundle for TLS.
|
||||
#####
|
||||
```
|
||||
|
||||
The following is a sequence diagram illustrating the steps involved in listing managed services and Plans available from a service broker:
|
||||
|
||||
{:height="80%" width="80%"}
|
||||
|
||||
1. Once the `ClusterServiceBroker` resource is added to Service Catalog, it triggers a call to the external service broker for a list of available services.
|
||||
1. The service broker returns a list of available managed services and a list of Service Plans, which are cached locally as `ClusterServiceClass` and `ClusterServicePlan` resources respectively.
|
||||
1. A cluster operator can then get the list of available managed services using the following command:
|
||||
|
||||
kubectl get clusterserviceclasses -o=custom-columns=SERVICE\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName
|
||||
|
||||
It should output a list of service names with a format similar to:
|
||||
|
||||
SERVICE NAME EXTERNAL NAME
|
||||
4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468 cloud-provider-service
|
||||
... ...
|
||||
|
||||
They can also view the Service Plans available using the following command:
|
||||
|
||||
kubectl get clusterserviceplans -o=custom-columns=PLAN\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName
|
||||
|
||||
It should output a list of plan names with a format similar to:
|
||||
|
||||
PLAN NAME EXTERNAL NAME
|
||||
86064792-7ea2-467b-af93-ac9694d96d52 service-plan-name
|
||||
... ...
|
||||
|
||||
|
||||
### Provisioning a new instance
|
||||
|
||||
A cluster operator can initiate the provisioning of a new instance by creating a `ServiceInstance` resource.
|
||||
|
||||
This is an example of a `ServiceInstance` resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: servicecatalog.k8s.io/v1beta1
|
||||
kind: ServiceInstance
|
||||
metadata:
|
||||
name: cloud-queue-instance
|
||||
namespace: cloud-apps
|
||||
spec:
|
||||
# References one of the previously returned services
|
||||
clusterServiceClassExternalName: cloud-provider-service
|
||||
clusterServicePlanExternalName: service-plan-name
|
||||
#####
|
||||
# Additional parameters can be added here,
|
||||
# which may be used by the service broker.
|
||||
#####
|
||||
```
|
||||
|
||||
The following sequence diagram illustrates the steps involved in provisioning a new instance of a managed service:
|
||||
|
||||
{:height="80%" width="80%"}
|
||||
|
||||
1. When the `ServiceInstance` resource is created, Service Catalog initiates a call to the external service broker to provision an instance of the service.
|
||||
1. The service broker creates a new instance of the managed service and returns an HTTP response.
|
||||
1. A cluster operator can then check the status of the instance to see if it is ready.
|
||||
|
||||
### Binding to a managed service
|
||||
|
||||
After a new instance has been provisioned, a cluster operator must bind to the managed service to get the connection credentials and service account details necessary for the application to use the service. This is done by creating a `ServiceBinding` resource.
|
||||
|
||||
The following is an example of a `ServiceBinding` resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: servicecatalog.k8s.io/v1beta1
|
||||
kind: ServiceBinding
|
||||
metadata:
|
||||
name: cloud-queue-binding
|
||||
namespace: cloud-apps
|
||||
spec:
|
||||
instanceRef:
|
||||
name: cloud-queue-instance
|
||||
#####
|
||||
# Additional information can be added here, such as a secretName or
|
||||
# service account parameters, which may be used by the service broker.
|
||||
#####
|
||||
```
|
||||
|
||||
The following sequence diagram illustrates the steps involved in binding to a managed service instance:
|
||||
|
||||
{:height="80%" width="80%"}
|
||||
|
||||
1. After the `ServiceBinding` is created, Service Catalog makes a call to the external service broker requesting the information necessary to bind with the service instance.
|
||||
1. The service broker enables the application permissions/roles for the appropriate service account.
|
||||
1. The service broker returns the information necessary to connect and access the managed service instance. This is provider and service-specific so the information returned may differ between Service Providers and their managed services.
|
||||
|
||||
### Mapping the connection credentials
|
||||
|
||||
After binding, the final step involves mapping the connection credentials and service-specific information into the application.
|
||||
These pieces of information are stored in secrets that the application in the cluster can access and use to connect directly with the managed service.
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
|
||||
#### Pod configuration File
|
||||
|
||||
One method to perform this mapping is to use a declarative Pod configuration.
|
||||
|
||||
The following example describes how to map service account credentials into the application. A key called `sa-key` is stored in a volume named `provider-cloud-key`, and the application mounts this volume at `/var/secrets/provider/key.json`. The environment variable `PROVIDER_APPLICATION_CREDENTIALS` is mapped from the value of the mounted file.
|
||||
|
||||
```yaml
|
||||
...
|
||||
spec:
|
||||
volumes:
|
||||
- name: provider-cloud-key
|
||||
secret:
|
||||
secretName: sa-key
|
||||
containers:
|
||||
...
|
||||
volumeMounts:
|
||||
- name: provider-cloud-key
|
||||
mountPath: /var/secrets/provider
|
||||
env:
|
||||
- name: PROVIDER_APPLICATION_CREDENTIALS
|
||||
value: "/var/secrets/provider/key.json"
|
||||
```
|
||||
|
||||
The following example describes how to map secret values into application environment variables. In this example, the messaging queue topic name is mapped from a secret named `provider-queue-credentials` with a key named `topic` to the environment variable `TOPIC`.
|
||||
|
||||
|
||||
```yaml
|
||||
...
|
||||
env:
|
||||
- name: "TOPIC"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: provider-queue-credentials
|
||||
key: topic
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
* If you are familiar with {{< glossary_tooltip text="Helm Charts" term_id="helm-chart" >}}, [install Service Catalog using Helm](/docs/tasks/service-catalog/install-service-catalog-using-helm/) into your Kubernetes cluster. Alternatively, you can [install Service Catalog using the SC tool](/docs/tasks/service-catalog/install-service-catalog-using-sc/).
|
||||
* View [sample service brokers](https://github.com/openservicebrokerapi/servicebroker/blob/master/gettingStarted.md#sample-service-brokers).
|
||||
* Explore the [kubernetes-incubator/service-catalog](https://github.com/kubernetes-incubator/service-catalog) project.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user