Release 1.7 (#4094)
* Minor fixes in the Deployment doc Signed-off-by: Michail Kargakis <mkargaki@redhat.com> * add NodeRestriction to admission-controllers (#3842) * Admins Can Configure Zones in Storage Class The PR #38505 (https://github.com/kubernetes/kubernetes/pull/38505) added zones optional parameter to Storage Class for AWS and GCE provisioners. That's why documentation needs to be updated accordingly. * document custom resource definitions * add host paths to psp (#3971) * add host paths to psp * add italics * Update ConfigMap doc to explain TTL-based cache updates (#3989) * Update ConfigMap doc to explain TTL-based cache updates * swap word order Change "When a ConfigMap being already consumed..." to "When a ConfigMap already being consumed..." * Update NetworkPolicy docs for v1 * StorageOS Volume plugin * Update GPU docs * docs: HPA autoscaling/v2alpha1 status conditions This commit documents the new status conditions feature for HPA autoscaling/v2alpha1. It demonstrates how to get the status conditions using `kubectl describe`, and how to interpret them. * Update description about NodeRestriction kubelet node can alse create mirror pods for their own static pods. * adding storage as a supported resource to node allocatable Signed-off-by: Vishnu kannan <vishnuk@google.com> * Add documentation for podpreset opt-out annotation This adds the annotation for having the podpreset admission controller to skip (opt-out) manipulating the pod spec. Also, the annotation format for what presets have acted on a pod has been modified to add a prefix of "podpreset-". The new naming makes it such that there is no chance of collision with the newly introduced opt-out annotation (or future ones yet to be added). Opt-out annotation PR: kubernetes/kubernetes#44965 * Update PDB documentation to explain new field (#3885) * update-docs-pdb * Addressed erictune@'s comments * Fix title and add a TOC to the logging concept page * Patch #4118 for typos * Describe setting coredns server in nameserver resolv chain * Address comments in PR #3997. Comment is in https://github.com/kubernetes/kubernetes.github.io/pull/3997/files/f6eb59c67e28efc298c87b1ef49a96bc6adacd1e#diff-7a14981f3dd8eb203f897ce6c11d9828 * Update task for DaemonSet history and rollback (#4098) * Update task for DaemonSet history and rollback Also remove mentions of templateGeneration field because it's deprecated * Address comments * removed lt and gt as operators (#4152) * removed lt and gt as operators * replace lt and gt for node-affinfity * updated based on bsalamat review * Initial draft of upgrade guide for kubeadm clusters. In-place upgrades are supported between 1.6 and 1.7 releases. Rollback instructions to come in a separate commit. Fixes https://github.com/kubernetes/kubeadm/issues/278 * Add local volume documentation (#4050) * Add local volume documentation * Add PV local volume example * Patch PR #3999 * Add documentation for Stackdriver event exporter * Add documentation about controller metrics * Federation: Add task for setting up placement policies (#4075) * Add task for setting up placement policies * Update version of management sidecar in policy engine deployment * Address @nikhiljindal's comments - Lower case filenames - Comments in policy - Typo fixes - Removed type LoadBalancer from OPA Service * Add example that sets cluster selector Per-@nikhiljindal's suggestion * Fix wording and templating per @chenopis * PodDisruptionBudget documentation Improvements (#4140) * Changes from #3885 Title: Update PDB documentation to explain new field Author: foxish * Added Placeholder Disruptions Concept Guide New file: docs/concepts/workloads/pods/disruptions.md Intented contents: concept for Pod Disruption Budget, cross reference to Eviction and Preemption docs. Linked from: concepts > workloads > pods * Added placeholder Configuring PDB Task New file: docs/tasks/run-application/configure-pdb.md Intented contents: task for writing a Pod Disruption Budget. Linked from: tasks > configuring-applications > configure pdb. * Add refs to the "drain a node" task. * Refactor PDB docs. Move the "Requesting an eviction" section from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md -- which is going away -- to: docs/tasks/administer-cluster/safely-drain-node.md The move is verbatim, except for an introductory sentence. Also added assignees. * Refactor of PDB docs Moved the section: Specifying a PodDisruptionBudget from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md to: docs/tasks/run-application/configure-pdb.md because that former file is going away. Move is verbatim. * Explain how Eviction tools should handle failures * Refactor PDB docs Move text from: docs/tasks/administer-cluster/configure-pod-disruption-budget.md to: docs/concepts/workloads/pods/disruptions.md Delete the now empty: docs/tasks/administer-cluster/configure-pod-disruption-budget.md Added a redirects_from section to the new doc, containing the path of the now-deleted doc, plus all the redirects from the deleted doc. * Expand PDB Concept guide Building on a little content from the old task, greatly expanded the Disruptions concept guide, including an abstract example. * Update creating a pdb Task. * Address review comments. * Fixed for all cody-clark's review comments * Address review comments from mml * Address review comments from maisem * Fix missing backtick * Api and Kubectl reference docs updates for 1.7 (#4193) * Fix includes groups * Generated kubectl docs for 1.7 * Generated references docs for 1.7 api * Document node authorization mode * API Aggregator (#4173) * API Aggregator * Additional bullet points * incorporated feedback for apiserver-aggregation.md * split setup-api-aggregator.md into two docs and address feedback * fix link * addressed docs feedback * incorporate feedback * integrate feedback * Add documentation for DNS stub domains (#4063) * Add documentation for DNS stub domains * add additional prereq * fix image path * review feedback * minor grammar and style nits * documentation for using hostAliases to manage hosts file (#4080) * documentation for using hostAliases to manage hosts file * add to table of contents * review comments * update the right command to see hosts file * reformat doc based on suggestion and change some wording * Fix typo for #4080 * Patch PR #4063 * Fix wording in placement policy task introduction * Add update to statefulset concepts and basic tutorial (#4174) * Add update to statefulset concpets and basic tutorial * Address tech comments. * Update ESIPP docs for new added API fields * Custom resource docs * update audit document with advanced audit features added in 1.7 * kubeadm v1.7 documentation updates (#4018) * v1.7 updates for kubeadm * Address review comments * Address Luke's comments * Encrypting secrets at rest and cluster security guide * Edits for Custom DNS Documentation (#4207) * reorganize custom dns doc * format fixes * Update version numbers to 1.7 * Patch PR #4140 (#4215) * Patch PR #4140 * fix link and typos * Update PR template * Update TLS bootstrapping with 1.7 features This includes documenting the new CSR approver built into the controller manager and the kubelet alpha features for certificate rotation. Since the CSR approver changed over the 1.7 release cycle we need to call out the migration steps for those using the alpha feature. This document as a whole could probably use some updates, but the main focus of this PR is just to get these features minimally documented before the release. * Federated ClusterSelector formatting updates from review * complete PR #4181 (#4223) * complete PR #4181 * fix security link * Extensible admission controller (#4092) * extensible-admission-controllers * Update extensible-admission-controllers.md * more on initializers * fixes * Expand external admission webhooks documentation * wrap at 80 chars * more * add reference * Use correct apigroup for network policy * Docs changes to PR #4092 (#4224) * Docs changes to PR #4092 * address feedback * add doc for --as-group in cli Add doc for this pr: https://github.com/kubernetes/kubernetes/pull/43696
This commit is contained in:
zhangchaolei
@@ -81,7 +81,9 @@ information through `kubectl`:
|
||||
```bash
|
||||
kubectl describe services example-service
|
||||
```
|
||||
|
||||
which should produce output like this:
|
||||
|
||||
```bash
|
||||
Name: example-service
|
||||
Selector: app=example
|
||||
@@ -97,10 +99,29 @@ which should produce output like this:
|
||||
|
||||
The IP address is listed next to `LoadBalancer Ingress`.
|
||||
|
||||
## Annotation to modify the LoadBalancer behavior for preservation of Source IP
|
||||
In 1.5, a Beta feature has been added that changes the behavior of the external LoadBalancer feature.
|
||||
## Preserving the client source IP
|
||||
|
||||
This feature can be activated by adding the beta annotation below to the metadata section of the Service Configuration file.
|
||||
Due to the implementation of this feature, the source IP seen in the target
|
||||
container will *not be the original source IP* of the client. To enable
|
||||
preservation of the client IP, the following fields can be configured in the
|
||||
service spec (supported in GCE/GKE environments):
|
||||
|
||||
* `service.spec.externalTrafficPolicy` - denotes if this Service desires to route
|
||||
external traffic to node-local or cluster-wide endpoints. There are two available
|
||||
options: "Cluster" (default) and "Local". "Cluster" obscures the client source
|
||||
IP and may cause a second hop to another node, but should have good overall
|
||||
load-spreading. "Local" preserves the client source IP and avoids a second hop
|
||||
for LoadBalancer and NodePort type services, but risks potentially imbalanced
|
||||
traffic spreading.
|
||||
* `service.spec.healthCheckNodePort` - specifies the healthcheck nodePort
|
||||
(numeric port number) for the service. If not specified, healthCheckNodePort is
|
||||
created by the service API backend with the allocated nodePort. It will use the
|
||||
user-specified nodePort value if specified by the client. It only has an
|
||||
effect when type is set to "LoadBalancer" and externalTrafficPolicy is set
|
||||
to "Local".
|
||||
|
||||
This feature can be activated by setting `externalTrafficPolicy` to "Local" in the
|
||||
Service Configuration file.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -108,9 +129,6 @@ This feature can be activated by adding the beta annotation below to the metadat
|
||||
"apiVersion": "v1",
|
||||
"metadata": {
|
||||
"name": "example-service",
|
||||
"annotations": {
|
||||
"service.beta.kubernetes.io/external-traffic": "OnlyLocal"
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"ports": [{
|
||||
@@ -120,11 +138,33 @@ This feature can be activated by adding the beta annotation below to the metadat
|
||||
"selector": {
|
||||
"app": "example"
|
||||
},
|
||||
"type": "LoadBalancer"
|
||||
"type": "LoadBalancer",
|
||||
"externalTrafficPolicy": "Local"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Feature availability
|
||||
|
||||
| k8s version | Feature support |
|
||||
| :---------: |:-----------:|
|
||||
| 1.7+ | Supports the full API fields |
|
||||
| 1.5 - 1.6 | Supports Beta Annotations |
|
||||
| <1.5 | Unsupported |
|
||||
|
||||
Below you could find the deprecated Beta annotations used to enable this feature
|
||||
prior to its stable version. Newer Kubernetes versions may stop supporting these
|
||||
after v1.7. Please update existing applications to use the fields directly.
|
||||
|
||||
* `service.beta.kubernetes.io/external-traffic` annotation <-> `service.spec.externalTrafficPolicy` field
|
||||
* `service.beta.kubernetes.io/healthcheck-nodeport` annotation <-> `service.spec.healthCheckNodePort` field
|
||||
|
||||
`service.beta.kubernetes.io/external-traffic` annotation has a different set of values
|
||||
compared to the `service.spec.externalTrafficPolicy` field. The values match as follows:
|
||||
|
||||
* "OnlyLocal" for annotation <-> "Local" for field
|
||||
* "Global" for annotation <-> "Cluster" for field
|
||||
|
||||
**Note that this feature is not currently implemented for all cloudproviders/environments.**
|
||||
|
||||
|
||||
@@ -138,11 +178,6 @@ It is important to note that the datapath for this functionality is provided by
|
||||
|
||||
When the service type is set to `LoadBalancer`, Kubernetes provides functionality equivalent to `type=<ClusterIP>` to pods within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the Kubernetes VMs. The Kubernetes service controller automates the creation of the external load balancer, health checks (if needed), firewall rules (if needed) and retrieves the external IP allocated by the cloud provider and populates it in the service object.
|
||||
|
||||
## Loss of client source IP for external traffic
|
||||
|
||||
Due to the implementation of this feature, the source IP for sessions as seen in the target container will *not be the original source IP* of the client. This is the default behavior as of Kubernetes v1.5. However, starting in v1.5, an optional beta feature has been added
|
||||
that will preserve the client Source IP for GCE/GKE environments. This feature will be phased in for other cloud providers in subsequent releases.
|
||||
|
||||
## Caveats and Limitations when preserving source IPs
|
||||
|
||||
GCE/AWS load balancers do not provide weights for their target pools. This was not an issue with the old LB
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Configure the aggregation layer
|
||||
assignees:
|
||||
- lavalamp
|
||||
- cheftako
|
||||
- chenopis
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
Configuring the [aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/) allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
{% include task-tutorial-prereqs.md %}
|
||||
|
||||
**Note:** There are a few setup requirements for getting the aggregation layer working in your environment to support mutual TLS auth between the proxy and extension apiservers. Kubernetes and the kube-apiserver have multiple CAs, so make sure that the proxy is signed by the aggregation layer CA and not by something else, like the master CA.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Enable apiserver flags
|
||||
|
||||
Enable the aggregation layer via the following kube-apiserver flags. They may have already been taken care of by your provider.
|
||||
|
||||
--requestheader-client-ca-file=<path to aggregator CA cert>
|
||||
--requestheader-allowed-names=aggregator
|
||||
--requestheader-extra-headers-prefix=X-Remote-Extra-
|
||||
--requestheader-group-headers=X-Remote-Group
|
||||
--requestheader-username-headers=X-Remote-User
|
||||
--proxy-client-cert-file=<path to aggregator proxy cert>
|
||||
--proxy-client-key-file=<path to aggregator proxy key>
|
||||
|
||||
The [Kubernetes Architectural Roadmap](https://docs.google.com/a/google.com/document/d/1XkjVm4bOeiVkj-Xt1LgoGiqWsBfNozJ51dyI-ljzt1o/edit?usp=sharing) recommends not running kube-proxy on the master. If you follow this recommendation, then you must make sure that the system is enabled with the following apiserver flag. Again, this may have already been taken care of by your provider.
|
||||
|
||||
--enable-aggregator-routing=true
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
|
||||
* [Setup an extension api-server](/docs/tasks/access-kubernetes-api/setup-extension-api-server/) to work with the aggregation layer.
|
||||
* For a high level overview, see [Extending the Kubernetes API with the aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/).
|
||||
* Learn how to [Extend the Kubernetes API Using Custom Resource Definitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
---
|
||||
title: Extend the Kubernetes API with CustomResourceDefinitions
|
||||
assignees:
|
||||
- deads2k
|
||||
- enisoc
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
This page shows how to install a [custom resource](/docs/concepts/api-extension/custom-resources/)
|
||||
into the Kubernetes API by creating a CustomResourceDefinition.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
* Read about [custom resources](/docs/concepts/api-extension/custom-resources/).
|
||||
* Make sure your Kubernetes cluster has a master version of 1.7.0 or higher.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
## Create a CustomResourceDefinition
|
||||
|
||||
When you create a new *CustomResourceDefinition* (CRD), the Kubernetes API Server
|
||||
reacts by creating a new RESTful resource path, either namespaced or cluster-scoped,
|
||||
as specified in the CRD's `scope` field. As with existing built-in objects, deleting a
|
||||
namespace deletes all custom objects in that namespace.
|
||||
CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.
|
||||
|
||||
For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
# name must match the spec fields below, and be in the form: <plural>.<group>
|
||||
name: crontabs.stable.example.com
|
||||
spec:
|
||||
# group name to use for REST API: /apis/<group>/<version>
|
||||
group: stable.example.com
|
||||
# version name to use for REST API: /apis/<group>/<version>
|
||||
version: v1
|
||||
# either Namespaced or Cluster
|
||||
scope: Namespaced
|
||||
names:
|
||||
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
|
||||
plural: crontabs
|
||||
# singular name to be used as an alias on the CLI and for display
|
||||
singular: crontab
|
||||
# kind is normally the CamelCased singular type. Your resource manifests use this.
|
||||
kind: CronTab
|
||||
# shortNames allow shorter string to match your resource on the CLI
|
||||
shortNames:
|
||||
- ct
|
||||
```
|
||||
|
||||
And create it:
|
||||
|
||||
```shell
|
||||
kubectl create -f resourcedefinition.yaml
|
||||
```
|
||||
|
||||
Then a new namespaced RESTful API endpoint is created at:
|
||||
|
||||
```
|
||||
/apis/stable.example.com/v1/namespaces/*/crontabs/...
|
||||
```
|
||||
|
||||
This endpoint URL can then be used to create and manage custom objects.
|
||||
The `kind` of these objects will be `CronTab` from the spec of the
|
||||
CustomResourceDefinition object you created above.
|
||||
|
||||
|
||||
## Create custom objects
|
||||
|
||||
After the CustomResourceDefinition object has been created, you can create
|
||||
custom objects. Custom objects can contain custom fields. These fields can
|
||||
contain arbitrary JSON.
|
||||
In the following example, the `cronSpec` and `image` custom fields are set in a
|
||||
custom object of kind `CronTab`. The kind `CronTab` comes from the spec of the
|
||||
CustomResourceDefinition object you created above.
|
||||
|
||||
If you save the following YAML to `my-crontab.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: "stable.example.com/v1"
|
||||
kind: CronTab
|
||||
metadata:
|
||||
name: my-new-cron-object
|
||||
spec:
|
||||
cronSpec: "* * * * /5"
|
||||
image: my-awesome-cron-image
|
||||
```
|
||||
|
||||
and create it:
|
||||
|
||||
```shell
|
||||
kubectl create -f my-crontab.yaml
|
||||
```
|
||||
|
||||
You can then manage your CronTab objects using kubectl. For example:
|
||||
|
||||
```shell
|
||||
kubectl get crontab
|
||||
```
|
||||
|
||||
Should print a list like this:
|
||||
|
||||
```console
|
||||
NAME KIND
|
||||
my-new-cron-object CronTab.v1.stable.example.com
|
||||
```
|
||||
|
||||
Note that resource names are not case-sensitive when using kubectl,
|
||||
and you can use either the singular or plural forms defined in the CRD,
|
||||
as well as any short names.
|
||||
|
||||
You can also view the raw JSON data:
|
||||
|
||||
```shell
|
||||
kubectl get ct -o yaml
|
||||
```
|
||||
|
||||
You should see that it contains the custom `cronSpec` and `image` fields
|
||||
from the yaml you used to create it:
|
||||
|
||||
```console
|
||||
apiVersion: v1
|
||||
items:
|
||||
- apiVersion: stable.example.com/v1
|
||||
kind: CronTab
|
||||
metadata:
|
||||
clusterName: ""
|
||||
creationTimestamp: 2017-05-31T12:56:35Z
|
||||
deletionGracePeriodSeconds: null
|
||||
deletionTimestamp: null
|
||||
name: my-new-cron-object
|
||||
namespace: default
|
||||
resourceVersion: "285"
|
||||
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
|
||||
uid: 9423255b-4600-11e7-af6a-28d2447dc82b
|
||||
spec:
|
||||
cronSpec: '* * * * /5'
|
||||
image: my-awesome-cron-image
|
||||
kind: List
|
||||
metadata:
|
||||
resourceVersion: ""
|
||||
selfLink: ""
|
||||
```
|
||||
{% endcapture %}
|
||||
|
||||
{% capture discussion %}
|
||||
## Advanced topics
|
||||
|
||||
### Finalizers
|
||||
|
||||
*Finalizers* allow controllers to implement asynchronous pre-delete hooks.
|
||||
Custom objects support finalizers just like built-in objects.
|
||||
|
||||
You can add a finalizer to a custom object like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: "stable.example.com/v1"
|
||||
kind: CronTab
|
||||
metadata:
|
||||
finalizers:
|
||||
- finalizer.stable.example.com
|
||||
```
|
||||
|
||||
The first delete request on an object with finalizers merely sets a value for the
|
||||
`metadata.deletionTimestamp` field instead of deleting it.
|
||||
This triggers controllers watching the object to execute any finalizers they handle.
|
||||
|
||||
Each controller then removes its finalizer from the list and issues the delete request again.
|
||||
This request only deletes the object if the list of finalizers is now empty,
|
||||
meaning all finalizers are done.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
* Learn how to [Migrate a ThirdPartyResource to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/).
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
---
|
||||
assignees:
|
||||
- enisoc
|
||||
- IanLewis
|
||||
title: Extend the Kubernetes API Using Third Party Resources
|
||||
title: Extend the Kubernetes API with ThirdPartyResources
|
||||
redirect_from:
|
||||
- "/docs/user-guide/thirdpartyresources/"
|
||||
- "/docs/user-guide/thirdpartyresources.html"
|
||||
@@ -9,15 +10,21 @@ redirect_from:
|
||||
- "/docs/concepts/ecosystem/thirdpartyresource.html"
|
||||
---
|
||||
|
||||
{% assign for_k8s_version="1.7" %}{% include feature-state-deprecated.md %}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## What is ThirdPartyResource?
|
||||
|
||||
**WARNING: ThirdPartyResources are not yet considered stable, and the API and/or storage could change before GA.
|
||||
Development and outstanding issues are tracked at [https://github.com/kubernetes/features/issues/95](https://github.com/kubernetes/features/issues/95).**
|
||||
**ThirdPartyResource is deprecated as of Kubernetes 1.7 and may be removed in version 1.8 in
|
||||
accordance with the [deprecation policy](/docs/reference/deprecation-policy) for beta features.**
|
||||
|
||||
Kubernetes comes with many built-in API objects. However, there are often times when you might need to extend Kubernetes with their own API objects in order to do custom automation.
|
||||
**To avoid losing data stored in ThirdPartyResources, you must
|
||||
[migrate to CustomResourceDefinition](/docs/tasks/access-kubernetes-api/migrate-third-party-resource/)
|
||||
before upgrading to Kubernetes 1.8 or higher.**
|
||||
|
||||
Kubernetes comes with many built-in API objects. However, there are often times when you might need to extend Kubernetes with your own API objects in order to do custom automation.
|
||||
|
||||
`ThirdPartyResource` objects are a way to extend the Kubernetes API with a new API object type. The new API object type will be given an API endpoint URL and support CRUD operations, and watch API. You can then create custom objects using this API endpoint. You can think of `ThirdPartyResources` as being much like the schema for a database table. Once you have created the table, you can then start storing rows in the table. Once created, `ThirdPartyResources` can act as the data model behind custom controllers or automation programs.
|
||||
|
||||
@@ -69,12 +76,12 @@ Then a new RESTful API endpoint is created at:
|
||||
|
||||
This endpoint URL can then be used to create and manage custom objects.
|
||||
The `kind` of these objects will be `CronTab` following the camel case
|
||||
rules applied to the `metadata.name` of this `ThirdPartyResource`
|
||||
rules applied to the `metadata.name` of this `ThirdPartyResource`
|
||||
(`cron-tab.stable.example.com`)
|
||||
|
||||
## Creating Custom Objects
|
||||
|
||||
After the `ThirdPartyResource` object has been created you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON.
|
||||
After the `ThirdPartyResource` object has been created you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON.
|
||||
In the following example, a `cronSpec` and `image` custom fields are set to the custom object of kind `CronTab`. The kind `CronTab` is derived from the
|
||||
`metadata.name` of the `ThirdPartyResource` object we created above.
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: Migrate a ThirdPartyResource to CustomResourceDefinition
|
||||
assignees:
|
||||
- enisoc
|
||||
- deads2k
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
This page shows how to migrate data stored in a ThirdPartyResource (TPR) to a CustomResourceDefinition (CRD).
|
||||
|
||||
Kubernetes does not automatically migrate existing TPRs.
|
||||
This is due to API changes introduced as part of
|
||||
[graduating to beta](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md)
|
||||
under a new name and API group.
|
||||
Instead, both TPR and CRD are available and operate independently in Kubernetes 1.7.
|
||||
Users must migrate each TPR one by one to preserve their data before upgrading to Kubernetes 1.8.
|
||||
|
||||
The simplest way to migrate is to stop all clients that use a given TPR, then delete the TPR and
|
||||
start from scratch with a CRD.
|
||||
This page describes an optional process that eases the transition by migrating existing TPR data for
|
||||
you **on a best-effort basis**.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
* Make sure your Kubernetes cluster has a **master version of exactly 1.7.x** (any patch release),
|
||||
as this is the only version that supports both TPR and CRD.
|
||||
* If you use a TPR-based custom controller, check with the author of the controller first.
|
||||
Some or all of these steps may be unnecessary if the custom controller handles the migration for
|
||||
you.
|
||||
* Be familiar with the concept of [custom resources](/docs/concepts/api-extension/custom-resources/),
|
||||
which were known as *third-party resources* until Kubernetes 1.7.
|
||||
* Be familiar with [CustomResourceDefinitions](/docs/concepts/api-extension/custom-resources/#customresourcedefinitions),
|
||||
which are a simple way to implement custom resources.
|
||||
* **Before performing a migration on real data, conduct a dry run by going through these steps in a test cluster.**
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
## Migrate TPR data
|
||||
|
||||
1. **Rewrite the TPR definition**
|
||||
|
||||
Clients that access the REST API for your custom resource should not need any changes.
|
||||
However, you will need to rewrite your TPR definition as a CRD.
|
||||
|
||||
Make sure you specify values for the CRD fields that match what the server used to fill in for
|
||||
you with TPR.
|
||||
|
||||
For example, if your ThirdPartyResource looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: ThirdPartyResource
|
||||
metadata:
|
||||
name: cron-tab.stable.example.com
|
||||
description: "A specification of a Pod to run on a cron style schedule"
|
||||
versions:
|
||||
- name: v1
|
||||
```
|
||||
|
||||
A matching CustomResourceDefinition could look like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
name: crontabs.stable.example.com
|
||||
spec:
|
||||
scope: Namespaced
|
||||
group: stable.example.com
|
||||
version: v1
|
||||
names:
|
||||
kind: CronTab
|
||||
plural: crontabs
|
||||
singular: crontab
|
||||
```
|
||||
|
||||
1. **Install the CustomResourceDefinition**
|
||||
|
||||
While the source TPR is still active, install the matching CRD with `kubectl create`.
|
||||
Existing TPR data remains accessible because TPRs take precedence over CRDs when both try
|
||||
to serve the same resource.
|
||||
|
||||
After you create the CRD, make sure the *Established* condition goes to True.
|
||||
You can check it with a command like this:
|
||||
|
||||
```shell
|
||||
kubectl get crd -o 'custom-columns=NAME:{.metadata.name},ESTABLISHED:{.status.conditions[?(@.type=="Established")].status}'
|
||||
```
|
||||
|
||||
The output should look like this:
|
||||
|
||||
```console
|
||||
NAME ESTABLISHED
|
||||
crontabs.stable.example.com True
|
||||
```
|
||||
|
||||
1. **Stop all clients that use the TPR**
|
||||
|
||||
The API server attempts to prevent TPR data for the resource from changing while it
|
||||
copies objects to the CRD, but it can't guarantee consistency in all cases, such as with
|
||||
[multiple masters](/docs/admin/high-availability/).
|
||||
Stopping clients, such as TPR-based custom controllers, helps to avoid inconsistencies in
|
||||
the copied data.
|
||||
|
||||
In addition, clients that watch TPR data do not receive any more events once the migration
|
||||
begins.
|
||||
You must restart them after the migration completes so they start watching CRD data instead.
|
||||
|
||||
1. **Back up TPR data**
|
||||
|
||||
In case the data migration fails, save a copy of existing data for the resource:
|
||||
|
||||
```shell
|
||||
kubectl get crontabs --all-namespaces -o yaml > crontabs.yaml
|
||||
```
|
||||
|
||||
You should also save a copy of the TPR definition if you don't have one already:
|
||||
|
||||
```shell
|
||||
kubectl get thirdpartyresource cron-tab.stable.example.com -o yaml --export > tpr.yaml
|
||||
```
|
||||
|
||||
1. **Delete the TPR definition**
|
||||
|
||||
Normally, when you delete a TPR definition, the API server tries to clean up any objects stored
|
||||
in that resource.
|
||||
Because a matching CRD exists, the server copies objects to the CRD instead of deleting them.
|
||||
|
||||
```shell
|
||||
kubectl delete thirdpartyresource cron-tab.stable.example.com
|
||||
```
|
||||
|
||||
1. **Verify the new CRD data**
|
||||
|
||||
It can take up to 10 seconds for the TPR controller to notice when you delete the TPR definition
|
||||
and to initiate the migration. The TPR data remains accessible during this time.
|
||||
|
||||
Once the migration completes, the resource begins serving through the CRD.
|
||||
Check that all your objects were correctly copied:
|
||||
|
||||
```shell
|
||||
kubectl get crontabs --all-namespaces -o yaml
|
||||
```
|
||||
|
||||
If the copy failed, you can quickly revert to the set of objects that existed just before the
|
||||
migration by recreating the TPR definition:
|
||||
|
||||
```shell
|
||||
kubectl create -f tpr.yaml
|
||||
```
|
||||
|
||||
1. **Restart clients**
|
||||
|
||||
After verifying the CRD data, restart any clients you stopped before the migration, such as
|
||||
custom controllers and other watchers.
|
||||
These clients now access CRD data when they make requests on the same API endpoints
|
||||
that the TPR previously served.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
* Learn more about [custom resources](/docs/concepts/api-extension/custom-resources/).
|
||||
* Learn more about [using CustomResourceDefinitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: Setup an extension API server
|
||||
assignees:
|
||||
- lavalamp
|
||||
- cheftako
|
||||
- chenopis
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
Setting up an extension API server to work the aggregation layer allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
* You need to have a Kubernetes cluster running.
|
||||
* You must [configure the aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/) and enable the apiserver flags.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Setup an extension api-server to work with the aggregation layer
|
||||
|
||||
The following steps describe how to set up an extension-apiserver *at a high level*. For a concrete example of how they can be implemented, you can look at the [sample-apiserver](https://github.com/kubernetes/sample-apiserver/blob/master/README.md) in the Kubernetes repo.
|
||||
|
||||
Alternatively, you can use an existing 3rd party solution, such as [apiserver-builder](https://github.com/Kubernetes-incubator/apiserver-builder/blob/master/README.md), which should generate a skeleton and automate all of the following steps for you.
|
||||
|
||||
1. Make sure the APIService API is enabled (check `--runtime-config`). It should be on by default, unless it's been deliberately turned off in your cluster.
|
||||
1. You may need to make an RBAC rule allowing you to add APIService objects, or get your cluster administrator to make one. (Since API extensions affect the entire cluster, it is not recommended to do testing/development/debug of an API extension in a live cluster.)
|
||||
1. Create the Kubernetes namespace you want to run your extension api-service in.
|
||||
1. Create/get a CA cert to be used to sign the server cert the extension api-server uses for HTTPS.
|
||||
1. Create a server cert/key for the api-server to use for HTTPS. This cert should be signed by the above CA. It should also have a CN of the Kube DNS name. This is derived from the Kubernetes service and be of the form <service name>.<service name namespace>.svc
|
||||
1. Create a Kubernetes secret with the server cert/key in your namespace.
|
||||
1. Create a Kubernetes deployment for the extension api-server and make sure you are loading the secret as a volume. It should contain a reference to a working image of your extension api-server. The deployment should also be in your namespace.
|
||||
1. Make sure that your extension-apiserver loads those certs from that volume and that they are used in the HTTPS handshake.
|
||||
1. Create a Kubernetes service account in your namespace.
|
||||
1. Create a Kubernetes cluster role for the operations you want to allow on your resources.
|
||||
1. Create a Kubernetes cluster role binding from the default service account in your namespace to the cluster role you just created.
|
||||
1. Create a Kubernetes apiservice. The CA cert above should be base 64 encoded, stripped of new lines and used as the spec.caBundle in the apiservce. This should not be namespaced.
|
||||
1. Use kubectl to get your resource. It should return "No resources found." Which means that everything worked but you currently have no objects of that resource type created yet.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
|
||||
* If you haven't already, [configure the aggregation layer](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/) and enable the apiserver flags.
|
||||
* For a high level overview, see [Extending the Kubernetes API with the aggregation layer](/docs/concepts/api-extension/apiserver-aggregation/).
|
||||
* Learn how to [Extend the Kubernetes API Using Custom Resource Definitions](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/).
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
|
||||
@@ -1,116 +0,0 @@
|
||||
---
|
||||
assignees:
|
||||
- davidopp
|
||||
title: Configure a Pod Disruption Budget
|
||||
redirect_from:
|
||||
- "/docs/admin/disruptions/"
|
||||
- "/docs/admin/disruptions.html"
|
||||
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
|
||||
- "/docs/tasks/configure-pod-container/configure-pod-disruption-budget/"
|
||||
---
|
||||
|
||||
This guide is for anyone wishing to specify safety constraints on pods or anyone
|
||||
wishing to write software (typically automation software) that respects those
|
||||
constraints.
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Rationale
|
||||
|
||||
Various cluster management operations may voluntarily evict pods. "Voluntary"
|
||||
means an eviction can be safely delayed for a reasonable period of time. The
|
||||
principal examples today are draining a node for maintenance or upgrade
|
||||
(`kubectl drain`), and cluster autoscaling down. In the future the
|
||||
[rescheduler](https://git.k8s.io/community/contributors/design-proposals/rescheduling.md)
|
||||
may also perform voluntary evictions. By contrast, something like evicting pods
|
||||
because a node has become unreachable or reports `NotReady`, is not "voluntary."
|
||||
|
||||
For voluntary evictions, it can be useful for applications to be able to limit
|
||||
the number of pods that are down simultaneously. For example, a quorum-based application would
|
||||
like to ensure that the number of replicas running is never brought below the
|
||||
number needed for a quorum, even temporarily. Or a web front end might want to
|
||||
ensure that the number of replicas serving load never falls below a certain
|
||||
percentage of the total, even briefly. `PodDisruptionBudget` is an API object
|
||||
that specifies the minimum number or percentage of replicas of a collection that
|
||||
must be up at a time. Components that wish to evict a pod subject to disruption
|
||||
budget use the `/eviction` subresource; unlike a regular pod deletion, this
|
||||
operation may be rejected by the API server if the eviction would cause a
|
||||
disruption budget to be violated.
|
||||
|
||||
## Specifying a PodDisruptionBudget
|
||||
|
||||
A `PodDisruptionBudget` has two components: a label selector `selector` to specify the set of
|
||||
pods to which it applies, and `minAvailable` which is a description of the number of pods from that
|
||||
set that must still be available after the eviction, i.e. even in the absence
|
||||
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
|
||||
So for example, 100% means no voluntary evictions from the set are permitted. In
|
||||
typical usage, a single budget would be used for a collection of pods managed by
|
||||
a controller—for example, the pods in a single ReplicaSet.
|
||||
|
||||
Note that a disruption budget does not truly guarantee that the specified
|
||||
number/percentage of pods will always be up. For example, a node that hosts a
|
||||
pod from the collection may fail when the collection is at the minimum size
|
||||
specified in the budget, thus bringing the number of available pods from the
|
||||
collection below the specified size. The budget can only protect against
|
||||
voluntary evictions, not all causes of unavailability.
|
||||
|
||||
You can find an example of a pod disruption budget defined below. It matches pods with the label
|
||||
`app: zookeeper`.
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: zk-pdb
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: zookeeper
|
||||
```
|
||||
|
||||
## Requesting an eviction
|
||||
|
||||
If you are writing infrastructure software that wants to produce these voluntary
|
||||
evictions, you will need to use the eviction API. The eviction subresource of a
|
||||
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
|
||||
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
|
||||
*create* an eviction), you POST an attempted operation. Here's an example:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1beta1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
You can attempt an eviction using `curl`:
|
||||
|
||||
```bash
|
||||
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
|
||||
```
|
||||
|
||||
The API can respond in one of three ways.
|
||||
|
||||
1. If the eviction is granted, then the pod is deleted just as if you had sent
|
||||
a `DELETE` request to the pod's URL and you get back `200 OK`.
|
||||
2. If the current state of affairs wouldn't allow an eviction by the rules set
|
||||
forth in the budget, you get back `429 Too Many Requests`. This is
|
||||
typically used for generic rate limiting of *any* requests, but here we mean
|
||||
that this request isn't allowed *right now* but it may be allowed later.
|
||||
Currently, callers do not get any `Retry-After` advice, but they may in
|
||||
future versions.
|
||||
3. If there is some kind of misconfiguration, like multiple budgets pointing at
|
||||
the same pod, you will get `500 Internal Server Error`.
|
||||
|
||||
For a given eviction request, there are two cases.
|
||||
|
||||
1. There is no budget that matches this pod. In this case, the server always
|
||||
returns `200 OK`.
|
||||
2. There is at least one budget. In this case, any of the three above responses may
|
||||
apply.
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
assignees:
|
||||
- caseydavenport
|
||||
- danwinship
|
||||
title: Declare Network Policy
|
||||
redirect_from:
|
||||
- "/docs/getting-started-guides/network-policy/walkthrough/"
|
||||
@@ -68,35 +69,11 @@ Connecting to nginx (10.100.0.16:80)
|
||||
|
||||
## Limit access to the `nginx` service
|
||||
|
||||
Let's say you want to limit access to the `nginx` service so that only pods with the label `access: true` can query it. The first step is to enable ingress isolation on the `default` namespace. This prevents **_any_** pods from accessing the `nginx` service.
|
||||
|
||||
```console
|
||||
$ kubectl annotate ns default "net.beta.kubernetes.io/network-policy={\"ingress\": {\"isolation\": \"DefaultDeny\"}}"
|
||||
```
|
||||
|
||||
## Test the access limitation
|
||||
|
||||
Test to see that with ingress isolation in place, you no longer have access to the `nginx` service:
|
||||
|
||||
```console
|
||||
$ kubectl run busybox --rm -ti --image=busybox /bin/sh
|
||||
Waiting for pod default/busybox-472357175-y0m47 to be running, status is Pending, pod ready: false
|
||||
|
||||
Hit enter for command prompt
|
||||
|
||||
/ # wget --spider --timeout=1 nginx
|
||||
Connecting to nginx (10.100.0.16:80)
|
||||
wget: download timed out
|
||||
/ #
|
||||
```
|
||||
|
||||
## Create a policy that allows connections from authorized pods
|
||||
|
||||
Next, create a `NetworkPolicy` that allows connections from pods with the label `access: true`.
|
||||
Let's say you want to limit access to the `nginx` service so that only pods with the label `access: true` can query it. To do that, create a `NetworkPolicy` that allows connections only from those pods:
|
||||
|
||||
```yaml
|
||||
kind: NetworkPolicy
|
||||
apiVersion: extensions/v1beta1
|
||||
apiVersion: networking/v1
|
||||
metadata:
|
||||
name: access-nginx
|
||||
spec:
|
||||
@@ -119,7 +96,7 @@ networkpolicy "access-nginx" created
|
||||
```
|
||||
|
||||
## Test access to the service when access label is not defined
|
||||
If we attempt to access the nginx Service from a pod without the correct labels, the request will still time out:
|
||||
If we attempt to access the nginx Service from a pod without the correct labels, the request will now time out:
|
||||
|
||||
```console
|
||||
$ kubectl run busybox --rm -ti --image=busybox /bin/sh
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
assignees:
|
||||
- bowei
|
||||
- zihongz
|
||||
title: Configure private DNS zones and upstream nameservers in Kubernetes
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
This page shows how to add custom private DNS zones (stub domains) and upstream
|
||||
nameservers.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
* {% include task-tutorial-prereqs.md %}
|
||||
* Kubernetes version 1.6 and above.
|
||||
* The cluster must be configured to use the `kube-dns` addon.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Configure stub-domain and upstream DNS servers
|
||||
|
||||
Cluster administrators can specify custom stub domains and upstream nameservers
|
||||
by providing a ConfigMap for kube-dns (`kube-system:kube-dns`).
|
||||
|
||||
For example, the following ConfigMap sets up a DNS configuration with a single stub domain and two
|
||||
upstream nameservers.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kube-dns
|
||||
namespace: kube-system
|
||||
data:
|
||||
stubDomains: |
|
||||
{“acme.local”: [“1.2.3.4”]}
|
||||
upstreamNameservers: |
|
||||
[“8.8.8.8”, “8.8.4.4”]
|
||||
```
|
||||
|
||||
As specified, DNS requests with the “.acme.local” suffix
|
||||
are forwarded to a DNS listening at 1.2.3.4. Google Public DNS
|
||||
serves the upstream queries.
|
||||
|
||||
The table below describes how queries with certain domain names would map to their destination DNS servers:
|
||||
|
||||
| Domain name | Server answering the query |
|
||||
| ----------- | -------------------------- |
|
||||
| kubernetes.default.svc.cluster.local| kube-dns |
|
||||
| foo.acme.local| custom DNS (1.2.3.4) |
|
||||
| widget.com | upstream DNS (one of 8.8.8.8, 8.8.4.4) |
|
||||
|
||||
See [ConfigMap options](#configmap-options) for
|
||||
details about the configuration option format.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture discussion %}
|
||||
|
||||
## Understanding name resolution in Kubernetes
|
||||
|
||||
DNS policies can be set on a per-pod basis. Currently Kubernetes supports two pod-specific DNS policies: “Default” and “ClusterFirst”. These policies are specified with the `dnsPolicy` flag.
|
||||
|
||||
*NOTE: "Default" is not the default DNS policy. If `dnsPolicy` is not
|
||||
explicitly specified, then “ClusterFirst” is used.*
|
||||
|
||||
### "Default" DNS Policy
|
||||
|
||||
If `dnsPolicy` is set to “Default”, then the name resolution configuration is
|
||||
inherited from the node that the pods run on. Custom upstream nameservers and stub domains cannot be used in conjunction with this policy.
|
||||
|
||||
### "ClusterFirst" DNS Policy
|
||||
|
||||
If the `dnsPolicy` is set to "ClusterFirst", name resolution is handled differently, *depending on whether stub-domain and upstream DNS servers are configured*.
|
||||
|
||||
**Without custom configurations**: Any query that does not match the configured cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream nameserver inherited from the node.
|
||||
|
||||
**With custom configurations**: If stub domains and upstream DNS servers are configured (as in the [previous example](#configuring-stub-domain-and-upstream-dns-servers)), DNS queries will be
|
||||
routed according to the following flow:
|
||||
|
||||
1. The query is first sent to the DNS caching layer in kube-dns.
|
||||
|
||||
1. From the caching layer, the suffix of the request is examined and then forwarded to the appropriate DNS, based on the following cases:
|
||||
|
||||
* *Names with the cluster suffix* (e.g.".cluster.local"): The request is sent to kube-dns.
|
||||
|
||||
* *Names with the stub domain suffix* (e.g. ".acme.local"): The request is sent to the configured custom DNS resolver (e.g. listening at 1.2.3.4).
|
||||
|
||||
* *Names without a matching suffix* (e.g."widget.com"): The request is forwarded to the upstream DNS (e.g. Google public DNS servers at 8.8.8.8 and 8.8.4.4).
|
||||
|
||||
|
||||
|
||||
## ConfigMap options
|
||||
|
||||
Options for the kube-dns `kube-system:kube-dns` ConfigMap
|
||||
|
||||
| Field | Format | Description |
|
||||
| ----- | ------ | ----------- |
|
||||
| `stubDomains` (optional) | A JSON map using a DNS suffix key (e.g. “acme.local”) and a value consisting of a JSON array of DNS IPs. | The target nameserver may itself be a Kubernetes service. For instance, you can run your own copy of dnsmasq to export custom DNS names into the ClusterDNS namespace. |
|
||||
| `upstreamNameservers` (optional) | A JSON array of DNS IPs. | Note: If specified, then the values specified replace the nameservers taken by default from the node’s `/etc/resolv.conf`. Limits: a maximum of three upstream nameservers can be specified. |
|
||||
|
||||
## Additional examples
|
||||
|
||||
### Example: Stub domain
|
||||
|
||||
In this example, the user has a Consul DNS service discovery system that they wish to
|
||||
integrate with kube-dns. The consul domain server is located at 10.150.0.1, and
|
||||
all consul names have the suffix “.consul.local”. To configure Kubernetes, the
|
||||
cluster administrator simply creates a ConfigMap object as shown below.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kube-dns
|
||||
namespace: kube-system
|
||||
data:
|
||||
stubDomains: |
|
||||
{“consul.local”: [“10.150.0.1”]}
|
||||
```
|
||||
|
||||
Note that the cluster administrator did not wish to override the node’s
|
||||
upstream nameservers, so they did not specify the optional
|
||||
`upstreamNameservers` field.
|
||||
|
||||
### Example: Upstream nameserver
|
||||
|
||||
In this example the cluster administrator wants to explicitly force all
|
||||
non-cluster DNS lookups to go through their own nameserver at 172.16.0.1.
|
||||
Again, this is easy to accomplish; they just need to create a ConfigMap with the
|
||||
`upstreamNameservers` field specifying the desired nameserver.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kube-dns
|
||||
namespace: kube-system
|
||||
data:
|
||||
upstreamNameservers: |
|
||||
[“172.16.0.1”]
|
||||
```
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
@@ -0,0 +1,197 @@
|
||||
---
|
||||
assignees:
|
||||
- smarterclayton
|
||||
title: Encrypting data at rest
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
This page shows how to enable and configure encryption of secret data at rest.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
* {% include task-tutorial-prereqs.md %}
|
||||
|
||||
* Kubernetes version 1.7.0 or later is required
|
||||
|
||||
* Encryption at rest is alpha in 1.7.0 which means it may change without notice. Users may be required to decrypt their data prior to upgrading to 1.8.0.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Configuration and determining whether encryption at rest is already enabled
|
||||
|
||||
The `kube-apiserver` process accepts an argument `--experimental-encryption-provider-config`
|
||||
that controls how API data is encrypted in etcd. An example configuration
|
||||
is provided below.
|
||||
|
||||
## Understanding the encryption at rest configuration.
|
||||
|
||||
```yaml
|
||||
kind: EncryptionConfig
|
||||
apiVersion: v1
|
||||
resources:
|
||||
- resources:
|
||||
- secrets
|
||||
providers:
|
||||
- identity: {}
|
||||
- aesgcm:
|
||||
keys:
|
||||
- name: key1
|
||||
secret: c2VjcmV0IGlzIHNlY3VyZQ==
|
||||
- name: key2
|
||||
secret: dGhpcyBpcyBwYXNzd29yZA==
|
||||
- aescbc:
|
||||
keys:
|
||||
- name: key1
|
||||
secret: c2VjcmV0IGlzIHNlY3VyZQ==
|
||||
- name: key2
|
||||
secret: dGhpcyBpcyBwYXNzd29yZA==
|
||||
- secretbox:
|
||||
keys:
|
||||
- name: key1
|
||||
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
|
||||
```
|
||||
|
||||
Each `resources` array item is a separate config and contains a complete configuration. The
|
||||
`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
|
||||
that should be encrypted. The `providers` array is an ordered list of the possible encryption
|
||||
providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
|
||||
but not both in the same item).
|
||||
|
||||
The first provider in the list is used to encrypt resources going into storage. When reading
|
||||
resources from storage each provider that matches the stored data attempts to decrypt the data in
|
||||
order. If no provider can read the stored data due to a mismatch in format or secret key, an error
|
||||
is returned which prevents clients from accessing that resource.
|
||||
|
||||
**IMPORTANT:** If any resource is not readable via the encryption config (because keys were changed),
|
||||
the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
|
||||
read that resource will fail until it is deleted or a valid decryption key is provided.
|
||||
|
||||
### Providers:
|
||||
|
||||
Name | Encryption | Strength | Speed | Key Length | Other Considerations
|
||||
-----|------------|----------|-------|------------|---------------------
|
||||
`identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
|
||||
`aescbc` | AES-CBC with PKCS#7 padding | Strongest | Fast | 32-byte | The recommended choice for encryption at rest but may be slightly slower than `secretbox`.
|
||||
`secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review.
|
||||
`aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented.
|
||||
|
||||
Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
|
||||
is the first provider, the first key is used for encryption.
|
||||
|
||||
## Encrypting your data
|
||||
|
||||
Create a new encryption config file
|
||||
|
||||
```yaml
|
||||
kind: EncryptionConfig
|
||||
apiVersion: v1
|
||||
resources:
|
||||
- resources:
|
||||
- secrets
|
||||
providers:
|
||||
- aescbc:
|
||||
keys:
|
||||
- name: key1
|
||||
secret: <BASE 64 ENCODED SECRET>
|
||||
- identity: {}
|
||||
```
|
||||
|
||||
To create a new secret perform the following steps:
|
||||
|
||||
1. Generate a 32 byte random key and base64 encode it. If you're on Linux or Mac OS X, run the following command:
|
||||
|
||||
```
|
||||
head -c 32 /dev/urandom | base64 -i - -o -
|
||||
```
|
||||
|
||||
2. Place that value in the secret field.
|
||||
3. Set the `--experimental-encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file
|
||||
4. restart your API server.
|
||||
|
||||
**IMPORTANT:** Your config file contains keys that can decrypt content in etcd, so you must properly restrict permissions on your masters so only the user who runs the kube-apiserver can read it.
|
||||
|
||||
|
||||
## Verifying that data is encrypted
|
||||
|
||||
Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or
|
||||
updated secret should be encrypted when stored. To check, you can use the `etcdctl` command line
|
||||
program to retrieve the contents of your secret.
|
||||
|
||||
1. Create a new secret called `secret1` in the `default` namespace:
|
||||
|
||||
```
|
||||
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
|
||||
```
|
||||
|
||||
2. Using the etcdctl commandline, read that secret out of etcd:
|
||||
|
||||
```
|
||||
ETCDCTL_API=3 etcdctl get /kubernetes.io/secrets/default/secret1 [...] | hexdump -C
|
||||
```
|
||||
|
||||
where `[...]` must be the additional arguments for connecting to the etcd server.
|
||||
3. Verify the stored secret is prefixed with `k8s:enc:aescbc:v1:` which indicates the `aescbc` provider has encrypted the resulting data.
|
||||
4. Verify the secret is correctly decrypted when retrieved via the API:
|
||||
|
||||
```
|
||||
kubectl describe secret generic -n default
|
||||
```
|
||||
|
||||
should match `mykey: mydata`
|
||||
|
||||
|
||||
## Ensure all secrets are encrypted
|
||||
|
||||
Since secrets are encrypted on write, performing an update on a secret will encrypt that content.
|
||||
|
||||
```
|
||||
kubectl get secrets -o json | kubectl update -f -
|
||||
```
|
||||
|
||||
The command above reads all secrets and then updates them to apply server side encryption.
|
||||
If an error occurs due to a conflicting write, retry the command.
|
||||
For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
|
||||
|
||||
|
||||
## Rotating a decryption key
|
||||
|
||||
Changing the secret without incurring downtime requires a multi step operation, especially in
|
||||
the presence of a highly available deployment where multiple `kube-apiserver` processes are running.
|
||||
|
||||
1. Generate a new key and add it as the second key entry for the current provider on all servers
|
||||
2. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key
|
||||
3. Make the new key the first entry in the `keys` array so that it is used for encryption in the config
|
||||
4. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key
|
||||
5. Run `kubectl get secrets -o json | kubectl update -f -` to update all secrets
|
||||
6. Remove the old decryption key from the config after you back up etcd with the new key in use and update all secrets
|
||||
|
||||
With a single `kube-apiserver`, step 2 may be skipped
|
||||
|
||||
|
||||
## Decrypting all data
|
||||
|
||||
To disable encryption at rest place the `identity` provider as the first entry in the config:
|
||||
|
||||
```yaml
|
||||
kind: EncryptionConfig
|
||||
apiVersion: v1
|
||||
resources:
|
||||
- resources:
|
||||
- secrets
|
||||
providers:
|
||||
- identity: {}
|
||||
- aescbc:
|
||||
keys:
|
||||
- name: key1
|
||||
secret: <BASE 64 ENCODED SECRET>
|
||||
```
|
||||
|
||||
and restart all `kube-apiserver` processes. Then run the command `kubectl get secrets -o json | kubectl update -f -`
|
||||
to force all secrets to be decrypted.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
assignees:
|
||||
- pipejakob
|
||||
title: Upgrading kubeadm clusters from 1.6 to 1.7
|
||||
redirect_from:
|
||||
- "/docs/admin/kubeadm-upgrade-1-7/"
|
||||
- "/docs/admin/kubeadm-upgrade-1-7.html"
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
This guide is for upgrading kubeadm clusters from version 1.6.x to 1.7.x.
|
||||
Upgrades are not supported for clusters lower than 1.6, which is when kubeadm
|
||||
became Beta.
|
||||
|
||||
**WARNING**: These instructions will **overwrite** all of the resources managed
|
||||
by kubeadm (static pod manifest files, service accounts and RBAC rules in the
|
||||
`kube-system` namespace, etc.), so any customizations you may have made to these
|
||||
resources after cluster setup will need to be reapplied after the upgrade. The
|
||||
upgrade will not disturb other static pod manifest files or objects outside the
|
||||
`kube-system` namespace.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
You need to have a Kubernetes cluster running version 1.6.x.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## On the master
|
||||
|
||||
1. Upgrade system packages.
|
||||
|
||||
Upgrade your OS packages for kubectl, kubeadm, kubelet, and kubernetes-cni.
|
||||
|
||||
a. On Debian, this can be accomplished with:
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get upgrade
|
||||
|
||||
b. On CentOS/Fedora, you would instead run:
|
||||
|
||||
sudo yum update
|
||||
|
||||
2. Restart kubelet.
|
||||
|
||||
sudo systemctl restart kubelet
|
||||
|
||||
3. Delete the `kube-proxy` DaemonSet.
|
||||
|
||||
Although most components are automatically upgraded by the next step,
|
||||
`kube-proxy` currently needs to be manually deleted so it can be recreated at
|
||||
the correct version:
|
||||
|
||||
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete daemonset kube-proxy -n kube-system
|
||||
|
||||
4. Perform kubeadm upgrade.
|
||||
|
||||
sudo kubeadm init --skip-preflight-checks --kubernetes-version <DESIRED_VERSION>
|
||||
|
||||
For instance, if you want to upgrade to `1.7.0`, you would run:
|
||||
|
||||
sudo kubeadm init --skip-preflight-checks --kubernetes-version v1.7.0
|
||||
|
||||
5. Upgrade CNI provider.
|
||||
|
||||
Your CNI provider might have its own upgrade instructions to follow now.
|
||||
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
|
||||
find your CNI provider and see if there are additional upgrade steps
|
||||
necessary.
|
||||
|
||||
## On each node
|
||||
|
||||
1. Upgrade system packages.
|
||||
|
||||
Upgrade your OS packages for kubectl, kubeadm, kubelet, and kubernetes-cni.
|
||||
|
||||
a. On Debian, this can be accomplished with:
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get upgrade
|
||||
|
||||
b. On CentOS/Fedora, you would instead run:
|
||||
|
||||
sudo yum update
|
||||
|
||||
2. Restart kubelet.
|
||||
|
||||
sudo systemctl restart kubelet
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -45,8 +45,7 @@ on each node.
|
||||
|
||||
`Allocatable` on a Kubernetes node is defined as the amount of compute resources
|
||||
that are available for pods. The scheduler does not over-subscribe
|
||||
`Allocatable`. `CPU` and `memory` are supported as of now. Support for `storage`
|
||||
is expected to be added in the future.
|
||||
`Allocatable`. `CPU`, `memory` and `storage` are supported as of now.
|
||||
|
||||
Node Allocatable is exposed as part of `v1.Node` object in the API and as part
|
||||
of `kubectl describe node` in the CLI.
|
||||
@@ -81,7 +80,7 @@ be configured to use the `systemd` cgroup driver.
|
||||
|
||||
### Kube Reserved
|
||||
|
||||
- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi]`
|
||||
- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][storage=1Gi]`
|
||||
- **Kubelet Flag**: `--kube-reserved-cgroup=`
|
||||
|
||||
`kube-reserved` is meant to capture resource reservation for kubernetes system
|
||||
@@ -111,7 +110,7 @@ exist. Kubelet will fail if an invalid cgroup is specified.
|
||||
|
||||
### System Reserved
|
||||
|
||||
- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi]`
|
||||
- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][storage=1Gi]`
|
||||
- **Kubelet Flag**: `--system-reserved-cgroup=`
|
||||
|
||||
|
||||
@@ -194,20 +193,22 @@ So expect a drop in `Allocatable` capacity in future releases.
|
||||
|
||||
Here is an example to illustrate Node Allocatable computation:
|
||||
|
||||
* Node has `32Gi` of `memory` and `16 CPUs`
|
||||
* `--kube-reserved` is set to `cpu=1,memory=2Gi`
|
||||
* `--system-reserved` is set to `cpu=500m,memory=1Gi`
|
||||
* `--eviction-hard` is set to `memory.available<500Mi`
|
||||
* Node has `32Gi` of `memory`, `16 CPUs` and `100Gi` of `Storage`
|
||||
* `--kube-reserved` is set to `cpu=1,memory=2Gi,storage=1Gi`
|
||||
* `--system-reserved` is set to `cpu=500m,memory=1Gi,storage=1Gi`
|
||||
* `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`
|
||||
|
||||
Under this scenario, `Allocatable` will be `14.5 CPUs` & `28.5Gi` of memory.
|
||||
Scheduler ensures that the total `requests` across all pods on this node does
|
||||
not exceed `28.5Gi`. Kubelet evicts pods whenever the overall memory usage
|
||||
exceeds across pods exceed `28.5Gi`. If all processes on the node consume as
|
||||
Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
|
||||
`98Gi` of local storage.
|
||||
Scheduler ensures that the total memory `requests` across all pods on this node does
|
||||
not exceed `28.5Gi` and storage doesn't exceed `88Gi`.
|
||||
Kubelet evicts pods whenever the overall memory usage exceeds across pods exceed `28.5Gi`,
|
||||
or if overall disk usage exceeds `88Gi` If all processes on the node consume as
|
||||
much CPU as they can, pods together cannot consume more than `14.5 CPUs`.
|
||||
|
||||
If `kube-reserved` and/or `system-reserved` is not enforced and system daemons
|
||||
exceed their reservation, `kubelet` evicts pods whenever the overall node memory
|
||||
usage is higher than `31.5Gi`.
|
||||
usage is higher than `31.5Gi` or `storage` is greater than `90Gi`
|
||||
|
||||
## Feature Availability
|
||||
|
||||
@@ -231,3 +232,5 @@ required to drain their nodes prior to upgrade of the `kubelet` from prior
|
||||
versions in order to ensure pods and their associated containers are launched in
|
||||
the proper part of the cgroup hierarchy.
|
||||
|
||||
As of Kubernetes version 1.7, `kubelet` supports specifying `storage` as a resource
|
||||
for `kube-reserved` and `system-reserved`.
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
assignees:
|
||||
- davidopp
|
||||
- mml
|
||||
- foxish
|
||||
- kow3ns
|
||||
title: Safely Drain a Node while Respecting Application SLOs
|
||||
---
|
||||
|
||||
@@ -14,8 +17,12 @@ disruption SLOs you have specified using PodDisruptionBudget.
|
||||
This task assumes that you have met the following prerequisites:
|
||||
|
||||
* You are using Kubernetes release >= 1.5.
|
||||
* You have created [PodDisruptionBudget(s)](/docs/tasks/configure-pod-container/configure-pod-disruption-budget/) to express the
|
||||
application-level disruption SLOs you want the system to enforce.
|
||||
* Either:
|
||||
1. You do not require your applications to be highly available during the
|
||||
node drain, or
|
||||
1. You have read about the [PodDisruptionBudget concept](/docs/concepts/workloads/pods/disruptions.md)
|
||||
and [Configured PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb.md) for
|
||||
applications that need them.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
@@ -81,11 +88,75 @@ that only one pod is unavailable at any given time. Any drains that
|
||||
would cause the number of ready replicas to fall below the specified
|
||||
budget are blocked.
|
||||
|
||||
## The Eviction API
|
||||
|
||||
If you prefer not to use [kubectl drain](/docs/user-guide/kubectl/v1.6/#drain) (such as
|
||||
to avoid calling to an external command, or to get finer control over over the pod
|
||||
eviction process), you can also programmatically cause evictions using the eviction API.
|
||||
|
||||
You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api.md#programmatic-access-to-the-api).
|
||||
|
||||
The eviction subresource of a
|
||||
pod can be thought of as a kind of policy-controlled DELETE operation on the pod
|
||||
itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
|
||||
*create* an eviction), you POST an attempted operation. Here's an example:
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1beta1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
"namespace": "default"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
You can attempt an eviction using `curl`:
|
||||
|
||||
```bash
|
||||
$ curl -v -H 'Content-type: application/json' http://127.0.0.1:8080/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
|
||||
```
|
||||
|
||||
The API can respond in one of three ways:
|
||||
|
||||
- If the eviction is granted, then the pod is deleted just as if you had sent
|
||||
a `DELETE` request to the pod's URL and you get back `200 OK`.
|
||||
- If the current state of affairs wouldn't allow an eviction by the rules set
|
||||
forth in the budget, you get back `429 Too Many Requests`. This is
|
||||
typically used for generic rate limiting of *any* requests, but here we mean
|
||||
that this request isn't allowed *right now* but it may be allowed later.
|
||||
Currently, callers do not get any `Retry-After` advice, but they may in
|
||||
future versions.
|
||||
- If there is some kind of misconfiguration, like multiple budgets pointing at
|
||||
the same pod, you will get `500 Internal Server Error`.
|
||||
|
||||
For a given eviction request, there are two cases.
|
||||
|
||||
- There is no budget that matches this pod. In this case, the server always
|
||||
returns `200 OK`.
|
||||
- There is at least one budget. In this case, any of the three above responses may
|
||||
apply.
|
||||
|
||||
In some cases, an application may reach a broken state where it will never return anything
|
||||
other than 429 or 500. This can happen, for example, if the replacement pod created by the
|
||||
application's controller does not become ready, or if the last pod evicted has a very long
|
||||
termination grace period.
|
||||
|
||||
In this case, there are two potential solutions:
|
||||
|
||||
- Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation.
|
||||
- After a suitably long wait, `DELETE` the pod instead of using the eviction API.
|
||||
|
||||
Kubernetes does not specify what the behavior should be in this case; it is up to the
|
||||
application owners and cluster owners to establish an agreement on behavior in these cases.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture whatsnext %}
|
||||
*TODO: link to other docs about Stateful Set?*
|
||||
|
||||
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application//configure-pdb.md).
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
assignees:
|
||||
- smarterclayton
|
||||
title: Securing a Cluster
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
This document covers topics related to protecting a cluster from accidental or malicious access
|
||||
and provides recommendations on overall security.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
* {% include task-tutorial-prereqs.md %}
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Controlling access to the Kubernetes API
|
||||
|
||||
As Kubernetes is entirely API driven, controlling and limiting who can access the cluster and what actions
|
||||
they are allowed to perform is the first line of defense.
|
||||
|
||||
### Use Transport Level Security (TLS) for all API traffic
|
||||
|
||||
Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the
|
||||
majority of installation methods will allow the necessary certificates to be created and distributed to
|
||||
the cluster components. Note that some components and installation methods may enable local ports over
|
||||
HTTP and administrators should familiarize themselves with the settings of each component to identify
|
||||
potentially unsecured traffic.
|
||||
|
||||
### API Authentication
|
||||
|
||||
Choose an authentication mechanism for the API servers to use that matches the common access patterns
|
||||
when you install a cluster. For instance, small single user clusters may wish to use a simple certificate
|
||||
or static Bearer token approach. Larger clusters may wish to integrate an existing or OIDC or LDAP server that
|
||||
allow users to be subdivided into groups.
|
||||
|
||||
All API clients must be authenticated, even those that are part of the infrastructure like nodes,
|
||||
proxies, the scheduler, and volume plugins. These clients are typically [service accounts](/docs/admin/service-accounts-admin/) or use x509 client certificates, and they are created automatically at cluster startup or are setup as part of the cluster installation.
|
||||
|
||||
Consult the [authentication reference document](/docs/admin/authentication/) for more information.
|
||||
|
||||
### API Authorization
|
||||
|
||||
Once authenticated, every API call is also expected to pass an authorization check. Kubernetes ships
|
||||
an integrated [Role-Based Access Control (RBAC)](/docs/admin/authorization/rbac/) component that matches an incoming user or group to a
|
||||
set of permissions bundled into roles. These permissions combine verbs (get, create, delete) with
|
||||
resources (pods, services, nodes) and can be namespace or cluster scoped. A set of out of the box
|
||||
roles are provided that offer reasonable default separation of responsibility depending on what
|
||||
actions a client might want to perform. It is recommended that you use the [Node](/docs/admin/authorization/node/) and [RBAC](/docs/admin/authorization/rbac/) authorizers together, in combination with the
|
||||
[NodeRestriction](/docs/admin/admission-controllers/#noderestriction) admission plugin.
|
||||
|
||||
As with authentication, simple and broad roles may be appropriate for smaller clusters, but as
|
||||
more users interact with the cluster, it may become necessary to separate teams into separate
|
||||
namespaces with more limited roles.
|
||||
|
||||
With authorization, it is important to understand how updates on one object may cause actions in
|
||||
other places. For instance, a user may not be able to create pods directly, but allowing them to
|
||||
create a deployment, which creates pods on their behalf, will let them create those pods
|
||||
indirectly. Likewise, deleting a node from the API will result in the pods scheduled to that node
|
||||
being terminated and recreated on other nodes. The out of the box roles represent a balance
|
||||
between flexibility and the common use cases, but more limited roles should be carefully reviewed
|
||||
to prevent accidental escalation. You can make roles specific to your use case if the out-of-box ones don't meet your needs.
|
||||
|
||||
Consult the [authorization reference section](/docs/admin/authorization) for more information.
|
||||
|
||||
|
||||
## Controlling the capabilities of a workload or user at runtime
|
||||
|
||||
Authorization in Kubernetes is intentionally high level, focused on coarse actions on resources.
|
||||
More powerful controls exist as **policies** to limit by use case how those objects act on the
|
||||
cluster, themselves, and other resources.
|
||||
|
||||
### Limiting resource usage on a cluster
|
||||
|
||||
[Resource quota](/docs/concepts/policy/resource-quotas/) limits the number or capacity of
|
||||
resources granted to a namespace. This is most often used to limit the amount of CPU, memory,
|
||||
or persistent disk a namespace can allocate, but can also control how many pods, services, or
|
||||
volumes exist in each namespace.
|
||||
|
||||
[Limit ranges](/docs/admin/limitrange) restrict the maximum or minimum size of some of the
|
||||
resources above, to prevent users from requesting unreasonably high or low values for commonly
|
||||
reserved resources like memory, or to provide default limits when none are specified.
|
||||
|
||||
|
||||
### Controlling what privileges containers run with
|
||||
|
||||
A pod definition contains a [security context](/docs/tasks/configure-pod-container/security-context/)
|
||||
that allows it to request access to running as a specific Linux user on a node (like root),
|
||||
access to run privileged or access the host network, and other controls that would otherwise
|
||||
allow it to run unfettered on a hosting node. [Pod security policies](/docs/concepts/policy/pod-security-policy/)
|
||||
can limit which users or service accounts can provide dangerous security context settings. For example, pod security policies can limit volume mounts, especially `hostPath`, which are aspects of a pod that should be controlled.
|
||||
|
||||
Generally, most application workloads need limited access to host resources so they can
|
||||
successfully run as a root process (uid 0) without access to host information. However,
|
||||
considering the privileges associated with the root user, you should write application
|
||||
containers to run as a non-root user. Similarly, administrators who wish to prevent
|
||||
client applications from escaping their containers should use a restrictive pod security
|
||||
policy.
|
||||
|
||||
|
||||
### Restricting network access
|
||||
|
||||
The [network policies](/docs/tasks/administer-cluster/declare-network-policy/) for a namespace
|
||||
allows application authors to restrict which pods in other namespaces may access pods and ports
|
||||
within their namespace. Many of the supported [Kubernetes networking providers](/docs/concepts/cluster-administration/networking/)
|
||||
now respect network policy.
|
||||
|
||||
Quota and limit ranges can also be used to control whether users may request node ports or
|
||||
load balanced services, which on many clusters can control whether those users applications
|
||||
are visible outside of the cluster.
|
||||
|
||||
Additional protections may be available that control network rules on a per plugin or per
|
||||
environment basis, such as per-node firewalls, physically separating cluster nodes to
|
||||
prevent cross talk, or advanced networking policy.
|
||||
|
||||
|
||||
### Controlling which nodes pods may access
|
||||
|
||||
By default, there are no restrictions on which nodes may run a pod. Kubernetes offers a
|
||||
[rich set of policies for controlling placement of pods onto nodes](/docs/concepts/configuration/assign-pod-node/)
|
||||
that are available to end users. For many clusters use of these policies to separate workloads
|
||||
can be a convention that authors adopt or enforce via tooling.
|
||||
|
||||
As an administrator, a beta admission plugin `PodNodeSelector` can be used to force pods
|
||||
within a namespace to default or require a specific node selector, and if end users cannot
|
||||
alter namespaces, this can strongly limit the placement of all of the pods in a specific workload.
|
||||
|
||||
|
||||
## Protecting cluster components from compromise
|
||||
|
||||
This section describes some common patterns for protecting clusters from compromise.
|
||||
|
||||
### Restrict access to etcd
|
||||
|
||||
Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster,
|
||||
and read access can be used to escalate fairly quickly. Administrators should always use strong
|
||||
credentials from the API servers to their etcd server, such as mutual auth via TLS client certificates,
|
||||
and it is often recommended to isolate the etcd servers behind a firewall that only the API servers
|
||||
may access.
|
||||
|
||||
**CAUTION:** Allowing other components within the cluster to access the master etcd instance with
|
||||
read or write access to the full keyspace is equivalent to granting cluster-admin access. Using
|
||||
separate etcd instances for non-master components or using etcd ACLs to restrict read and write
|
||||
access to a subset of the keyspace is strongly recommended.
|
||||
|
||||
### Enable audit logging
|
||||
|
||||
The [audit logger](/docs/admin/audit/) is an alpha feature that records actions taken by the
|
||||
API for later analysis in the event of a compromise. It is recommended to enable audit logging
|
||||
and archive the audit file on a secure server.
|
||||
|
||||
### Restrict access to alpha or beta features
|
||||
|
||||
Alpha and beta Kubernetes features are in active development and may have limitations or bugs
|
||||
that result in security vulnerabilities. Always assess the value an alpha or beta feature may
|
||||
provide against the possible risk to your security posture. When in doubt, disable features you
|
||||
do not use.
|
||||
|
||||
### Rotate infrastructure credentials frequently
|
||||
|
||||
The shorter the lifetime of a secret or credential the harder it is for an attacker to make
|
||||
use of that credential. Set short lifetimes on certificates and automate their rotation. Use
|
||||
an authentication provider that can control how long issued tokens are available and use short
|
||||
lifetimes where possible. If you use service account tokens in external integrations, plan to
|
||||
rotate those tokens frequently. For example, once the bootstrap phase is complete, a bootstrap token used for setting up nodes should be revoked or its authorization removed.
|
||||
|
||||
### Review third party integrations before enabling them
|
||||
|
||||
Many third party integrations to Kubernetes may alter the security profile of your cluster. When
|
||||
enabling an integration, always review the permissions that an extension requests before granting
|
||||
it access. For example, many security integrations may request access to view all secrets on
|
||||
your cluster which is effectively making that component a cluster admin. When in doubt,
|
||||
restrict the integration to functioning in a single namespace if possible.
|
||||
|
||||
Components that create pods may also be unexpectedly powerful if they can do so inside namespaces
|
||||
like the `kube-system` namespace, because those pods can gain access to service account secrets
|
||||
or run with elevated permissions if those service accounts are granted access to permissive
|
||||
[pod security policies](/docs/concepts/policy/pod-security-policy/).
|
||||
|
||||
### Encrypt secrets at rest
|
||||
|
||||
In general, the etcd database will contain any information accessible via the Kubernetes API
|
||||
and may grant an attacker significant visibility into the state of your cluster. Always encrypt
|
||||
your backups using a well reviewed backup and encryption solution, and consider using full disk
|
||||
encryption where possible.
|
||||
|
||||
Kubernetes 1.7 contains [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/), an alpha feature that will encrypt `Secret` resources in etcd, preventing
|
||||
parties that gain access to your etcd backups from viewing the content of those secrets. While
|
||||
this feature is currently experimental, it may offer an additional level of defense when backups
|
||||
are not encrypted or an attacker gains read access to etcd.
|
||||
|
||||
### Receiving alerts for security updates and reporting vulnerabilities
|
||||
|
||||
Join the [kubernetes-announce](https://groups.google.com/forum/#!forum/kubernetes-announce)
|
||||
group for emails about security announcements. See the [security reporting](/security/)
|
||||
page for more on how to report vulnerabilities.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -69,6 +69,40 @@ kubefed unjoin gondor --host-cluster-context=rivendell
|
||||
You can find more details on unjoin in the
|
||||
[kubefed guide](/docs/tutorials/federation/set-up-cluster-federation-kubefed/#removing-a-cluster-from-a-federation).
|
||||
|
||||
## Labeling Clusters
|
||||
|
||||
You can label clusters the same way as any other Kubernetes object, which can help with grouping clusters and can also be leveraged by the ClusterSelector.
|
||||
|
||||
``` shell
|
||||
kubectl --context=rivendell label cluster gondor key1=value1 key2=value2
|
||||
```
|
||||
|
||||
## ClusterSelector Annotation
|
||||
|
||||
Starting in Kubernetes 1.7, there is alpha support for directing objects across the federated clusters with the annotation `federation.alpha.kubernetes.io/cluster-selector`. The *ClusterSelector* is conceptually similar to `nodeSelector`, but instead of selecting against labels on nodes, it selects against labels on federated clusters.
|
||||
|
||||
The annotation value must be json formatted and must be parsable into the [ClusterSelector API type](/docs/reference/federation/v1beta1/definitions/#_v1beta1_clusterselector). For example: `[{"key": "load", "operator": "Lt", "values": ["10"]}]`. Content that doesn't parse correctly will throw an error and prevent distribution of the object to any federated clusters. Objects of type Configmap, Secret, Daemonset, Service and Ingress are included in the alpha implementation.
|
||||
|
||||
Here is an example ClusterSelector annotation, which will only select clusters WITH the label `pci=true` and WITHOUT the label `environment=test`:
|
||||
|
||||
``` yaml
|
||||
metadata:
|
||||
annotations:
|
||||
federation.alpha.kubernetes.io/cluster-selector: '[{"key": "pci", "operator":
|
||||
"In", "values": ["true"]}, {"key": "environment", "operator": "NotIn", "values":
|
||||
["test"]}]'
|
||||
```
|
||||
|
||||
The *key* is matched against label names on the federated clusters.
|
||||
|
||||
The *values* are matched against the label values on the federated clusters.
|
||||
|
||||
The possible *operators* are: `In`, `NotIn`, `Exists`, `DoesNotExist`, `Gt`, `Lt`.
|
||||
|
||||
The *values* field is expected to be empty when `Exists` or `DoesNotExist` is specified and may include more than one string when `In` or `NotIn` are used.
|
||||
|
||||
Currently, only integers are supported with `Gt` or `Lt`.
|
||||
|
||||
## Clusters API reference
|
||||
|
||||
The full clusters API reference is currently in `federation/v1beta1` and more details can be found in details in the
|
||||
|
||||
@@ -273,6 +273,10 @@ very
|
||||
You can project keys to specific paths and specific permissions on a per-file
|
||||
basis. The [Secrets](/docs/concepts/configuration/secret#using-secrets-as-files-from-a-pod) user guide explains the syntax.
|
||||
|
||||
### Mounted ConfigMaps are updated automatically
|
||||
|
||||
When a ConfigMap already being consumed in a volume is updated, projected keys are eventually updated as well. Kubelet is checking whether the mounted ConfigMap is fresh on every periodic sync. However, it is using its local ttl-based cache for getting the current value of the ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of ConfigMaps cache in kubelet.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture discussion %}
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
assignees:
|
||||
- soltysh
|
||||
- sttts
|
||||
- ericchiang
|
||||
title: Auditing
|
||||
redirect_from:
|
||||
- "/docs/admin/audit/"
|
||||
@@ -17,6 +18,7 @@ Kubernetes Audit provides a security-relevant chronological set of records docum
|
||||
the sequence of activities that have affected system by individual users, administrators
|
||||
or other components of the system. It allows cluster administrator to
|
||||
answer the following questions:
|
||||
|
||||
- what happened?
|
||||
- when did it happen?
|
||||
- who initiated it?
|
||||
@@ -25,36 +27,27 @@ answer the following questions:
|
||||
- from where was it initiated?
|
||||
- to where was it going?
|
||||
|
||||
NOTE: Currently, Kubernetes provides only basic audit capabilities, there is still a lot
|
||||
of work going on to provide fully featured auditing capabilities (see [this issue](https://github.com/kubernetes/features/issues/22)).
|
||||
## Audit logs
|
||||
|
||||
Kubernetes audit is part of [kube-apiserver](/docs/admin/kube-apiserver) logging all requests
|
||||
coming to the server. Each audit log contains two entries:
|
||||
Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests
|
||||
processed by the server. Each audit log entry contains two lines:
|
||||
|
||||
1. The request line containing:
|
||||
- unique id allowing to match the response line (see 2)
|
||||
- source ip of the request
|
||||
- HTTP method being invoked
|
||||
- original user invoking the operation
|
||||
- original user's groups info
|
||||
- impersonated user for the operation
|
||||
- impersonated groups info
|
||||
- namespace of the request or <none>
|
||||
- URI as requested
|
||||
2. The response line containing:
|
||||
- the unique id from 1
|
||||
- response code
|
||||
1. The request line containing a unique ID to match the response and request metadata, such as the source IP, requesting user, impersonation information, resource being requested, etc.
|
||||
2. The response line containing a unique ID matching the request line and the response code.
|
||||
|
||||
Example output for user `admin` asking for a list of pods:
|
||||
Example output for `admin` user listing pods in the `default` namespace:
|
||||
|
||||
```
|
||||
2017-03-21T03:57:09.106841886-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" ip="127.0.0.1" method="GET" user="admin" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
|
||||
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
Note that this format changes when enabling the `AdvancedAuditing` feature discussed
|
||||
later in this document.
|
||||
|
||||
[Kube-apiserver](/docs/admin/kube-apiserver) provides following options which are responsible
|
||||
### Configuration
|
||||
|
||||
[Kube-apiserver][kube-apiserver] provides the following options which are responsible
|
||||
for configuring where and how audit logs are handled:
|
||||
|
||||
- `audit-log-path` - enables the audit log pointing to a file where the requests are being logged to.
|
||||
@@ -70,3 +63,227 @@ the file name (before the file extension) and create a new audit log file.
|
||||
Kubernetes may delete old log files when creating a new log file; you can configure
|
||||
how many files are retained and how old they can be by specifying the `audit-log-maxbackup`
|
||||
and `audit-log-maxage` options.
|
||||
|
||||
## Advanced audit
|
||||
|
||||
Kubernetes 1.7 expands auditing with experimental functionality such as event
|
||||
filtering and a webhook for integration with external systems. The rest of this
|
||||
document covers features that are __alpha__ and may change in backward incompatible
|
||||
ways.
|
||||
|
||||
Enable the alpha auditing features using the `AdvancedAuditing` feature gate on
|
||||
the [kube-apiserver][kube-apiserver]:
|
||||
|
||||
```
|
||||
--feature-gates=AdvancedAuditing=true
|
||||
```
|
||||
|
||||
`AdvancedAuditing`is customizeable in two ways. Policy, which determines what's recorded,
|
||||
and backends, which persist records. Backend implementations include logs files and
|
||||
webhooks.
|
||||
|
||||
The structure of audit events changes when enabling the `AdvancedAuditing` feature
|
||||
flag. This includes some cleanups, such as the `method` reflecting the verb evaluated
|
||||
by the [authoriation layer](/docs/admin/authorization/) instead of the HTTP verb
|
||||
("create" instead of "POST"). Also, instead of always generating two events per
|
||||
request, events are recorded with an associated "stage." The known stages are:
|
||||
|
||||
- `RequestReceived` - The stage for events generated as soon as the audit handler receives the request.
|
||||
- `ResponseStarted` - Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch).
|
||||
- `ResponseComplete` - Once the response body has been completed.
|
||||
- `Panic` - Events generated when a panic occured.
|
||||
|
||||
### Audit Policy
|
||||
|
||||
Audit policy is a document defining rules about what events should be recorded.
|
||||
The policy is passed to the [kube-apiserver][kube-apiserver] using the
|
||||
`--audit-policy-file` flag.
|
||||
|
||||
```
|
||||
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
|
||||
```
|
||||
|
||||
If `AdvancedAuditing` is enabled and this flag is omitted, no events are logged.
|
||||
|
||||
The policy file holds rules that determine the level of an event. Known audit levels are:
|
||||
|
||||
- `None` - don't log events that match this rule.
|
||||
- `Metadata` - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
|
||||
- `Request` - log event metadata and request body but not response body.
|
||||
- `RequestResponse` - log event metadata, request and response bodies.
|
||||
|
||||
When an event is processed it's compared against the list of rules in order.
|
||||
The first matching rule sets the audit level of the event. The audit policy is
|
||||
defined by the [`audit.k8s.io` API group][audit-api].
|
||||
|
||||
An example audit policy file:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
|
||||
- level: None
|
||||
users: ["system:kube-proxy"]
|
||||
verbs: ["watch"]
|
||||
resources:
|
||||
- group: "" # core API group
|
||||
resources: ["endpoints", "services"]
|
||||
|
||||
# Don't log authenticated requests to certain non-resource URL paths.
|
||||
- level: None
|
||||
userGroups: ["system:authenticated"]
|
||||
nonResourceURLs:
|
||||
- "/api*" # Wildcard matching.
|
||||
- "/version"
|
||||
|
||||
# Log the request body of configmap changes in kube-system.
|
||||
- level: Request
|
||||
resources:
|
||||
- group: "" # core API group
|
||||
resources: ["configmaps"]
|
||||
# This rule only applies to resources in the "kube-system" namespace.
|
||||
# The empty string "" can be used to select non-namespaced resources.
|
||||
namespaces: ["kube-system"]
|
||||
|
||||
# Log configmap and secret changes in all other namespaces at the Metadata level.
|
||||
- level: Metadata
|
||||
resources:
|
||||
- group: "" # core API group
|
||||
resources: ["secrets", "configmaps"]
|
||||
|
||||
# Log all other resources in core and extensions at the Request level.
|
||||
- level: Request
|
||||
resources:
|
||||
- group: "" # core API group
|
||||
- group: "extensions" # Version of group should NOT be included.
|
||||
|
||||
# A catch-all rule to log all other requests at the Metadata level.
|
||||
- level: Metadata
|
||||
```
|
||||
|
||||
You can use a minimal audit policy file to log all requests at the `Metadata` level:
|
||||
|
||||
```yaml
|
||||
# Log all requests at the Metadata level.
|
||||
rules:
|
||||
- level: Metadata
|
||||
```
|
||||
|
||||
The [audit profile used by GCE][gce-audit-profile] should be used as reference by
|
||||
admins constructing their own audit profiles.
|
||||
|
||||
### Audit backends
|
||||
|
||||
Audit backends implement strategies for emitting events. The [kube-apiserver][kube-apiserver]
|
||||
provides a logging and webhook backend.
|
||||
|
||||
Each request to the API server can generate multiple events, one when the request is received,
|
||||
another when the response is sent, and additional events for log running requests (such as
|
||||
watches). The ID of events will be the same if they were generated from the same request.
|
||||
|
||||
The event format is defined by the `audit.k8s.io` API group. The `v1alpha1` format of this
|
||||
API can be found [here][audit-api] with more details about the exact fields captured.
|
||||
|
||||
#### Log backend
|
||||
|
||||
The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing`
|
||||
feature flag. This includes the cleanups discussed above, such as changes to the `method`
|
||||
values and the introduction of a "stage" for each event. As before, the `id` field of
|
||||
the log line indicates which events were generated from the same request. Events are
|
||||
formatted as follows:
|
||||
|
||||
```
|
||||
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="<deferred>"
|
||||
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="200"
|
||||
```
|
||||
|
||||
Logged evens omit the request and response bodies. The `Request` and
|
||||
`RequestResponse` levels are equivalent to `Metadata` for this backend.
|
||||
|
||||
#### Webhook backend
|
||||
|
||||
The audit webhook backend can be used to have [kube-apiserver][kube-apiserver]
|
||||
send audit events to a remote service. The webhook requires the `AdvancedAuditing`
|
||||
feature flag and is configured using the following command line flags:
|
||||
|
||||
```
|
||||
--audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
|
||||
--audit-webhook-mode=batch
|
||||
```
|
||||
|
||||
`audit-webhook-mode` controls buffering strategies used by the webhook. Known modes are:
|
||||
|
||||
- `batch` - buffer events and asynchronously send the set of events to the external service.
|
||||
- `blocking` - block API server responses on sending each event to the external service.
|
||||
|
||||
The webhook config file uses the kubeconfig format to specify the remote address of
|
||||
the service and credentials used to connect to it.
|
||||
|
||||
```
|
||||
# clusters refers to the remote service.
|
||||
clusters:
|
||||
- name: name-of-remote-audit-service
|
||||
cluster:
|
||||
certificate-authority: /path/to/ca.pem # CA for verifying the remote service.
|
||||
server: https://audit.example.com/audit # URL of remote service to query. Must use 'https'.
|
||||
|
||||
# users refers to the API server's webhook configuration.
|
||||
users:
|
||||
- name: name-of-api-server
|
||||
user:
|
||||
client-certificate: /path/to/cert.pem # cert for the webhook plugin to use
|
||||
client-key: /path/to/key.pem # key matching the cert
|
||||
|
||||
# kubeconfig files require a context. Provide one for the API server.
|
||||
current-context: webhook
|
||||
contexts:
|
||||
- context:
|
||||
cluster: name-of-remote-audit-service
|
||||
user: name-of-api-sever
|
||||
name: webhook
|
||||
```
|
||||
|
||||
Events are POSTed as a JSON serialized `EventList`. An example payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "EventList",
|
||||
"apiVersion": "audit.k8s.io/v1alpha1",
|
||||
"items": [
|
||||
{
|
||||
"metadata": {
|
||||
"creationTimestamp": null
|
||||
},
|
||||
"level": "Metadata",
|
||||
"timestamp": "2017-06-15T23:07:40Z",
|
||||
"auditID": "4faf711a-9094-400f-a876-d9188ceda548",
|
||||
"stage": "ResponseComplete",
|
||||
"requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer",
|
||||
"verb": "get",
|
||||
"user": {
|
||||
"username": "system:apiserver",
|
||||
"uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e",
|
||||
"groups": [
|
||||
"system:masters"
|
||||
]
|
||||
},
|
||||
"sourceIPs": [
|
||||
"127.0.0.1"
|
||||
],
|
||||
"objectRef": {
|
||||
"resource": "rolebindings",
|
||||
"namespace": "kube-public",
|
||||
"name": "system:controller:bootstrap-signer",
|
||||
"apiVersion": "rbac.authorization.k8s.io/v1beta1"
|
||||
},
|
||||
"responseStatus": {
|
||||
"metadata": {},
|
||||
"code": 200
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1alpha1/types.go
|
||||
[kube-apiserver]: /docs/admin/kube-apiserver
|
||||
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/cluster/gce/gci/configure-helper.sh#L490
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: event-exporter-sa
|
||||
namespace: default
|
||||
labels:
|
||||
app: event-exporter
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1beta1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: event-exporter-rb
|
||||
namespace: default
|
||||
labels:
|
||||
app: event-exporter
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: view
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: event-exporter-sa
|
||||
namespace: default
|
||||
---
|
||||
apiVersion: apps/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: event-exporter-v0.1.0
|
||||
namespace: default
|
||||
labels:
|
||||
app: event-exporter
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: event-exporter
|
||||
spec:
|
||||
serviceAccountName: event-exporter-sa
|
||||
containers:
|
||||
- name: event-exporter
|
||||
image: gcr.io/google-containers/event-exporter:v0.1.0
|
||||
command:
|
||||
- '/event-exporter'
|
||||
terminationGracePeriodSeconds: 30
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
assignees:
|
||||
- crassirostris
|
||||
- piosz
|
||||
title: Events in Stackdriver
|
||||
---
|
||||
|
||||
|
||||
|
||||
Kubernetes events are objects that provide insight into what is happening
|
||||
inside a cluster, such as what decisions were made by scheduler or why some
|
||||
pods were evicted from the node. You can read more about using events
|
||||
for debugging your application in the [Application Introspection and Debugging
|
||||
](/docs/tasks/debug-application-cluster/debug-application-introspection/)
|
||||
section.
|
||||
|
||||
Since events are API objects, they are stored in the apiserver on master. To
|
||||
avoid filling up master's disk, a retention policy is enforced: events are
|
||||
removed one hour after the last occurrence. To provide longer history
|
||||
and aggregation capabilities, a third party solution should be installed
|
||||
to capture events.
|
||||
|
||||
This article describes a solution that exports Kubernetes events to
|
||||
Stackdriver Logging, where they can be processed and analyzed.
|
||||
|
||||
**Note:** it is not guaranteed that all events happening in a cluster will be
|
||||
exported to Stackdriver. One possible scenario when events will not be
|
||||
exported is when event exporter is not running (e.g. during restart or
|
||||
upgrade). In most cases it's fine to use events for purposes like setting up
|
||||
[metrics][sdLogMetrics] and [alerts][sdAlerts], but you should be aware
|
||||
of the potential inaccuracy.
|
||||
|
||||
[sdLogMetrics]: https://cloud.google.com/logging/docs/view/logs_based_metrics
|
||||
[sdAlerts]: https://cloud.google.com/logging/docs/view/logs_based_metrics#creating_an_alerting_policy
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Deployment
|
||||
|
||||
### Google Container Engine
|
||||
|
||||
In Google Container Engine (GKE), if cloud logging is enabled, event exporter
|
||||
is deployed by default to the clusters with master running version 1.7 and
|
||||
higher. To prevent disturbing your workloads, event exporter does not have
|
||||
resources set and is in the best effort QOS class, which means that it will
|
||||
be the first to be killed in the case of resource starvation. If you want
|
||||
your events to be exported, make sure you have enough resources to facilitate
|
||||
the event exporter pod. This may vary depending on the workload, but on
|
||||
average, approximately 100Mb RAM and 100m CPU is needed.
|
||||
|
||||
### Deploying to the Existing Cluster
|
||||
|
||||
Deploy event exporter to your cluster using the following command:
|
||||
|
||||
```shell
|
||||
kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/event-exporter-deploy.yaml
|
||||
```
|
||||
|
||||
Since event exporter accesses the Kubernetes API, it requires permissions to
|
||||
do so. The following deployment is configured to work with RBAC
|
||||
authorization. It sets up a service account and a cluster role binding
|
||||
to allow event exporter to read events. To make sure that event exporter
|
||||
pod will not be evicted from the node, you can additionally set up resource
|
||||
requests. As mentioned earlier, 100Mb RAM and 100m CPU should be enough.
|
||||
|
||||
{% include code.html language="yaml" file="event-exporter-deploy.yaml" ghlink="/docs/tasks/debug-application-cluster/event-exporter-deploy.yaml" %}
|
||||
|
||||
## User Guide
|
||||
|
||||
Events are exported to the `GKE Cluster` resource in Stackdriver Logging.
|
||||
You can find them by selecting an appropriate option from a drop-down menu
|
||||
of available resources:
|
||||
|
||||
<img src="/images/docs/stackdriver-event-exporter-resource.png" alt="Events location in the Stackdriver Logging interface" width="500">
|
||||
|
||||
You can filter based on the event object fields using Stackdriver Logging
|
||||
[filtering mechanism](https://cloud.google.com/logging/docs/view/advanced_filters).
|
||||
For example, the following query will show events from the scheduler
|
||||
about pods from deployment `nginx-deployment`:
|
||||
|
||||
```
|
||||
resource.type="gke_cluster"
|
||||
jsonPayload.kind="Event"
|
||||
jsonPayload.source.component="default-scheduler"
|
||||
jsonPayload.involvedObject.name:"nginx-deployment"
|
||||
```
|
||||
|
||||
<img src="/images/docs/stackdriver-event-exporter-filter.png" alt="Filtered events in the Stackdriver Logging interface" width="500">
|
||||
@@ -0,0 +1,34 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: opa
|
||||
name: opa
|
||||
namespace: federation-system
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: opa
|
||||
name: opa
|
||||
spec:
|
||||
containers:
|
||||
- name: opa
|
||||
image: openpolicyagent/opa:0.4.10
|
||||
args:
|
||||
- "run"
|
||||
- "--server"
|
||||
- name: kube-mgmt
|
||||
image: openpolicyagent/kube-mgmt:0.2
|
||||
args:
|
||||
- "-kubeconfig=/srv/kubernetes/kubeconfig"
|
||||
- "-cluster=federation/v1beta1/clusters"
|
||||
volumeMounts:
|
||||
- name: federation-kubeconfig
|
||||
mountPath: /srv/kubernetes
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: federation-kubeconfig
|
||||
secret:
|
||||
secretName: federation-controller-manager-kubeconfig
|
||||
@@ -0,0 +1,13 @@
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: opa
|
||||
namespace: federation-system
|
||||
spec:
|
||||
selector:
|
||||
app: opa
|
||||
ports:
|
||||
- name: http
|
||||
protocol: TCP
|
||||
port: 8181
|
||||
targetPort: 8181
|
||||
@@ -0,0 +1,74 @@
|
||||
# OPA supports a high-level declarative language named Rego for authoring and
|
||||
# enforcing policies. For more infomration on Rego, visit
|
||||
# http://openpolicyagent.org.
|
||||
|
||||
# Rego policies are namespaced by the "package" directive.
|
||||
package kubernetes.placement
|
||||
|
||||
# Imports provide aliases for data inside the policy engine. In this case, the
|
||||
# policy simply refers to "clusters" below.
|
||||
import data.kubernetes.clusters
|
||||
|
||||
# The "annotations" rule generates a JSON object containing the key
|
||||
# "federation.kubernetes.io/replica-set-preferences" mapped to <preferences>.
|
||||
# The preferences values is generated dynamically by OPA when it evaluates the
|
||||
# rule.
|
||||
#
|
||||
# The SchedulingPolicy Admission Controller running inside the Federation API
|
||||
# server will merge these annotatiosn into incoming Federated resources. By
|
||||
# setting replica-set-preferences, we can control the placement of Federated
|
||||
# ReplicaSets.
|
||||
#
|
||||
# Rules are defined to generate JSON values (booleans, strings, objects, etc.)
|
||||
# When OPA evaluates a rule, it generates a value IF all of the expressions in
|
||||
# the body evaluate successfully. All rules can be understood intuitively as
|
||||
# <head> if <body> where <body> is true if <expr-1> AND <expr-2> AND ...
|
||||
# <expr-N> is true (for some set of data.)
|
||||
annotations["federation.kubernetes.io/replica-set-preferences"] = preferences {
|
||||
input.kind = "ReplicaSet"
|
||||
value = {"clusters": cluster_map, "rebalance": true}
|
||||
json.marshal(value, preferences)
|
||||
}
|
||||
|
||||
# This "annotations" rule generates a value for the "federation.alpha.kubernetes.io/cluster-selector"
|
||||
# annotation.
|
||||
#
|
||||
# In English, the policy asserts that resources in the "production" namespace
|
||||
# that are not annotated with "criticality=low" MUST be placed on clusters
|
||||
# labelled with "on-premise=true".
|
||||
annotations["federation.alpha.kubernetes.io/cluster-selector"] = selector {
|
||||
input.metadata.namespace = "production"
|
||||
not input.metadata.annotations.criticality = "low"
|
||||
json.marshal([{
|
||||
"operator": "=",
|
||||
"key": "on-premise",
|
||||
"values": "[true]",
|
||||
}], selector)
|
||||
}
|
||||
|
||||
# Generates a set of cluster names that satisfy the incoming Federated
|
||||
# ReplicaSet's requirements. In this case, just PCI compliance.
|
||||
replica_set_clusters[cluster_name] {
|
||||
clusters[cluster_name]
|
||||
not insufficient_pci[cluster_name]
|
||||
}
|
||||
|
||||
# Generates a set of clusters that must not be used for Federated ReplicaSets
|
||||
# that request PCI compliance.
|
||||
insufficient_pci[cluster_name] {
|
||||
clusters[cluster_name]
|
||||
input.metadata.annotations["requires-pci"] = "true"
|
||||
not pci_clusters[cluster_name]
|
||||
}
|
||||
|
||||
# Generates a set of clusters that are PCI certified. In this case, we assume
|
||||
# clusters are annotated to indicate if they have passed PCI compliance audits.
|
||||
pci_clusters[cluster_name] {
|
||||
clusters[cluster_name].metadata.annotations["pci-certified"] = "true"
|
||||
}
|
||||
|
||||
# Helper rule to generate a mapping of desired clusters to weights. In this
|
||||
# case, weights are static.
|
||||
cluster_map[cluster_name] = {"weight": 1} {
|
||||
replica_set_clusters[cluster_name]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: ReplicaSet
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx-pci
|
||||
name: nginx-pci
|
||||
annotations:
|
||||
requires-pci: "true"
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx-pci
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx-pci
|
||||
spec:
|
||||
containers:
|
||||
- image: nginx
|
||||
name: nginx-pci
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: admission
|
||||
namespace: federation-system
|
||||
data:
|
||||
config.yml: |
|
||||
apiVersion: apiserver.k8s.io/v1alpha1
|
||||
kind: AdmissionConfiguration
|
||||
plugins:
|
||||
- name: SchedulingPolicy
|
||||
path: /etc/kubernetes/admission/scheduling-policy-config.yml
|
||||
scheduling-policy-config.yml: |
|
||||
kubeconfig: /etc/kubernetes/admission/opa-kubeconfig
|
||||
opa-kubeconfig: |
|
||||
clusters:
|
||||
- name: opa-api
|
||||
cluster:
|
||||
server: http://opa.federation-system.svc.cluster.local:8181/v0/data/kubernetes/placement
|
||||
users:
|
||||
- name: scheduling-policy
|
||||
user:
|
||||
token: deadbeefsecret
|
||||
contexts:
|
||||
- name: default
|
||||
context:
|
||||
cluster: opa-api
|
||||
user: scheduling-policy
|
||||
current-context: default
|
||||
@@ -97,9 +97,11 @@ coredns-provider.conf has below format:
|
||||
[Global]
|
||||
etcd-endpoints = http://etcd-cluster.my-namespace:2379
|
||||
zones = example.com.
|
||||
coredns-endpoints = <coredns-server-ip>:<port>
|
||||
|
||||
- `etcd-endpoints` is the endpoint to access etcd.
|
||||
- `zones` is the federation domain for which CoreDNS is authoritative and is same as --dns-zone-name flag of `kubefed init`.
|
||||
- `coredns-endpoints` is the endpoint to access coredns server. This is an optional parameter introduced from v1.7 onwards.
|
||||
|
||||
*Note: middleware.etcd.zones in CoreDNS configuration and --dns-zone-name
|
||||
flag to kubefed init should match.*
|
||||
@@ -107,6 +109,11 @@ flag to kubefed init should match.*
|
||||
|
||||
## Setup CoreDNS server in nameserver resolv.conf chain
|
||||
|
||||
*Note: The following section applies only to versions prior to v1.7
|
||||
and will be automatically taken care of if the `coredns-endpoints`
|
||||
parameter is configured in `coredns-provider.conf` as described in
|
||||
section above.*
|
||||
|
||||
Once the federation control plane is deployed and federated clusters
|
||||
are joined to the federation, you need to add the CoreDNS server to the
|
||||
pod's nameserver resolv.conf chain in all the federated clusters as this
|
||||
@@ -118,9 +125,6 @@ achieved by adding the below line to `dnsmasq` container's arg in
|
||||
|
||||
Replace `example.com` above with federation domain.
|
||||
|
||||
*Note: Adding CoreDNS server to the pod's nameserver resolv.conf chain will be
|
||||
automated in subsequent releases.*
|
||||
|
||||
|
||||
Now the federated cluster is ready for cross-cluster service discovery!
|
||||
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: Set up placement policies in Federation
|
||||
redirect_from:
|
||||
- "/docs/tutorials/federation/set-up-placement-policies-federation/"
|
||||
- "/docs/tutorials/federation/set-up-placement-policies-federation.html"
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
This page shows how to enforce policy-based placement decisions over Federated
|
||||
resources using an external policy engine.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
You need to have a running Kubernetes cluster (which is referenced as host
|
||||
cluster). Please see one of the [getting started](/docs/getting-started-guides/)
|
||||
guides for installation instructions for your platform.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Deploying Federation and configuring an external policy engine
|
||||
|
||||
The Federation control plane can be deployed using `kubefed init`.
|
||||
|
||||
After deploying the Federation control plane, you must configure an Admission
|
||||
Controller in the Federation API server that enforces placement decisions
|
||||
received from the external policy engine.
|
||||
|
||||
kubectl create -f scheduling-policy-admission.yaml
|
||||
|
||||
Shown below is an example ConfigMap for the Admission Controller:
|
||||
|
||||
{% include code.html language="yaml" file="scheduling-policy-admission.yaml"
|
||||
ghlink="/docs/tutorials/federation/scheduling-policy-admission.yaml" %}
|
||||
|
||||
The ConfigMap contains three files:
|
||||
|
||||
* `config.yml` specifies the location of the `SchedulingPolicy` Admission
|
||||
Controller config file.
|
||||
* `scheduling-policy-config.yml` specifies the location of the kubeconfig file
|
||||
required to contact the external policy engine. This file can also include a
|
||||
`retryBackoff` value that controls the initial retry backoff delay in
|
||||
milliseconds.
|
||||
* `opa-kubeconfig` is a standard kubeconfig containing the URL and credentials
|
||||
needed to contact the external policy engine.
|
||||
|
||||
Edit the Federation API server deployment to enable the `SchedulingPolicy`
|
||||
Admission Controller.
|
||||
|
||||
kubectl -n federation-system edit deployment federation-apiserver
|
||||
|
||||
Update the Federation API server command line arguments to enable the Admission
|
||||
Controller and mount the ConfigMap into the container. If there's an existing
|
||||
`--admission-control` flag, append `,SchedulingPolicy` instead of adding
|
||||
another line.
|
||||
|
||||
--admission-control=SchedulingPolicy
|
||||
--admission-control-config-file=/etc/kubernetes/admission/config.yml
|
||||
|
||||
Add the following volume to the Federation API server pod:
|
||||
|
||||
- name: admission-config
|
||||
configMap:
|
||||
name: admission
|
||||
|
||||
Add the following volume mount the Federation API server `apiserver` container:
|
||||
|
||||
volumeMounts:
|
||||
- name: admission-config
|
||||
mountPath: /etc/kubernetes/admission
|
||||
|
||||
## Deploying an external policy engine
|
||||
|
||||
The [Open Policy Agent (OPA)](http://openpolicyagent.org) is an open source,
|
||||
general-purpose policy engine that you can use to enforce policy-based placement
|
||||
decisions in the Federation control plane.
|
||||
|
||||
Create a Service in the host cluster to contact the external policy engine:
|
||||
|
||||
kubectl create -f policy-engine-service.yaml
|
||||
|
||||
Shown below is an example Service for OPA.
|
||||
|
||||
{% include code.html language="yaml" file="policy-engine-service.yaml"
|
||||
ghlink="/docs/tutorials/federation/policy-engine-service.yaml" %}
|
||||
|
||||
Create a Deployment in the host cluster with the Federation control plane:
|
||||
|
||||
kubectl create -f policy-engine-deployment.yaml
|
||||
|
||||
Shown below is an example Deployment for OPA.
|
||||
|
||||
{% include code.html language="yaml" file="policy-engine-deployment.yaml"
|
||||
ghlink="/docs/tutorials/federation/policy-engine-deployment.yaml" %}
|
||||
|
||||
## Configuring placement policies via ConfigMaps
|
||||
|
||||
The external policy engine will discover placement policies created in the
|
||||
`kube-federation-scheduling-policy` namespace in the Federation API server.
|
||||
|
||||
Create the namespace if it does not already exist:
|
||||
|
||||
kubectl --context=federation create namespace kube-federation-scheduling-policy
|
||||
|
||||
Configure a sample policy to test the external policy engine:
|
||||
|
||||
{% include code.html language="yaml" file="policy.rego"
|
||||
ghlink="/docs/tutorials/federation/policy.rego" %}
|
||||
|
||||
Shown below is the command to create the sample policy:
|
||||
|
||||
kubectl --context=federation -n kube-federation-scheduling-policy create configmap scheduling-policy --from-file=policy.rego
|
||||
|
||||
This sample policy illustrates a few key ideas:
|
||||
|
||||
* Placement policies can refer to any field in Federated resources.
|
||||
* Placement policies can leverage external context (for example, Cluster
|
||||
metadata) to make decisions.
|
||||
* Administrative policy can be managed centrally.
|
||||
* Policies can define simple interfaces (such as the `requires-pci` annotation) to
|
||||
avoid duplicating logic in manifests.
|
||||
|
||||
## Testing placement policies
|
||||
|
||||
Annotate one of the clusters to indicate that it is PCI certified.
|
||||
|
||||
kubectl --context=federation annotate clusters cluster-name-1 pci-certified=true
|
||||
|
||||
Deploy a Federated ReplicaSet to test the placement policy.
|
||||
|
||||
{% include code.html language="yaml" file="replicaset-example-policy.yaml"
|
||||
ghlink="/docs/tutorials/federation/replicaset-example-policy.yaml" %}
|
||||
|
||||
Shown below is the command to deploy a ReplicaSet that *does* match the policy.
|
||||
|
||||
kubectl --context=federation create -f replicaset-example-policy.yaml
|
||||
|
||||
Inspect the ReplicaSet to confirm the appropriate annotations have been applied:
|
||||
|
||||
kubectl --context=federation get rs nginx-pci -o jsonpath='{.metadata.annotations}'
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -51,7 +51,7 @@ Kubernetes modifies the Pod Spec.
|
||||
|
||||
Kubernetes annotates the resulting modified pod spec to show that it was
|
||||
modified by a `PodPreset`. The annotation is of the form
|
||||
`podpreset.admission.kubernetes.io/<pod-preset name>": "<resource version>"`.
|
||||
`podpreset.admission.kubernetes.io/podpreset-<pod-preset name>": "<resource version>"`.
|
||||
|
||||
|
||||
## Enable Pod Preset
|
||||
@@ -63,6 +63,12 @@ following
|
||||
1. You have enabled the admission controller `PodPreset`
|
||||
1. You have defined your pod presets
|
||||
|
||||
## Disable Pod Preset for a pod
|
||||
|
||||
There may be instances where you wish for a pod to not be altered by any pod
|
||||
preset mutations. For these events, one can add an annotation in the pod spec
|
||||
of the form: `podpreset.admission.kubernetes.io/exclude: "true"`.
|
||||
|
||||
## Create a Pod Preset
|
||||
|
||||
### Simple Pod Spec Example
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
assignees:
|
||||
- janetkuo
|
||||
title: Performing a Rollback on a DaemonSet
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
This page shows how to perform a rollback on a DaemonSet.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
{% capture prerequisites %}
|
||||
|
||||
* The DaemonSet rollout history and DaemonSet rollback features are only
|
||||
supported in `kubectl` in Kubernetes version 1.7 or later.
|
||||
* Make sure you know how to [perform a rolling update on a
|
||||
DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/)
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Performing a Rollback on a DaemonSet
|
||||
|
||||
### Step 1: Find the DaemonSet revision you want to roll back to
|
||||
|
||||
You can skip this step if you just want to roll back to the last revision.
|
||||
|
||||
List all revisions of a DaemonSet:
|
||||
|
||||
```shell
|
||||
kubectl rollout history daemonset <daemonset-name>
|
||||
```
|
||||
|
||||
This returns a list of DaemonSet revisions:
|
||||
|
||||
```shell
|
||||
daemonsets "<daemonset-name>"
|
||||
REVISION CHANGE-CAUSE
|
||||
1 ...
|
||||
2 ...
|
||||
...
|
||||
```
|
||||
|
||||
* Change cause is copied from DaemonSet annotation `kubernetes.io/change-cause`
|
||||
to its revisions upon creation. You may specify `--record=true` in `kubectl`
|
||||
to record the command executed in the change cause annotation.
|
||||
|
||||
To see the details of a specific revision:
|
||||
|
||||
```shell
|
||||
kubectl rollout history daemonset <daemonset-name> --revision=1
|
||||
```
|
||||
|
||||
This returns the details of that revision:
|
||||
|
||||
```shell
|
||||
daemonsets "<daemonset-name>" with revision #1
|
||||
Pod Template:
|
||||
Labels: foo=bar
|
||||
Containers:
|
||||
app:
|
||||
Image: ...
|
||||
Port: ...
|
||||
Environment: ...
|
||||
Mounts: ...
|
||||
Volumes: ...
|
||||
```
|
||||
|
||||
### Step 2: Roll back to a specific revision
|
||||
|
||||
```shell
|
||||
# Specify the revision number you get from Step 1 in --to-revision
|
||||
kubectl rollout undo daemonset <daemonset-name> --to-revision=<revision>
|
||||
```
|
||||
|
||||
If it succeeds, the command returns:
|
||||
|
||||
```shell
|
||||
daemonset "<daemonset-name>" rolled back
|
||||
```
|
||||
|
||||
If `--to-revision` flag is not specified, the last revision will be picked.
|
||||
|
||||
### Step 3: Watch the progress of the DaemonSet rollback
|
||||
|
||||
`kubectl rollout undo daemonset` tells the server to start rolling back the
|
||||
DaemonSet. The real rollback is done asynchronously on the server side.
|
||||
|
||||
To watch the progress of the rollback:
|
||||
|
||||
```shell
|
||||
kubectl rollout status ds/<daemonset-name>
|
||||
```
|
||||
|
||||
When the rollback is complete, the output is similar to this:
|
||||
|
||||
```shell
|
||||
daemon set "<daemonset-name>" successfully rolled out
|
||||
```
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
{% capture discussion %}
|
||||
|
||||
## Understanding DaemonSet Revisions
|
||||
|
||||
In the previous `kubectl rollout history` step, you got a list of DaemonSet
|
||||
revisions. Each revision is stored in a resource named `ControllerRevision`.
|
||||
`ControllerRevision` is a resource only available in Kubernetes release 1.7 or
|
||||
later.
|
||||
|
||||
To see what is stored in each revision, find the DaemonSet revision raw
|
||||
resources:
|
||||
|
||||
```shell
|
||||
kubectl get controllerrevision -l <daemonset-selector-key>=<daemonset-selector-value>
|
||||
```
|
||||
|
||||
This returns a list of `ControllerRevisions`:
|
||||
|
||||
```shell
|
||||
NAME CONTROLLER REVISION AGE
|
||||
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 1 1h
|
||||
<daemonset-name>-<revision-hash> DaemonSet/<daemonset-name> 2 1h
|
||||
```
|
||||
|
||||
Each `ControllerRevision` stores the annotations and template of a DaemonSet
|
||||
revision.
|
||||
|
||||
`kubectl rollout undo` takes a specific `ControllerRevision` and replaces
|
||||
DaemonSet template with the template stored in the `ControllerRevision`.
|
||||
`kubectl rollout undo` is equivalent to updating DaemonSet template to a
|
||||
previous revision through other commands, such as `kubectl edit` or `kubectl
|
||||
apply`.
|
||||
|
||||
Note that DaemonSet revisions only roll forward. That is to say, after a
|
||||
rollback is complete, the revision number (`.revision` field) of the
|
||||
`ControllerRevision` being rolled back to will advance. For example, if you
|
||||
have revision 1 and 2 in the system, and roll back from revision 2 to revision
|
||||
1, the `ControllerRevision` with `.revision: 1` will become `.revision: 3`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
* See [troubleshooting DaemonSet rolling
|
||||
update](/docs/tasks/manage-daemon/update-daemon-set/#troubleshooting)
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -33,46 +33,18 @@ DaemonSet has two update strategy types :
|
||||
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
|
||||
will be created automatically, in a controlled fashion.
|
||||
|
||||
## Limitations
|
||||
|
||||
* DaemonSet rollout history is not supported yet.
|
||||
* DaemonSet rollback is not directly supported in `kubectl` yet. You can rollback
|
||||
by updating DaemonSet template to match the previous version.
|
||||
|
||||
## Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before
|
||||
|
||||
If you try to rolling update a DaemonSet that was created from Kubernetes
|
||||
If you try a rolling update on a DaemonSet that was created from Kubernetes
|
||||
version 1.5 or before, a rollout will be triggered when you *first* change the
|
||||
DaemonSet update strategy to `RollingUpdate`, even when DaemonSet template isn't
|
||||
modified. All existing DaemonSet pods will be restarted.
|
||||
DaemonSet update strategy to `RollingUpdate`, no matter if DaemonSet template is
|
||||
modified or not. If the DaemonSet template is not changed, all existing DaemonSet
|
||||
pods will be restarted (deleted and created).
|
||||
|
||||
To avoid this restart, first find the DaemonSet's current
|
||||
`.spec.templateGeneration`:
|
||||
Therefore, make sure you want to trigger a rollout before you first switch the
|
||||
strategy to `RollingUpdate`.
|
||||
|
||||
```shell{% raw %}
|
||||
kubectl get ds/<daemonset-name> -o go-template='{{.spec.templateGeneration}}{{"\n"}}'
|
||||
{% endraw %}```
|
||||
|
||||
The output should be a number *N*. If the output shows `<no value>`, N = 0.
|
||||
|
||||
Then, simply label the existing DaemonSet pods with `pod-template-generation=<N>`
|
||||
before changing DaemonSet `.spec.updateStrategy` to `RollingUpdate`:
|
||||
|
||||
```shell
|
||||
# Replace N with DaemonSet `.spec.templateGeneration`
|
||||
# Only run this on pods created from current DaemonSet template
|
||||
kubectl label pods -l <daemonset-selector-key>=<daemonset-selector-value> pod-template-generation=<N>
|
||||
```
|
||||
|
||||
This tells the DaemonSet that the labeled DaemonSet pods are created from current
|
||||
DaemonSet template. Therefore, you should only run this command to existing DaemonSet
|
||||
pods that are generated from current DaemonSet template.
|
||||
|
||||
Note that you only need to do this when you first change the update strategy of
|
||||
a DaemonSet created from Kubernetes version 1.5 or before to `RollingUpdate`,
|
||||
but don't want to update its template and start a new rollout yet.
|
||||
|
||||
## Setting DaemonSet update strategy for rolling update
|
||||
## Performing a Rolling Update
|
||||
|
||||
To enable the rolling update feature of a DaemonSet, you must set its
|
||||
`.spec.updateStrategy.type` to `RollingUpdate`.
|
||||
@@ -87,15 +59,15 @@ First, check the update strategy of your DaemonSet, and make sure it's set to
|
||||
RollingUpdate:
|
||||
|
||||
```shell{% raw %}
|
||||
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'{% endraw %}
|
||||
```
|
||||
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
|
||||
``` {% endraw %}
|
||||
|
||||
If you haven't created the DaemonSet in the system, check your DaemonSet
|
||||
manifest with the following command instead:
|
||||
|
||||
```shell{% raw %}
|
||||
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
|
||||
{% endraw %}```
|
||||
``` {% endraw %}
|
||||
|
||||
The output from both commands should be:
|
||||
|
||||
@@ -224,7 +196,9 @@ progress.
|
||||
|
||||
{% capture whatsnext %}
|
||||
|
||||
*TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
|
||||
* See [Task: Performing a rollback on a
|
||||
DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/)
|
||||
* *TODO: Link to "Task: Creating a DaemonSet to adopt existing DaemonSet pods"*
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
|
||||
@@ -109,8 +109,6 @@ The API presented here **will change** in an upcoming release to better support
|
||||
|
||||
As of now, CUDA libraries are expected to be pre-installed on the nodes.
|
||||
|
||||
The NVIDIA drivers will require privileged containers due to the permissions on ``/usr/lib/nvidia-367``.
|
||||
|
||||
To mitigate this, you can copy the libraries to a more permissive folder in ``/var/lib/`` or change the permissions directly. (Future releases will automatically perform this operation)
|
||||
|
||||
Pods can access the libraries using `hostPath` volumes.
|
||||
@@ -135,10 +133,10 @@ spec:
|
||||
name: lib
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /usr/lib/nvidia-367/bin
|
||||
path: /usr/lib/nvidia-375/bin
|
||||
name: bin
|
||||
- hostPath:
|
||||
path: /usr/lib/nvidia-367
|
||||
path: /usr/lib/nvidia-375
|
||||
name: lib
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: Specifying a Disruption Budget for your Application
|
||||
---
|
||||
|
||||
{% capture overview %}
|
||||
|
||||
This page shows how to limit the number of concurrent disruptions
|
||||
that your application experiences, allowing for higher availability
|
||||
while permitting the cluster administrator to manage the clusters
|
||||
nodes.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture prerequisites %}
|
||||
* You are the owner of an application running on a Kubernetes cluster that requires
|
||||
high availability.
|
||||
* You should know how to deploy [Replicated Stateless Applications](/docs/tasks/run-application/run-stateless-application-deployment.md)
|
||||
and/or [Replicated Stateful Applications](/docs/tasks/run-application/run-replicated-stateful-application.md).
|
||||
* You should have read about [Pod Disruptions](/docs/concepts/workloads/pods/disruptions/).
|
||||
* You should confirm with your cluster owner or service provider that they respect
|
||||
Pod Disruption Budgets.
|
||||
{% endcapture %}
|
||||
|
||||
{% capture steps %}
|
||||
|
||||
## Protecting an Application with a PodDisruptionBudget
|
||||
|
||||
1. Identify what application you want to protect with a PodDisruptionBudget (PDB).
|
||||
1. Think about how your application reacts to disruptions
|
||||
1. Create a PDB definition as a YAML file.
|
||||
1. Create the PDB object from the YAML file.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% capture discussion %}
|
||||
|
||||
## Identify an Application to Protect
|
||||
|
||||
The most common use case when you want to protect an application
|
||||
specified by one of the built-in Kubernetes controllers:
|
||||
|
||||
- Deployment
|
||||
- ReplicationController
|
||||
- ReplicaSet
|
||||
- StatefulSet
|
||||
|
||||
In this case, make a note of the controller's `.spec.selector`; the same
|
||||
selector goes into the PDBs `.spec.selector`.
|
||||
|
||||
You can also use PDBs with pods which are not controlled by one of the above
|
||||
controllers, or arbitrary groups of pods, but there are some restrictions,
|
||||
described in [Arbitrary Controllers and Selectors](#arbitrary-controllers-and-selectors).
|
||||
|
||||
|
||||
## Think about how your application reacts to disruptions
|
||||
|
||||
Decide how many instances can be down at the same time for a short period
|
||||
due to a voluntary disruption.
|
||||
|
||||
- Stateless frontends
|
||||
- Concern: don't reduce serving capacity by more than 10%.
|
||||
- Solution: use PDB with minAvailable 90% for example.
|
||||
- Single-instance Stateful Application
|
||||
- Concern: do not terminate this application without talking to me.
|
||||
- Possible Solution 1: Do not use a PDB and tolerate occasional downtime.
|
||||
- Possible Solution 2: Set PDB with maxUnavailable=0. Have an understanding
|
||||
(outside of Kubernetes) that the cluster operator needs to consult you before
|
||||
termination. When the cluster operator contacts you, prepare for downtime,
|
||||
and then delete the PDB to indicate readiness for disruption. Recreate afterwards.
|
||||
- Multiple-instance Stateful application such as Consul, ZooKeeper, or etcd
|
||||
- Concern: Do not reduce number of instances below quorum, otherwise writes fail.
|
||||
- Possible Solution 1: set maxUnavailable to 1 (works with varying scale of application).
|
||||
- Possible Solution 2: set minAvailable to quorum-size (e.g. 3 when scale is 5). (Allows more disruptions at once).
|
||||
- Restartable Batch Job:
|
||||
- Concern: Job needs to complete in case of voluntary disruption
|
||||
- Possible solution: Do not create a PDB. The Job controller will create a replacement pod.
|
||||
|
||||
## Specifying a PodDisruptionBudget
|
||||
|
||||
A `PodDisruptionBudget` has three fields:
|
||||
|
||||
* A label selector `.spec.selector` to specify the set of
|
||||
pods to which it applies. This field is required.
|
||||
* `.spec.minAvailable` which is a description of the number of pods from that
|
||||
set that must still be available after the eviction, even in the absence
|
||||
of the evicted pod. `minAvailable` can be either an absolute number or a percentage.
|
||||
* `.spec.maxUnavailable` (available in Kubernetes 1.7 and higher) which is a description
|
||||
of the number of pods from that set that can be unavailable after the eviction.
|
||||
It can be either an absolute number or a percentage.
|
||||
|
||||
You can specify only one of `maxUnavailable` and `minAvailable` in a single `PodDisruptionBudget`.
|
||||
`maxUnavailable` can only be used to control the eviction of pods
|
||||
that have an associated controller managing them. In the examples below, "desired replicas"
|
||||
is the `scale` of the controller managing the pods being selected by the
|
||||
`PodDisruptionBudget`.
|
||||
|
||||
Example 1: With a `minAvailable` of 5, evictions are be allowed as long as they leave behind
|
||||
5 or more healthy pods among those selected by the PodDisruptionBudget's `selector`.
|
||||
|
||||
Example 2: With a `minAvailable` of 30%, evictions are allowed as long as at least 30%
|
||||
of the number of desired replicas are healthy.
|
||||
|
||||
Example 3: With a `maxUnavailable` of 5, evictions are allowed as long as there are at most 5
|
||||
unhealthy replicas among the total number of desired replicas.
|
||||
|
||||
Example 4: With a `maxUnavailable` of 30%, evictions are allowed as long as no more than 30%
|
||||
of the desired replicas are unhealthy.
|
||||
|
||||
In typical usage, a single budget would be used for a collection of pods managed by
|
||||
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
|
||||
|
||||
**Note:** A disruption budget does not truly guarantee that the specified
|
||||
number/percentage of pods will always be up. For example, a node that hosts a
|
||||
pod from the collection may fail when the collection is at the minimum size
|
||||
specified in the budget, thus bringing the number of available pods from the
|
||||
collection below the specified size. The budget can only protect against
|
||||
voluntary evictions, not all causes of unavailability.
|
||||
|
||||
A `maxUnavailable` of 0% (or 0) or a `minAvailable` of 100% (or equal to the
|
||||
number of replicas) may block node drains entirely. This is permitted as per the
|
||||
semantics of `PodDisruptionBudget`.
|
||||
|
||||
You can find examples of pod disruption budgets defined below. They match pods with the label
|
||||
`app: zookeeper`.
|
||||
|
||||
Example PDB Using minAvailable:
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: zk-pdb
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: zookeeper
|
||||
```
|
||||
|
||||
Example PDB Using maxUnavailable (Kubernetes 1.7 or higher):
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: zk-pdb
|
||||
spec:
|
||||
maxUnavailable: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: zookeeper
|
||||
```
|
||||
|
||||
For example, if the above `zk-pdb` object selects the pods of a StatefulSet of size 3, both
|
||||
specifications have the exact same meaning. The use of `maxUnavailable` is recommended as it
|
||||
automatically responds to changes in the number of replicas of the corresponding controller.
|
||||
|
||||
# Create the PDB object
|
||||
|
||||
You can create the PDB object with a command like `kubectl create -f mypdb.yaml`.
|
||||
|
||||
You cannot update PDB objects. They must be deleted and re-created.
|
||||
|
||||
# Check the status of the PDB
|
||||
|
||||
Use kubectl to check that your PDB is created.
|
||||
|
||||
Assuming you don't actually have pods matching `app: zookeeper` in your namespace,
|
||||
then you'll see something like this:
|
||||
|
||||
```shell
|
||||
$ kubectl get poddisruptionbudgets
|
||||
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
|
||||
zk-pdb 2 0 7s
|
||||
```
|
||||
|
||||
If there are matching pods (say, 3), then you would see something like this:
|
||||
|
||||
```shell
|
||||
$ kubectl get poddisruptionbudgets
|
||||
NAME MIN-AVAILABLE ALLOWED-DISRUPTIONS AGE
|
||||
zk-pdb 2 1 7s
|
||||
```
|
||||
|
||||
The non-zero value for `ALLOWED-DISRUPTIONS` means that the disruption controller has seen the pods,
|
||||
counted the matching pods, and update the status of the PDB.
|
||||
|
||||
You can get more information about the status of a PDB with this command:
|
||||
|
||||
```shell
|
||||
$ kubectl get poddisruptionbudgets zk-pdb -o yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: zk-pdb
|
||||
...
|
||||
status:
|
||||
currentHealthy: 3
|
||||
desiredHealthy: 3
|
||||
disruptedPods: null
|
||||
disruptionsAllowed: 1
|
||||
expectedPods: 3
|
||||
observedGeneration: 1
|
||||
```
|
||||
|
||||
# Arbitrary Controllers and Selectors
|
||||
|
||||
You can skip this section if you only use PDBs with the built-in
|
||||
application controllers (Deployment, ReplicationController, ReplicaSet, and StatefulSet),
|
||||
with the PDB selector matching the controller's selector.
|
||||
|
||||
You can use a PDB with pods controlled by another type of controller, by an
|
||||
"operator", or bare pods, but with these restrictions:
|
||||
|
||||
- only `.spec.minAvailable` can be used, not `.spec.maxUnavailable`.
|
||||
- only an integer value can be used with `.spec.minAvailable`, not a percentage.
|
||||
|
||||
You can use a selector which selects a subset or superset of the pods belonging to a built-in
|
||||
controller. However, when there are multiple PDBs in a namespace, you must be careful not
|
||||
to create PDBs whose selectors overlap.
|
||||
|
||||
{% endcapture %}
|
||||
|
||||
{% include templates/task.md %}
|
||||
@@ -180,7 +180,7 @@ the only other supported resource metric is memory. These resources do not chan
|
||||
to cluster, and should always be available, as long as Heapster is deployed.
|
||||
|
||||
You can also specify resource metrics in terms of direct values, instead of as percentages of the
|
||||
requested value. To do so, use the `targetAverageValue` field insted of the `targetAverageUtilization`
|
||||
requested value. To do so, use the `targetAverageValue` field instead of the `targetAverageUtilization`
|
||||
field.
|
||||
|
||||
There are two other types of metrics, both of which are considered *custom metrics*: pod metrics and
|
||||
@@ -269,6 +269,48 @@ Then, your HorizontalPodAutoscaler would attempt to ensure that each pod was con
|
||||
50% of its requested CPU, serving 1000 packets per second, and that all pods behind the main-route
|
||||
Ingress were serving a total of 10000 requests per second.
|
||||
|
||||
## Appendix: Horizontal Pod Autoscaler Status Conditions
|
||||
|
||||
When using the `autoscaling/v2alpha1` form of the HorizontalPodAutoscaler, you will be able to see
|
||||
*status conditions* set by Kubernetes on the HorizontalPodAutoscaler. These status conditions indicate
|
||||
whether or not the HorizontalPodAutoscaler is able to scale, and whether or not it is currently restricted
|
||||
in any way.
|
||||
|
||||
The conditions appear in the `status.conditions` field. To see the conditions affecting a HorizontalPodAutoscaler,
|
||||
we can use `kubectl describe hpa`:
|
||||
|
||||
```shell
|
||||
$ kubectl describe hpa cm-test
|
||||
Name: cm-test
|
||||
Namespace: prom
|
||||
Labels: <none>
|
||||
Annotations: <none>
|
||||
CreationTimestamp: Fri, 16 Jun 2017 18:09:22 +0000
|
||||
Reference: ReplicationController/cm-test
|
||||
Metrics: ( current / target )
|
||||
"http_requests" on pods: 66m / 500m
|
||||
Min replicas: 1
|
||||
Max replicas: 4
|
||||
ReplicationController pods: 1 current / 1 desired
|
||||
Conditions:
|
||||
Type Status Reason Message
|
||||
---- ------ ------ -------
|
||||
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
|
||||
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric http_requests
|
||||
ScalingLimited False DesiredWithinRange the desired replica count is within the acceptible range
|
||||
Events:
|
||||
```
|
||||
|
||||
For this HorizontalPodAutoscaler, we can see several conditions in a healthy state. The first,
|
||||
`AbleToScale`, indicates whether or not the HPA is able to fetch and update scales, as well as
|
||||
whether or not any backoff-related conditions would prevent scaling. The second, `ScalingActive`,
|
||||
indicates whether or not the HPA is enabled (i.e. the replica count of the target is not zero) and
|
||||
is able to calculate desired scales. When it is `False`, it generally indicates problems with
|
||||
fetching metrics. Finally, the last condition, `ScalingLimitted`, indicates that the desired scale
|
||||
was capped by the maximum or minimum of the HorizontalPodAutoscaler. This is an indication that
|
||||
you may wish to raise or lower the minimum or maximum replica count constraints on your
|
||||
HorizontalPodAutoscaler.
|
||||
|
||||
## Appendix: Other possible scenarios
|
||||
|
||||
### Creating the autoscaler from a .yaml file
|
||||
|
||||
Reference in New Issue
Block a user