whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release 1.8 (#5659)

Browse Source 
* GC now supports non-core resources

* Add two examples about how to analysis audits of kube-apiserver (#4264)

* Deprecate system:nodes binding

* [1.8] StatefulSet `initialized` annotation is now ignored.

* inits the kubeadm upgrade docs

addresses kubernetes/kubernetes.github.io/issues/4689

* adds kubeadm upgrade cmd to ToC

addresses kubernetes/kubernetes.github.io/issues/4689

* add workload placement docs

* ScaleIO - document udpate for 1.8

* Add documentation on storageClass.mountOptions and PV.mountOptions (#5254)

* Add documentation on storageClass.mountOptions and PV.mountOptions

* convert notes into callouts

* Add docs for CustomResource validation

add info about supported fields

* advanced audit beta features (#5300)

* Update job workload doc with backoff failure policy (#5319)

Add to the Jobs documentation how to use the new backoffLimit field that
limit the number of Pod failure before considering the Job as failed.

* Documented additional AWS Service annotations (#4864)

* Add device plugin doc under concepts/cluster-administration. (#5261)

* Add device plugin doc under concepts/cluster-administration.

* Update device-plugins.md

* Update device-plugins.md

Add meta description. Fix typo. Change bare metal deployment to manual deployment.

* Update device-plugins.md

Fix typo again.

* Update page.version. (#5341)

* Add documentation on storageClass.reclaimPolicy (#5171)

* [Advanced audit] use new herf for audit-api (#5349)

This tag contains all the changes in v1beta1 version. Update it now.

* Added documentation around creating the InitializerConfiguration for the persistent volume label controller in the cloud-controller-manager (#5255)

* Documentation for kubectl plugins (#5294)

* Documentation for kubectl plugins

* Update kubectl-plugins.md

* Update kubectl-plugins.md

* Updated CPU manager docs to match implementation. (#5332)

* Noted limitation of alpha static cpumanager.

* Updated CPU manager docs to match implementation.

- Removed references to CPU pressure node condition and evictions.
- Added note about new --cpu-manager-reconcile-period flag.
- Added note about node allocatable requirements for static policy.
- Noted limitation of alpha static cpumanager.

* Move cpu-manager task link to rsc mgmt section.

* init containers annotation removed in  1.8 (#5390)

* Add documentation for TaintNodesByCondition (#5352)

* Add documentation for TaintNodesByCondition

* Update nodes.md

* Update taint-and-toleration.md

* Update daemonset.md

* Update nodes.md

* Update taint-and-toleration.md

* Update daemonset.md

* Fix deployments (#5421)

* Document extended resources and OIR deprecation. (#5399)

* Document extended resources and OIR deprecation.

* Updated extended resources doc per reviews.

* reverts extra spacing in _data/tasks.yml

* addresses `kubeadm upgrade` review comments

Feedback from @chenopis, @luxas, and @steveperry-53 addressed with this commit

* HugePages documentation (#5419)

* Update cpu-management-policies.md (#5407)

Fixed the bad link.
Modified "cpu" to "CPU".
Added more 'yaml' as supplement.

* Update RBAC docs for v1 (#5445)

* Add user docs for pod priority and preemption (#5328)

* Add user docs for pod priority and preemption

* Update pod-priority-preemption.md

* More updates

* Update docs/admin/kubeadm.md for 1.8 (#5440)

- Made a couple of minor wording changes (not strictly 1.8 related).
 - Did some reformatting (not strictly 1.8 related).
 - Updated references to the default token TTL (was infinite, now 24 hours).
 - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`.
 - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples.
 - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes.
 - Documented the new `--groups` flag for `kubeadm token create`.
 - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section.
 - Updated the component version table to drop 1.6 and add 1.8.
 - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`.

* Update StatefulSet Basics for 1.8 release (#5398)

* addresses `kubeadm upgrade` review comments

2nd iteration review comments by @luxas

* adds kubelet upgrade section to kubeadm upgrade

* Fix a bulleted list on docs/admin/kubeadm.md. (#5458)

I updated this doc yesterday and I was absolutely sure I fixed this, but I just saw that this commit got lost somehow.

This was introduced recently in https://github.com/kubernetes/kubernetes.github.io/pull/5440.

* Clarify the API to check for device plugins

* Moving Flexvolume to separate out-of-tree section

* addresses `kubeadm upgrade` review comments

CC: @luxas

* fixes kubeadm upgrade index

* Update Stackdriver Logging documentation (#5495)

* Re-update WordPress and MySQL PV doc to use apps/v1beta2 APIs (#5526)

* Update statefulset concepts doc to use apps/v1beta2 APIs (#5420)

* add document on kubectl's behavior regarding initializers (#5505)

* Update docs/admin/kubeadm.md to cover self-hosting in 1.8. (#5497)

This is a new beta feature in 1.8.

* Update kubectl patch doc to use apps/v1beta2 APIs (#5422)

* [1.8] Update "Run Applications" tasks to apps/v1beta2. (#5525)

* Update replicated stateful application task for 1.8.

* Update single instance stateful app task for 1.8.

* Update stateless app task for 1.8.

* Update kubectl patch task for 1.8.

* fix the link of persistent storage (#5515)

* update the admission-controllers.md index.md what-is-kubernetes.md link

* fix the link of persistent storage

* Add quota support for local ephemeral storage (#5493)

* Add quota support for local ephemeral storage

update the doc to this alpha feature

* Update resource-quotas.md

* Updated Deployments concepts doc (#5491)

* Updated Deployments concepts doc

* Addressed comments

* Addressed more comments

* Modify allocatable storage to ephemeral-storage (#5490)

Update the doc to use ephemeral-storage instead of storage

* Revamped concepts doc for ReplicaSet (#5463)

* Revamped concepts doc for ReplicaSet

* Minor changes to call out specific versions for selector defaulting and
immutability

* Addressed doc review comments

* Remove petset documentations (#5395)

* Update docs to use batch/v1beta1 cronjobs (#5475)

* add federation job doc (#5485)

* add federation job doc

* Update job.md

Edits for clarity and consistency

* Update job.md

Fixed a typo

* update DaemonSet concept for 1.8 release (#5397)

* update DaemonSet concept for 1.8 release

* Update daemonset.md

Fix typo. than -> then

* Update bootstrap tokens doc for 1.8. (#5479)

* Update bootstrap tokens doc for 1.8.

This has some changes I missed when I was updating the main kubeadm documention:
 - Bootstrap tokens are now beta, not alpha (https://github.com/kubernetes/features/issues/130)
 - The apiserver flag to enable the authenticator changedin 1.8 (https://github.com/kubernetes/kubernetes/pull/51198)
 - Added `auth-extra-groups` documentaion (https://github.com/kubernetes/kubernetes/pull/50933)
 - Updated the _Token Management with `kubeadm`_ section to link to the main kubeadm docs, since it was just duplicated information.

* Update bootstrap-tokens.md

* Updated the Cassandra tutorial to use apps/v1beta2 (#5548)

* add docs for AllowPrivilegeEscalation (#5448)

Signed-off-by: Jess Frazelle <acidburn@microsoft.com>

* Add local ephemeral storage alpha feature in managing compute resource (#5522)

* Add local ephemeral storage alpha feature in managing compute resource

Since 1.8, we add the local ephemeral storage alpha feature as one
resource type to manage. Add this feature into the doc.

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Update manage-compute-resources-container.md

* Added documentation for Metrics Server (#5560)

* authorization: improve authorization debugging docs (#5549)

* Document mount propagation (#5544)

* Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524)

This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in https://github.com/kubernetes/kubernetes/pull/49520 and some version bumps.

* Add task doc for alpha dynamic kubelet configuration (#5523)

* Fix input/output of selfsubjectaccess review (#5593)

* Add docs for implementing resize (#5528)

* Add docs for implementing resize

* Update admission-controllers.md

* Added link to PVC section

* minor typo fixes

* Update NetworkPolicy concept guide with egress and CIDR changes (#5529)

* update zookeeper tutorial for 1.8 release

* add doc for hostpath type (#5503)

* Federated Hpa feature doc (#5487)

* Federated Hpa feature doc

* Federated Hpa feature doc review fixes

* Update hpa.md

* Update hpa.md

* update cloud controller manager docs for v1.8

* Update cronjob with defaults information (#5556)

* Kubernetes 1.8 reference docs (#5632)

* Kubernetes 1.8 reference docs

* Kubectl reference docs for 1.8

* Update side bar with 1.8 kubectl and api ref docs links

* remove petset.md

* update on state of HostAlias in 1.8 with hostNetwork Pod support (#5644)

* Fix cron job deletion section (#5655)

* update imported docs (#5656)

* Add documentation for certificate rotation. (#5639)

* Link to using kubeadm page

* fix the command output

fix the command output

* fix typo in api/resources reference: "Worloads"

* Add documentation for certificate rotation.

* Create TOC entry for cloud controller manager. (#5662)

* Updates for new versions of API types

* Followup 5655: fix link to garbage collection (#5666)

* Temporarily redirect resources-reference to api-reference. (#5668)

* Update config for 1.8 release. (#5661)

* Update config for 1.8 release.

* Address reviewer comments.

* Switch references in HPA docs from alpha to beta (#5671)

The HPA docs still referenced the alpha version.  This switches them to
talk about v2beta1, which is the appropriate version for Kubernetes 1.8

* Deprecate openstack heat (#5670)

* Fix typo in pod preset conflict example

Move container port definition to the correct line.

* Highlight openstack-heat provider deprecation

The openstack-heat provider for kube-up is being deprecated and will be
removed in a future release.

* Temporarily fix broken links by redirecting. (#5672)

* Fix broken links. (#5675)

* Fix render of code block (#5674)

* Fix broken links. (#5677)

* Add a small note about auto-bootstrapped CSR ClusterRoles (#5660)

* Update kubeadm install doc for v1.8 (#5676)

* add draft workloads api content for 1.8 (#5650)

* add draft workloads api content for 1.8

* edits per review, add tables, for 1.8 workloads api doc

* fix typo

* Minor fixes to kubeadm 1.8 upgrade guide. (#5678)

- The kubelet upgrade instructions should be done on every host, not
  just worker nodes.
- We should just upgrade all packages, instead of calling out kubelet
  specifically. This will also upgrade kubectl, kubeadm, and
  kubernetes-cni, if installed.
- Draining nodes should also ignore daemonsets, and master errors can be
  ignored.
- Make sure that the new kubeadm download is chmoded correctly.
- Add a step to run `kubeadm version` to verify after downloading.
- Manually approve new kubelet CSRs if rotation is enabled (known issue).

* Release 1.8 (#5680)

* Fix versions for 1.8 API ref docs

* Updates for 1.8 kubectl reference docs

* Kubeadm /docs/admin/kubeadm.md cleanup, editing. (#5681)

* Update docs/admin/kubeadm.md (mostly 1.8 related).

This is Fabrizio's work, which I'm committing along with my edits (in a commit on top of this).

* A few of my own edits to clarify and clean up some Markdown.
This commit is contained in:
Steve Perry
2017-09-28 21:46:51 -07:00
committed by GitHub
parent 88d3ed821d
commit 799eaf4229
1553 changed files with 318572 additions and 3286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions.md
+120 -2
View File
@@ -85,7 +85,7 @@ kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * /5"
cronSpec: "* * * * */5"
image: my-awesome-cron-image
```
@@ -137,7 +137,7 @@ items:
selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
uid: 9423255b-4600-11e7-af6a-28d2447dc82b
spec:
cronSpec: '* * * * /5'
cronSpec: '* * * * */5'
image: my-awesome-cron-image
kind: List
metadata:
@@ -171,6 +171,124 @@ This triggers controllers watching the object to execute any finalizers they han
Each controller then removes its finalizer from the list and issues the delete request again.
This request only deletes the object if the list of finalizers is now empty,
meaning all finalizers are done.
### Validation
Validation of custom objects is possible via [OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject).
Additionally, the following restrictions are applied to the schema:
- The fields `default`, `nullable`, `discriminator`, `readOnly`, `writeOnly`, `xml` and
`deprecated` cannot be set.
- The field `uniqueItems` cannot be set to true.
- The field `additionalProperties` cannot be set to false.
This feature is __alpha__ in v1.8 and may change in backward incompatible ways.
Enable this feature using the `CustomResourceValidation` feature gate on
the [kube-apiserver](/docs/admin/kube-apiserver):
```
--feature-gates=CustomResourceValidation=true
```
The schema is defined in the CustomResourceDefinition. In the following example, the
CustomResourceDefinition applies the following validations on the custom object:
- `spec.cronSpec` must be a string and must be of the form described by the regular expression.
- `spec.replicas` must be an integer and must have a minimum value of 1 and a maximum value of 10.
Save the CustomResourceDefinition to `resourcedefinition.yaml`:
```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: crontabs.stable.example.com
spec:
group: stable.example.com
version: v1
scope: Namespaced
names:
plural: crontabs
singular: crontab
kind: CronTab
shortNames:
- ct
validation:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
properties:
spec:
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
replicas:
type: integer
minimum: 1
maximum: 10
```
And create it:
```shell
kubectl create -f resourcedefinition.yaml
```
A request to create a custom object of kind `CronTab` will be rejected if there are invalid values in its fields.
In the following example, the custom object contains fields with invalid values:
- `spec.cronSpec` does not match the regular expression.
- `spec.replicas` is greater than 10.
If you save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * *"
image: my-awesome-cron-image
replicas: 15
```
and create it:
```shell
kubectl create -f my-crontab.yaml
```
you will get an error:
```console
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10
```
If the fields contain valid values, the object creation request is accepted.
Save the following YAML to `my-crontab.yaml`:
```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "* * * * */5"
image: my-awesome-cron-image
replicas: 5
```
And create it:
```shell
kubectl create -f my-crontab.yaml
crontab "my-new-cron-object" created
```
{% endcapture %}
{% capture whatsnext %}
docs/tasks/access-kubernetes-api/extend-api-third-party-resource.md
+1 -1
View File
@@ -12,7 +12,7 @@ title: Extend the Kubernetes API with ThirdPartyResources
## What is ThirdPartyResource?
**ThirdPartyResource is deprecated as of Kubernetes 1.7 and may be removed in version 1.8 in
**ThirdPartyResource is deprecated as of Kubernetes 1.7 and has been removed in version 1.8 in
accordance with the [deprecation policy](/docs/reference/deprecation-policy) for beta features.**
**To avoid losing data stored in ThirdPartyResources, you must
docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml
+69
View File
@@ -0,0 +1,69 @@
# This is an example of how to setup cloud-controller-manger as a Daemonset in your cluster.
# It assumes that your masters can run pods and has the role node-role.kubernetes.io/master
# Note that this Daemonset will not work straight out of the box for your cloud, this is
# meant to be a guideline.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: system:cloud-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
k8s-app: cloud-controller-manager
name: cloud-controller-manager
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: cloud-controller-manager
template:
metadata:
labels:
k8s-app: cloud-controller-manager
spec:
serviceAccountName: cloud-controller-manager
containers:
- name: cloud-controller-manager
# for in-tree providers we use gcr.io/google_containers/cloud-controller-manager
# this can be replaced with any other image for out-of-tree providers
image: gcr.io/google_containers/cloud-controller-manager:v1.8.0
command:
- /usr/local/bin/cloud-controller-manager
- --cloud-provider=<YOUR_CLOUD_PROVIDER> # Add your own cloud provider here!
- --leader-elect=true
- --use-service-account-credentials
# these flags will vary for every cloud provider
- --allocate-node-cidrs=true
- --configure-cloud-routes=true
- --cluster-cidr=172.17.0.0/16
tolerations:
# this is required so CCM can bootstrap itself
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
# this is to have the daemonset runnable on master nodes
# the taint may vary depending on your cluster setup
- key: node-role.kubernetes.io/master
effect: NoSchedule
# this is to restrict CCM to only run on master nodes
# the node selector may vary depending on your cluster setup
nodeSelector:
node-role.kubernetes.io/master: ""
docs/tasks/administer-cluster/configure-multiple-schedulers.md
+1 -1
View File
@@ -100,7 +100,7 @@ First, update the following fields in your YAML file:
If RBAC is enabled on your cluster, you must update the `system:kube-scheduler` cluster role. Add you scheduler name to the resourceNames of the rule applied for endpoints resources, as in the following example:
```
$ kubectl edit clusterrole system:kube-scheduler
- apiVersion: rbac.authorization.k8s.io/v1beta1
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
docs/tasks/administer-cluster/cpu-management-policies.md
+192
View File
@@ -0,0 +1,192 @@
---
title: Control CPU Management Policies on the Node
---
* TOC
{:toc}
Kubernetes keeps many aspects of how pods execute on nodes abstracted
from the user. This is by design.  However, some workloads require
stronger guarantees in terms of latency and/or performance in order to operate
acceptably.  The kubelet provides methods to enable more complex workload
placement policies while keeping the abstraction free from explicit placement
directives.
## CPU Management Policies
By default, the kubelet uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler)
to enforce pod CPU limits.  When the node runs many CPU-bound pods,
the workload can move to different CPU cores depending on
whether the pod is throttled and which CPU cores are available at
scheduling time.  Many workloads are not sensitive to this migration and thus
work fine without any intervention.
However, in workloads where CPU cache affinity and scheduling latency
significantly affect workload performance, the kubelet allows alternative CPU
management policies to determine some placement preferences on the node.
### Configuration
The CPU Manager is introduced as an alpha feature in Kubernetes v1.8. It
must be explicitly enabled in the kubelet feature gates:
`--feature-gates=CPUManager=true`.
The CPU Manager policy is set with the `--cpu-manager-policy` kubelet
option. There are two supported policies:
* `none`: the default, which represents the existing scheduling behavior.
* `static`: allows pods with certain resource characteristics to be
granted increased CPU affinity and exclusivity on the node.
The CPU manager periodically writes resource updates through the CRI in
order to reconcile in-memory CPU assignments with cgroupfs. The reconcile
frequency is set through a new Kubelet configuration value
`--cpu-manager-reconcile-period`. If not specified, it defaults to the same
duration as `--node-status-update-frequency`.
### None policy
The `none` policy explicitly enables the existing default CPU
affinity scheme, providing no affinity beyond what the OS scheduler does
automatically.  Limits on CPU usage for
[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod/)
are enforced using CFS quota.
### Static policy
The `static` policy allows containers in `Guaranteed` pods with integer CPU
`requests` access to exclusive CPUs on the node. This exclusivity is enforced
using the [cpuset cgroup controller](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt).
**Note:** System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods.
{: .note}
**Note:** The alpha version of this policy does not guarantee static
exclusive allocations across Kubelet restarts.
{: .note}
This policy manages a shared pool of CPUs that initially contains all CPUs in the
node. The amount of exclusively allocatable CPUs is equal to the total
number of CPUs in the node minus any CPU reservations by the kubelet `--kube-reserved` or
`--system-reserved` options. CPUs reserved by these options are taken, in
integer quantity, from the initial shared pool in ascending order by physical
core ID.  This shared pool is the set of CPUs on which any containers in
`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional
CPU `requests` also run on CPUs in the shared pool. Only containers that are
both part of a `Guaranteed` pod and have integer CPU `requests` are assigned
exclusive CPUs.
**Note:** The kubelet requires a CPU reservation greater than zero be made
using either `--kube-reserved` and/or `--system-reserved` when the static
policy is enabled. This is because zero CPU reservation would allow the shared
pool to become empty.
{: .note}
As `Guaranteed` pods whose containers fit the requirements for being statically
assigned are scheduled to the node, CPUs are removed from the shared pool and
placed in the cpuset for the container. CFS quota is not used to bound
the CPU usage of these containers as their usage is bound by the scheduling domain
itself. In others words, the number of CPUs in the container cpuset is equal to the integer
CPU `limit` specified in the pod spec. This static assignment increases CPU
affinity and decreases context switches due to throttling for the CPU-bound
workload.
Consider the containers in the following pod specs:
```yaml
spec:
containers:
- name: nginx
image: nginx
```
This pod runs in the `BestEffort` QoS class because no resource `requests` or
`limits` are specified. It runs in the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
```
This pod runs in the `Burstable` QoS class because resource `requests` do not
equal `limits` and the `cpu` quantity is not specified. It runs in the shared
pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
requests:
memory: "100Mi"
cpu: "1"
```
This pod runs in the `Burstable` QoS class because resource `requests` do not
equal `limits`. It runs in the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
requests:
memory: "200Mi"
cpu: "2"
```
This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
And the container's resource limit for the CPU resource is an integer greater than
or equal to one. The `nginx` container is granted 2 exclusive CPUs.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "1.5"
requests:
memory: "200Mi"
cpu: "1.5"
```
This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
But the container's resource limit for the CPU resource is a fraction. It runs in
the shared pool.
```yaml
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "2"
```
This pod runs in the `Guaranteed` QoS class because only `limits` are specified
and `requests` are set equal to `limits` when not explicitly specified. And the
container's resource limit for the CPU resource is an integer greater than or
equal to one.The `nginx` container is granted 2 exclusive CPUs.
docs/tasks/administer-cluster/developing-cloud-controller-manager.md
+36
View File
@@ -0,0 +1,36 @@
---
approvers:
- luxas
- thockin
- wlan0
title: Developing Cloud Controller Manager
---
**Cloud Controller Manager is an alpha feature in 1.8. In upcoming releases it will
be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers
can develop their features independantly from the core Kubernetes release cycles.**
* TOC
{:toc}
## Background
Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisifed.
To dive a little deeper into implementation details, all cloud controller managers will import packages from Kubernetes core, the only difference being each project will register their own cloud providers by calling [cloudprovider.RegisterCloudProvier](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52) where a global variable of available cloud providers is updated.
## Developing
### Out of Tree
To build an out-of-tree cloud-controller-manager for your cloud, follow these steps:
1. Create a go package with an implementation that satisfies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
2. Use [main.go in cloud-controller-manager](https://github.com/kubernetes/kubernetes/blob/master/cmd/cloud-controller-manager/controller-manager.go) from Kubernestes core as a template for your main.go. As mentioned above, the only difference should be the cloud package that will be imported.
3. Import your cloud package in `main.go`, ensure your package has an `init` block to run [cloudprovider.RegisterCloudProvider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52).
Using existing out-of-tree cloud providers as an example may be helpful. You can find the list [here](/docs/tasks/administer-cluster/running-cloud-controller.md#examples).
### In Tree
For in-tree cloud providers, you can run the in-tree cloud controller manager as a [Daemonset](/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml) in your cluster. See the [running cloud controller manager docs](/docs/tasks/administer-cluster/running-cloud-controller.md) for more details.
docs/tasks/administer-cluster/kubeadm-upgrade-1-8.md
+281
View File
@@ -0,0 +1,281 @@
---
approvers:
- pipejakob
- luxas
- roberthbailey
- jbeda
title: Upgrading kubeadm clusters from 1.7 to 1.8
---
{% capture overview %}
This guide is for upgrading `kubeadm` clusters from version 1.7.x to 1.8.x, as well as 1.7.x to 1.7.y and 1.8.x to 1.8.y where `y > x`.
See also [upgrading kubeadm clusters from 1.6 to 1.7](/docs/tasks/administer-cluster/kubeadm-upgrade-1-7/) if you're on a 1.6 cluster currently.
{% endcapture %}
{% capture prerequisites %}
Before proceeding:
- You need to have a functional `kubeadm` Kubernetes cluster running version 1.7.0 or higher in order to use the process described here.
- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v180-beta1) carefully.
- As `kubeadm upgrade` does not upgrade etcd make sure to back it up. You can, for example, use `etcdctl backup` to take care of this.
- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up what's important to you. For example, any app-level state, such as a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
Also, note that only one minor version upgrade is supported. That is, you can only upgrade from, say 1.7 to 1.8, not from 1.7 to 1.9.
{% endcapture %}
{% capture steps %}
## Upgrading your control plane
You have to carry out the following steps by executing these commands on your master node:
1. Install the most recent version of `kubeadm` using `curl` like so:
```shell
$ export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
$ export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
$ curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
```
Verify that this download of kubeadm works, and has the expected version:
```shell
$ kubeadm version
```
2. If this the first time you use `kubeadm upgrade`, in order to preserve the configuration for future upgrades, do:
Note that for below you will need to recall what CLI args you passed to `kubeadm init` the first time.
If you used flags, do:
```shell
$ kubeadm config upload from-flags [flags]
```
Where `flags` can be empty.
If you used a config file, do:
```shell
$ kubeadm config upload from-file --config [config]
```
Where the `config` is mandatory.
3. On the master node, run the following:
```shell
$ kubeadm upgrade plan
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to:
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0
[upgrade/versions] Latest stable version: v1.8.0
[upgrade/versions] Latest version in the v1.7 series: v1.7.6
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.7.1 v1.7.6
Upgrade to the latest version in the v1.7 series:
COMPONENT CURRENT AVAILABLE
API Server v1.7.1 v1.7.6
Controller Manager v1.7.1 v1.7.6
Scheduler v1.7.1 v1.7.6
Kube Proxy v1.7.1 v1.7.6
Kube DNS 1.14.4 1.14.4
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.7.6
_____________________________________________________________________
Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.7.1 v1.8.0
Upgrade to the latest experimental version:
COMPONENT CURRENT AVAILABLE
API Server v1.7.1 v1.8.0
Controller Manager v1.7.1 v1.8.0
Scheduler v1.7.1 v1.8.0
Kube Proxy v1.7.1 v1.8.0
Kube DNS 1.14.4 1.14.4
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.8.0
Note: Before you do can perform this upgrade, you have to update kubeadm to v1.8.0
_____________________________________________________________________
```
The `kubeadm upgrade plan` checks that your cluster is in an upgradeable state and fetches the versions available to upgrade to in an user-friendly way.
4. Pick a version to upgrade to and run, for example, `kubeadm upgrade apply` as follows:
```shell
$ kubeadm upgrade apply v1.8.0
[preflight] Running pre-flight checks
[upgrade] Making sure the cluster is healthy:
[upgrade/health] Checking API Server health: Healthy
[upgrade/health] Checking Node health: All Nodes are healthy
[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/version] You have chosen to upgrade to version "v1.8.0"
[upgrade/versions] Cluster version: v1.7.1
[upgrade/versions] kubeadm version: v1.8.0
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.8.0"...
[upgrade/staticpods] Writing upgraded Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-scheduler.yaml"
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.8.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.
```
`kubeadm upgrade apply` does the following:
- It checks that your cluster is in an upgradeable state, that is:
- The API Server is reachable,
- All nodes are in the `Ready` state, and
- The control plane is healthy
- It enforces the version skew policies.
- It makes sure the control plane images are available or available to pull to the machine.
- It upgrades the control plane components or rollbacks if any of them fails to come up.
- It applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
5. Manually upgrade your Software Defined Network (SDN).
Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow now.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.
## Upgrading your master and node packages
For each host (referred to as `$HOST` below) in your cluster, upgrade `kubelet` by executing the following commands:
1. Prepare the host for maintenance, marking it unschedulable and evicting the workload:
```shell
$ kubectl drain $HOST --ignore-daemonsets
```
When running this command against the master host, this error is expected and can be safely ignored (since there are static pods running on the master):
```shell
node "master" already cordoned
error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): etcd-kubeadm, kube-apiserver-kubeadm, kube-controller-manager-kubeadm, kube-scheduler-kubeadm
```
2. Upgrade the Kubernetes package versions on the `$HOST` node by using a Linux distribution-specific package manager:
If the host is running a Debian-based distro such as Ubuntu, run:
```shell
$ apt-get update
$ apt-get upgrade
```
If the host is running CentOS or the like, run:
```shell
$ yum update
```
Now the new version of the `kubelet` should be running on the host. Verify this using the following command on `$HOST`:
```shell
$ systemctl status kubelet
```
3. Since certificate rotation is enabled by default, you may need to manually approve the new kubelet's CertificateSigningRequest before it can rejoin the cluster:
```shell
$ kubectl get csr | grep -v Approved
NAME AGE REQUESTOR CONDITION
node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM 2h system:bootstrap:033abb Pending
```
If you see any CSRs listed that aren't already approved, you can manually approve them using kubectl:
```shell
$ kubectl certificate approve node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM
certificatesigningrequest "node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM" approved
```
4. Bring the host back online by marking it schedulable:
```shell
$ kubectl uncordon $HOST
```
5. After upgrading `kubelet` on each host in your cluster, verify that all nodes are available again by executing the following (from anywhere, for example, from outside the cluster):
```shell
$ kubectl get nodes
```
If the `STATUS` column of the above command shows `Ready` for all of your hosts, you are done.
## Recovering from a bad state
If `kubeadm upgrade` somehow fails and fails to roll back, due to an unexpected shutdown during execution for instance,
you may run `kubeadm upgrade` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring.
You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/opaque-integer-resource-node.md
+1 -1
View File
@@ -9,7 +9,7 @@ This page shows how to specify opaque integer resources for a Node.
Opaque integer resources allow cluster administrators to advertise node-level
resources that would otherwise be unknown to Kubernetes.
{% include feature-state-alpha.md %}
{% include feature-state-deprecated.md %}
{% endcapture %}
docs/tasks/administer-cluster/persistent-volume-label-initializer-config.yaml
+13
View File
@@ -0,0 +1,13 @@
kind: InitializerConfiguration
apiVersion: admissionregistration.k8s.io/v1alpha1
metadata:
name: pvlabel.kubernetes.io
initializers:
- name: pvlabel.kubernetes.io
rules:
- apiGroups:
- ""
apiVersions:
- "*"
resources:
- persistentvolumes
docs/tasks/administer-cluster/reconfigure-kubelet.md
+470
View File
@@ -0,0 +1,470 @@
---
approvers:
- mtaufen
- dawnchen
title: Reconfigure a Node's Kubelet in a Live Cluster
---
{% capture overview %}
{% include feature-state-alpha.md %}
As of Kubernetes 1.8, the new
[Dynamic Kubelet Configuration](https://github.com/kubernetes/features/issues/281)
feature is available in alpha. This allows you to change the configuration of
Kubelets in a live Kubernetes cluster via first-class Kubernetes concepts.
Specifically, this feature allows you to configure individual Nodes' Kubelets
via ConfigMaps.
**Warning:** All Kubelet configuration parameters may be changed dynamically,
but not all parameters are safe to change dynamically. This feature is intended
for system experts who have a strong understanding of how configuration changes
will affect behavior. No documentation currently exists which plainly lists
"safe to change" fields, but we plan to add it before this feature graduates
from alpha.
{% endcapture %}
{% capture prerequisites %}
- A live Kubernetes cluster with both Master and Node at v1.8 or higher must be
running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
`--dynamic-config-dir` flag set to a writeable directory on the Node.
This flag must be set to enable Dynamic Kubelet Configuration.
- The kubectl command-line tool must be also be v1.8 or higher, and must be
configured to communicate with the cluster.
{% endcapture %}
{% capture steps %}
## Reconfiguring the Kubelet on a Live Node in your Cluster
### Basic Workflow Overview
The basic workflow for configuring a Kubelet in a live cluster is as follows:
1. Write a YAML or JSON configuration file containing the
Kubelet's configuration.
2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
3. Update the Kubelet's correspoinding Node object to use this ConfigMap.
Each Kubelet watches a configuration reference on its respective Node object.
When this reference changes, the Kubelet downloads the new configuration and
exits. For the feature to work correctly, you must be running a process manager
(like systemd) which will restart the Kubelet when it exits. When the Kubelet is
restarted, it will begin using the new configuration.
The new configuration completely overrides the old configuration; unspecified
fields in the new configuration will receive their canonical default values.
Some CLI flags do not have an associated configuration field, and will not be
affected by the new configuration. These fields are defined by the KubeletFlags
structure, [here](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go).
The status of the Node's Kubelet configuration is reported via the `ConfigOK`
condition in the Node status. Once you have updated a Node to use the new
ConfigMap, you can observe this condition to confirm that the Node is using the
intended configuration. A table describing the possible conditions can be found
at the end of this article.
This document describes editing Nodes using `kubectl edit`.
There are other ways to modify a Node's spec, including `kubectl patch`, for
example, which facilitate scripted workflows.
This document only describes a single Node consuming each ConfigMap. Keep in
mind that it is also valid for multiple Nodes to consume the same ConfigMap.
### Node Authorizer Workarounds
The Node Authorizer does not yet pay attention to which ConfigMaps are assigned
to which Nodes. If you currently use the Node authorizer, your Kubelets will not
be automatically granted permission to download their respective ConfigMaps.
The temporary workaround used in this document is to manually create the RBAC
Roles and RoleBindings for each ConfigMap. The Node Authorizer will be extended
before the Dynamic Kubelet Configuration feature graduates from alpha, so doing
this in production should never be necessary.
### Generating a file that contains the current configuration
The Dynamic Kubelet Configuration feature allows you to provide an override for
the entire configuration object, rather than a per-field overlay. This is a
simpler model that makes it easier to trace the source of configuration values
and debug issues. The compromise, however, is that you must start with knowledge
of the existing configuration to ensure that you only change the fields you
intend to change.
In the future, the Kubelet will be bootstrapped from a file on disk, and you
will simply edit a copy of this file (which, as a best practice, should live in
version control) while creating the first Kubelet ConfigMap. Today, however, the
Kubelet is still bootstrapped with command-line flags. Fortunately, there is a
dirty trick you can use to generate a config file containing a Node's current
configuration. The trick involves hitting the Kubelet server's `configz`
endpoint via the kubectl proxy. This endpoint, in its current implementation, is
intended to be used only as a debugging aid, which is part of why this is a
dirty trick. There is ongoing work to improve the endpoint, and in the future
this will be a less "dirty" operation. This trick also requires the `jq` command
to be installed on your machine, for unpacking and editing the JSON response
from the endpoint.
Do the following to generate the file:
1. Pick a Node to reconfigure. We will refer to this Node's name as NODE_NAME.
2. Start the kubectl proxy in the background with `kubectl proxy --port=8001 &`
3. Run the following command to download and unpack the configuration from the
configz endpoint:
```
$ export NODE_NAME=the-name-of-the-node-you-are-reconfiguring
$ curl -sSL http://localhost:8001/api/v1/proxy/nodes/${NODE_NAME}/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubeletconfig/v1alpha1"' > kubelet_configz_${NODE_NAME}
```
Note that we have to manually add the `kind` and `apiVersion` to the downloaded
object, as these are not reported by the configz endpoint. This is one of the
limitations of the endpoint that is planned to be fixed in the future.
### Edit the configuration file
Using your editor of choice, change one of the parameters in the
`kubelet_configz_${NODE_NAME}` file from the previous step. A QPS parameter,
`eventRecordQPS` for example, is a good candidate.
### Push the configuration file to the control plane
Push the edited configuration file to the control plane with the
following command:
```
$ kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
```
You should see a response similar to:
```
apiVersion: v1
data:
kubelet: |
{...}
kind: ConfigMap
metadata:
creationTimestamp: 2017-09-14T20:23:33Z
name: my-node-config-gkt4c2m4b2
namespace: kube-system
resourceVersion: "119980"
selfLink: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2
uid: 946d785e-998a-11e7-a8dd-42010a800006
```
Note that the configuration data must appear under the ConfigMap's
`kubelet` key.
We create the ConfigMap in the `kube-system` namespace, which is appropriate
because this ConfigMap configures a Kubernetes system component - the Kubelet.
The `--append-hash` option appends a short checksum of the ConfigMap contents
to the name. This is convenient for an edit->push workflow, as it will
automatically, yet deterministically, generate new names for new ConfigMaps.
We use the `-o yaml` output format so that the name, namespace, and uid are all
reported following creation. We will need these in the next step. We will refer
to the name as CONFIG_MAP_NAME and the uid as CONFIG_MAP_UID.
### Authorize your Node to read the new ConfigMap
Now that you've created a new ConfigMap, you need to authorize your node to
read it. First, create a Role for your new ConfigMap with the
following commands:
```
$ export CONFIG_MAP_NAME=name-from-previous-output
$ kubectl -n kube-system create role ${CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${CONFIG_MAP_NAME}
```
Next, create a RoleBinding to associate your Node with the new Role:
```
$ kubectl -n kube-system create rolebinding ${CONFIG_MAP_NAME}-reader --role=${CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Set the Node to use the new configuration
Edit the Node's reference to point to the new ConfigMap with the
following command:
```
kubectl edit node ${NODE_NAME}
```
Once in your editor, add the following YAML under `spec`:
```
configSource:
configMapRef:
name: CONFIG_MAP_NAME
namespace: kube-system
uid: CONFIG_MAP_UID
```
Be sure to specify all three of `name`, `namespace`, and `uid`.
### Observe that the Node begins using the new configuration
Retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and look for the
`ConfigOK` condition in `status.conditions`. You should see the message
`Using current (UID: CONFIG_MAP_UID)` when the Kubelet starts using the new
configuration.
For convenience, you can use the following command (using `jq`) to filter down
to the `ConfigOK` condition:
```
$ kubectl get no ${NODE_NAME} -o json | jq '.status.conditions|map(select(.type=="ConfigOK"))'
[
{
"lastHeartbeatTime": "2017-09-20T18:08:29Z",
"lastTransitionTime": "2017-09-20T18:08:17Z",
"message": "using current (UID: \"2ebc8d1a-9e2a-11e7-a8dd-42010a800006\")",
"reason": "passing all checks",
"status": "True",
"type": "ConfigOK"
}
]
```
If something goes wrong, you may see one of several different error conditions,
detailed in the Table of ConfigOK Conditions, below. When this happens, you
should check the Kubelet's log for more details.
### Edit the configuration file again
To change the configuration again, we simply repeat the above workflow.
Try editing the `kubelet` file, changing the previously changed parameter to a
new value.
### Push the newly edited configuration to the control plane
Push the new configuration to the control plane in a new ConfigMap with the
following command:
```
$ kubectl create configmap my-node-config --namespace=kube-system --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
```
This new ConfigMap will get a new name, as we have changed the contents.
We will refer to the new name as NEW_CONFIG_MAP_NAME and the new uid
as NEW_CONFIG_MAP_UID.
### Authorize your Node to read the new ConfigMap
Now that you've created a new ConfigMap, you need to authorize your node to
read it. First, create a Role for your new ConfigMap with the
following commands:
```
$ export NEW_CONFIG_MAP_NAME=name-from-previous-output
$ kubectl -n kube-system create role ${NEW_CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${NEW_CONFIG_MAP_NAME}
```
Next, create a RoleBinding to associate your Node with the new Role:
```
$ kubectl -n kube-system create rolebinding ${NEW_CONFIG_MAP_NAME}-reader --role=${NEW_CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Configure the Node to use the new configuration
Once more, edit the Node's `spec.configSource` with
`kubectl edit node ${NODE_NAME}`. Your new `spec.configSource` should look like
the following, with `name` and `uid` substituted as necessary:
```
configSource:
configMapRef:
name: NEW_CONFIG_MAP_NAME
namespace: kube-system
uid: NEW_CONFIG_MAP_UID
```
### Observe that the Kubelet is using the new configuration
Once more, retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and
look for the `ConfigOK` condition in `status.conditions`. You should the message
`Using current (UID: NEW_CONFIG_MAP_UID)` when the Kubelet starts using the
new configuration.
### Deauthorize your Node fom reading the old ConfigMap
Once you know your Node is using the new configuration and are confident that
the new configuration has not caused any problems, it is a good idea to
deauthorize the node from reading the old ConfigMap. Run the following
commands to remove the RoleBinding and Role:
```
$ kubectl -n kube-system delete rolebinding ${CONFIG_MAP_NAME}-reader
$ kubectl -n kube-system delete role ${CONFIG_MAP_NAME}-reader
```
Note that this does not necessarily prevent the Node from reverting to the old
configuration, as it may locally cache the old ConfigMap for an indefinite
period of time.
You may optionally also choose to remove the old ConfigMap:
```
$ kubectl -n kube-system delete configmap ${CONFIG_MAP_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
### Reset the Node to use its local default configuration
Finally, if you wish to reset the Node to use the configuration it was
provisioned with, simply edit the Node with `kubectl edit node ${NODE_NAME}` and
remove the `spec.configSource` subfield.
### Observe that the Node is using its local default configuration
After removing this subfield, you should eventually observe that the ConfigOK
condition's message reverts to either `using current (default)` or
`using current (init)`, depending on how the Node was provisioned.
### Deauthorize your Node fom reading the old ConfigMap
Once you know your Node is using the default configuraiton again, it is a good
idea to deauthorize the node from reading the old ConfigMap. Run the following
commands to remove the RoleBinding and Role:
```
$ kubectl -n kube-system delete rolebinding ${NEW_CONFIG_MAP_NAME}-reader
$ kubectl -n kube-system delete role ${NEW_CONFIG_MAP_NAME}-reader
```
Note that this does not necessarily prevent the Node from reverting to the old
ConfigMap, as it may locally cache the old ConfigMap for an indefinite
period of time.
You may optionally also choose to remove the old ConfigMap:
```
$ kubectl -n kube-system delete configmap ${NEW_CONFIG_MAP_NAME}
```
Once the Node Authorizer is updated to do this automatically, you will
be able to skip this step.
{% endcapture %}
{% capture discussion %}
## Kubectl Patch Example
As mentioned above, there are many ways to change a Node's configSource.
Here is an example command that uses `kubectl patch`:
```
kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMapRef\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"uid\":\"${CONFIG_MAP_UID}\"}}}}"
```
## Understanding ConfigOK Conditions
The following table describes several of the `ConfigOK` Node conditions you
might encounter in a cluster that has Dynamic Kubelet Config enabled. If you
observe a condition with `status=False`, you should check the Kubelet log for
more error details by searching for the message or reason text.
<table>
<table align="left">
<tr>
<th>Possible Messages</th>
<th>Possible Reasons</th>
<th>Status</th>
</tr>
<tr>
<td><p>using current (default)</p></td>
<td><p>current is set to the local default, and no init config was provided</p></td>
<td><p>True</p></td>
</tr>
<tr>
<td><p>using current (init)</p></td>
<td><p>current is set to the local default, and an init config was provided</p></td>
<td><p>True</p></td>
</tr>
<tr>
<td><p>using current (UID: CURRENT_CONFIG_MAP_UID)</p></td>
<td><p>passing all checks</p></td>
<td><p>True</p></td>
</tr>
<tr>
<td><p>using last-known-good (default)</p></td>
<td>
<ul>
<li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
<tr>
<td><p>using last-known-good (init)</p></td>
<td>
<ul>
<li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
<tr>
<td><p>using last-known-good (UID: LAST_KNOWN_GOOD_CONFIG_MAP_UID)</p></td>
<td>
<ul>
<li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
<li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
<tr>
<td>
<p>
The reasons in the next column could potentially appear for any of
the above messages.
</p>
<p>
This condition indicates that the Kubelet is having trouble
reconciling `spec.configSource`, and thus no change to the in-use
configuration has occurred.
</p>
<p>
The "failed to sync" reasons are specific to the failure that
occurred, and the next column does not necessarily contain all
possible failure reasons.
</p>
</td>
<td>
<p>failed to sync, reason:</p>
<ul>
<li>failed to read Node from informer object cache</li>
<li>failed to reset to local (default or init) config</li>
<li>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</li>
<li>invalid ObjectReference, all of UID, Name, and Namespace must be specified</li>
<li>invalid ObjectReference, UID SOME_UID does not match UID of downloaded ConfigMap SOME_OTHER_UID</li>
<li>failed to determine whether object with UID SOME_UID was already checkpointed</li>
<li>failed to download ConfigMap with name SOME_NAME from namespace SOME_NAMESPACE</li>
<li>failed to save config checkpoint for object with UID SOME_UID</li>
<li>failed to set current config checkpoint to default</li>
<li>failed to set current config checkpoint to object with UID SOME_UID</li>
</ul>
</td>
<td><p>False</p></td>
</tr>
</table>
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-cluster/reserve-compute-resources.md
+6 -6
View File
@@ -42,7 +42,7 @@ on each node.
`Allocatable` on a Kubernetes node is defined as the amount of compute resources
that are available for pods. The scheduler does not over-subscribe
`Allocatable`. `CPU`, `memory` and `storage` are supported as of now.
`Allocatable`. `CPU`, `memory` and `ephemeral-storage` are supported as of now.
Node Allocatable is exposed as part of `v1.Node` object in the API and as part
of `kubectl describe node` in the CLI.
@@ -77,7 +77,7 @@ be configured to use the `systemd` cgroup driver.
### Kube Reserved
- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][storage=1Gi]`
- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi]`
- **Kubelet Flag**: `--kube-reserved-cgroup=`
`kube-reserved` is meant to capture resource reservation for kubernetes system
@@ -107,7 +107,7 @@ exist. Kubelet will fail if an invalid cgroup is specified.
### System Reserved
- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][storage=1Gi]`
- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][ephemeral-storage=1Gi]`
- **Kubelet Flag**: `--system-reserved-cgroup=`
@@ -135,7 +135,7 @@ Memory pressure at the node level leads to System OOMs which affects the entire
node and all pods running on it. Nodes can go offline temporarily until memory
has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
provides [`Out of Resource`](./out-of-resource.md) management. Evictions are
supported for `memory` and `storage` only. By reserving some memory via
supported for `memory` and `ephemeral-storage` only. By reserving some memory via
`--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
availability on the node drops below the reserved value. Hypothetically, if
system daemons did not exist on a node, pods cannot use more than `capacity -
@@ -191,8 +191,8 @@ So expect a drop in `Allocatable` capacity in future releases.
Here is an example to illustrate Node Allocatable computation:
* Node has `32Gi` of `memory`, `16 CPUs` and `100Gi` of `Storage`
* `--kube-reserved` is set to `cpu=1,memory=2Gi,storage=1Gi`
* `--system-reserved` is set to `cpu=500m,memory=1Gi,storage=1Gi`
* `--kube-reserved` is set to `cpu=1,memory=2Gi,ephemeral-storage=1Gi`
* `--system-reserved` is set to `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`
* `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`
Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
docs/tasks/administer-cluster/running-cloud-controller.md
+75 -16
View File
@@ -1,34 +1,93 @@
---
approvers:
- luxas
- thockin
title: Build and Run cloud-controller-manager
- wlan0
title: Kubernetes Cloud Controller Manager
---
Kubernetes version 1.6 contains a new binary called as `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops in Kubernetes. These cloud-specific control loops were originally in the kube-controller-manager. However, cloud providers move at a different pace and schedule compared to the Kubernetes project, and abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud provider vendors to evolve independently from the core Kubernetes code.
The `cloud-controller-manager` can be linked to any cloud provider that satisifies the [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
In future Kubernetes releases, cloud vendors should link code that satisfies the above interface to the `cloud-controller-manager` project and compile `cloud-controller-manager` for their own clouds. Cloud providers would also be responsible for maintaining and evolving their code.
**Cloud Controller Manager is an alpha feature in 1.8. In upcoming releases it will be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers can develop their features independantly from the core Kubernetes release cycles.**
* TOC
{:toc}
### Building cloud-controller-manager for your cloud
## Cloud Controller Manager
To build cloud-controller-manager for your cloud, follow these steps:
Kubernetes v1.6 contains a new binary called `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the `kube-controller-manager`. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud vendors to evolve independently from the core Kubernetes code.
1. Write a cloudprovider that satisfies the [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
2. Link the cloudprovider to cloud-controller-manager.
The `cloud-controller-manager` can be linked to any cloud provider that satisifies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go). For backwards compatibility, the [cloud-controller-manager](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager) provided in the core Kubernetes project uses the same cloud libraries as `kube-controller-manager`. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core. In future Kubernetes releases, all cloud controller managers will be developed outside of the core Kubernetes project managed by sig leads or cloud vendors.
The methods in [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go) are self-explanatory. All of the
[existing providers](https://git.k8s.io/kubernetes/pkg/cloudprovider/providers) satisfy this interface. If your cloud is already a part
of the existing providers, you do not need to write a new provider; you can proceed directly with linking your cloud provider to the `cloud-controller-manager`.
## Administration
Once your code is ready, you must import that code into `cloud-controller-manager`. See the [rancher cloud sample](https://github.com/rancher/rancher-cloud-controller-manager) for a reference example. The import step in the sample is the only step required to link your cloud provider to the `cloud-controller-manager`.
### Requirements
Every cloud has their own set of requirements for running their own cloud provider integration, it should not be too different from the requirements when running `kube-controller-manager`. As a general rule of thumb you'll need:
* cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to their APIs
* kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the kubernetes apiserver
* high availabilty: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).
### Running cloud-controller-manager
To run `cloud-controller-manager`, add it to your existing Kubernetes cluster as a Master component. All other master components except `kube-controller-manager` can be run without any changes.
Successfully running cloud-controller-manager requires some changes to your cluster configuration.
The `kube-controller-manager` should not run any cloud-specific controllers, since the `cloud-controller-manager` takes over this responsibility. To prevent the `kube-controller-manager` from running cloud-specific controllers, you must set the `--cloud-provider` flag in `kube-controller-manager` to `external`.
* `kube-apiserver` and `kube-controller-manager` MUST NOT specify the `--cloud-provider` flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
* `kubelet` must run with `--cloud-provider=external`. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller since the cloud controller manager takes over labeling persistent volumes. To prevent the PersistentVolumeLabel admission plugin from running, make sure the `kube-apiserver` has a `--admission-control` flag with a value that does not include `PersistentVolumeLabel`.
* For the `cloud-controller-manager` to label persistent volumes, initializers will need to be enabled and an InitializerConifguration needs to be added to the system. Follow [these instructions](/docs/admin/extensible-admission-controllers.md#enable-initializers-alpha-feature) to enable initializers. Use the following YAML to create the InitializerConfiguration:
The `kube-apiserver` should not run the Persistent Volume Label admission controller either since the `cloud-controller-manager` takes over labeling persistent volumes. To prevent the Persistent Volume Label admission plugin from running, make sure the `kube-apiserver` has a `--admission-control` flag with a value that does not include `PersistentVolumeLabel`.
{% include code.html language="yaml" file="persistent-volume-label-initializer-config.yaml" ghlink="/docs/tasks/administer-cluster/persistent-volume-label-initializer-config.yaml" %}
Keep in mind that setting up your cluster to use cloud controller manager will change your cluster behaviour in a few ways:
* kubelets specifying `--cloud-provider=external` will add a taint `node.cloudprovider.kubernetes.io/uninitialized` with an effect `NoSchedule` during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unscheduable. The taint is important since the scheduler may require cloud specific information about nodes such as it's region or type (high cpu, gpu, high memory, spot instance, etc).
* cloud information about nodes in the cluster will no longer be retrieved using local metadata, but instead all API calls to retreive node information will go through cloud controller manager. This may mean you can restrict access to your cloud API on the kubelets for better security. For larger clusters you may want to consider if cloud controller manager will hit rate limits since it is now responsible for almost all API calls to your cloud from within the cluster.
As of v1.8, cloud controller manager can implement:
* node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
* service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
* route controller - responsible for setting up network routes on your cloud
* [PersistentVolumeLabel Admission Controller](/docs/admin/admission-controllers#persistentvolumelabel) - responsible for labeling persistent volumes on your cloud - ensure that the persistent volume label admission plugin is not enabled on your kube-apiserver.
* any other features you would like to implement if you are running an out-of-tree provider.
## Examples
If you are using a cloud that is currently supported in Kubernetes core and would like to adopt cloud controller manager, see the [cloud controller manager in kubernetes core](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager).
For cloud controller managers not in Kubernetes core, you can find the respective projects in repos maintained by cloud vendors or sig leads.
* [DigitalOcean](https://github.com/digitalocean/digitalocean-cloud-controller-manager)
* [keepalived](https://github.com/munnerz/keepalived-cloud-provider)
* [Rancher](https://github.com/rancher/rancher-cloud-controller-manager)
For providers already in Kubernetes core, you can run the in-tree cloud controller manager as a Daemonset in your cluster, use the following as a guideline:
{% include code.html language="yaml" file="cloud-controller-manager-daemonset-example.yaml" ghlink="/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml" %}
## Limitations
Running cloud controller manager comes with a few possible limitations. Although these limitations are being addressed in upcoming releases, it's important that you are aware of these limitations for production workloads.
### Support for Volumes
Cloud controller manager does not implement any of the volume controllers found in `kube-controller-manager` as the volume integrations also require coordination with kubelets. As we evolve CSI (container storage interface) and add stronger support for flex volume plugins, necessary support will be added to cloud controller manager so that clouds can fully integrate with volumes. Learn more about out-of-tree CSI volume plugins [here](https://github.com/kubernetes/features/issues/178).
### Scalability
In the previous architecture for cloud providers, we relied on kubelets using a local metadata service to retreive node information about itself. With this new architecture, we now fully rely on the cloud controller managers to retrieve information for all nodes. For very larger clusters, you should consider possible bottle necks such as resource requirements and API rate limiting.
### Chicken and Egg
The goal of the cloud controller manager project is to decouple development of cloud features from the core Kubernetes project. Unforunately, many aspects of the Kubernetes project has assumptions that cloud provider features are tightly integrated into the project. As a result, adopting this new architecture can create several situations where a request is being made for information from a cloud provider, but the cloud controller manager may not be able to return that information without the original request being complete.
A good example of this is the TLS bootstrapping feature in the Kubelet. Currently, TLS bootstraping assumes that the Kubelet has the ability to ask the cloud provider (or a local metadata service) for all its address types (private, public, etc) but cloud controller manager cannot set a node's address types without being initialzed in the first place which requires that the kubelet has TLS certificates to communicate with the apiserver.
As this initiative evolves, changes will be made to address these issues in upcoming releases.
## Developing your own Cloud Controller Manager
To build and develop your own cloud controller manager, read the [Developing Cloud Controller Manager](/docs/tasks/administer-cluster/developing-cloud-controller-manager.md) doc.
docs/tasks/administer-federation/hpa.md
+177
View File
@@ -0,0 +1,177 @@
---
title: Federated Horizontal Pod Autoscalers (HPA)
---
{% capture overview %}
{% include feature-state-alpha.md %}
This guide explains how to use federated horizontal pod autoscalers (HPAs) in the federation control plane.
HPAs in the federation control plane are similar to the traditional [Kubernetes
HPAs](/docs/tasks/run-application/horizontal-pod-autoscale/), and provide the same functionality.
Creating an HPA targeting a federated object in the federation control plane ensures that the
desired number of replicas of the target object are scaled across the registered clusters,
instead of a single cluster. Also, the control plane keeps monitoring the status of each
individual HPA in the federated clusters and ensures the workload replicas move where they are
needed most by manipulating the min and max limits of the HPA objects in the federated clusters.
{% endcapture %}
{% capture prerequisites %}
* {% include federated-task-tutorial-prereqs.md %}
* You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [HPAs](/docs/tasks/run-application/horizontal-pod-autoscale/) in particular.
The federated HPA is an alpha feature. The API is not enabled by default on the
federated API server. To use this feature, the user or the admin deploying the federation control
plane needs to run the federated API server with option `--runtime-config=api/all=true` to
enable all APIs, including alpha APIs. Additionally, the federated HPA only works
when used with CPU utilization metrics.
{% endcapture %}
{% capture steps %}
## Creating a federated HPA
The API for federated HPAs is 100% compatible with the
API for traditional Kubernetes HPA. You can create an HPA by sending
a request to the federation API server.
You can do that with [kubectl](/docs/user-guide/kubectl/) by running:
```shell
cat <<EOF | kubectl --context=federation-cluster create -f -
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
EOF
```
The `--context=federation-cluster` flag tells `kubectl` to submit the
request to the federation API server instead of sending it to a Kubernetes
cluster.
Once a federated HPA is created, the federation control plane partitions and
creates the HPA in all underlying Kubernetes clusters. As of Kubernetes V1.7,
[cluster selectors](/docs/tasks/administer-federation/cluster/#clusterselector-annotation)
can also be used to restrict any federated object, including the HPAs in a subset
of clusters.
You can verify the creation by checking each of the underlying clusters. For example, with a context named `gce-asia-east1a`
configured in your client for your cluster in that zone:
```shell
kubectl --context=gce-asia-east1a get HPA php-apache
```
The HPA in the underlying clusters will match the federation HPA
except in the number of min and max replicas. The federation control plane ensures that the sum of max replicas in each cluster matches the specified
max replicas on the federated HPA object, and the sum of minimum replicas will be greater
than or equal to the minimum specified on the federated HPA object.
**Note:** A particular cluster cannot have a minimum replica sum of 0.
{: .note}
### Spreading HPA min and max replicas in underlying clusters
By default, first max replicas are spread equally in all the underlying clusters, then min replicas are distributed to those clusters that received their maximum value. This means
that each cluster will get an HPA if the specified max replicas are greater than
the total clusters participating in this federation, and some clusters will be
skipped if specified max replicas are less than the total clusters participating
in the federation.
For example: if you have 3 registered clusters and you create a federated HPA with
`spec.maxReplicas = 9`, and `spec.minReplicas = 2`, then each HPA in the 3 clusters
will get `spec.maxReplicas=3` and `spec.minReplicas = 1`.
Currently the default distribution is only available on the federated HPA, but in the
future, users preferences could also be specified to control and/or restrict this
distribution.
## Updating a federated ReplicaSet
You can update a federated HPA as you would update a Kubernetes
HPA; however, for a federated HPA, you must send the request to
the federation API server instead of sending it to a specific Kubernetes cluster.
The Federation control plane ensures that whenever the federated HPA is
updated, it updates the corresponding HPA in all underlying clusters to
match it.
If your update includes a change in the number of replicas, the federation
control plane will change the number of replicas in underlying clusters to
ensure that the sum of the max and min replicas remains matched as specified
in the previous section.
## Deleting a federated HPA
You can delete a federated HPA as you would delete a Kubernetes
HPA; however, for a federated HPA, you must send the request to
the federation API server instead of sending it to a specific Kubernetes cluster.
It should also be noted that for the federated resource to be deleted from
all underlying clusters, [cascading deletion](/docs/concepts/cluster-administration/federation/#cascading-deletion)
should be used.
For example, you can do that using `kubectl` by running:
```shell
kubectl --context=federation-cluster delete HPA php-apache
```
## Alternative ways to use federated HPA
To a federation user interacting with federated control plane (or simply federation),
the interaction is almost identical to interacting with a normal Kubernetes cluster (but
with a limited set of APIs that are federated). As both Deployments and
HorizontalPodAutoscalers are now federated, `kubectl` commands like `kubectl run`
and `kubectl autoscale` work on federation. Given this fact, the mechanism specified in
[horizontal pod autoscaler walkthrough](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough)
will also work when used with federation.
Care however will need to be taken that when
[generating load on a target deployment](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#step-three-increase-load),
it should be done against a specific federated cluster (or multiple clusters) not the federation.
## Conclusion
The use of federated HPA is to ensure workload replicas move to the cluster(s) where
they are needed most, or in other words where the load is beyond expected threshold.
The federated HPA feature achieves this by manipulating the min and max replicas on the
HPAs it creates in the federated clusters. It does not directly monitor the target
object metrics from the federated clusters. It actually relies on the in-cluster HPA
controllers to monitor the metrics and update relevant fields. The in-cluster HPA
controller monitors the target pod metrics and updates the fields like desired
replicas (after metrics based calculations) and current replicas (observing the
current status of in cluster pods). The federated HPA controller, on the other hand,
monitors only the cluster-specific HPA object fields and updates the min replica and
max replica fields of those in cluster HPA objects, which have replicas matching thresholds.
For example, if a cluster has both desired replicas and current replicas the same as the max replicas,
and averaged current CPU utilization still higher than the target CPU utilization (all of which
are fields on local HPA object), then the target app in this cluster
needs more replicas, and the scaling is currently restricted by max replicas set on this local
HPA object. In such a scenario, the federated HPA controller scans all clusters and tries to
find clusters which do not have such a condition (meaning the the desired replicas are less
than the max, and current averaged cpu utilization is lower then the threshold). If it finds such
a cluster, it reduces the max replica on the HPA in this cluster and increases the max replicas
on the HPA in the cluster which needed the replicas.
There are many other similar conditions which the federated HPA controller checks and moves the max
replicas and min replicas around the local HPAs in federated clusters to eventually ensure that
the replicas move (or remain) in the cluster(s) which need them.
For more information, see ["federated HPA design proposal"](https://github.com/kubernetes/community/pull/593).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/administer-federation/job.md
+102
View File
@@ -0,0 +1,102 @@
---
title: Federated Jobs
---
{% capture overview %}
This guide explains how to use jobs in the federation control plane.
Jobs in the federation control plane (referred to as "federated jobs" in
this guide) are similar to the traditional [Kubernetes
jobs](/docs/concepts/workloads/controllers/job/), and provide the same functionality.
Creating jobs in the federation control plane ensures that the desired number of
parallelism and completions exist across the registered clusters.
{% endcapture %}
{% capture prerequisites %}
* {% include federated-task-tutorial-prereqs.md %}
* You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [jobs](/docs/concepts/workloads/controllers/jobs-run-to-completion/) in particular.
{% endcapture %}
{% capture steps %}
## Creating a federated job
The API for federated jobs is fully compatible with the
API for traditional Kubernetes jobs. You can create a job by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f myjob.yaml
```
The '--context=federation-cluster' flag tells kubectl to submit the
request to the federation API server instead of sending it to a Kubernetes
cluster.
Once a federated job is created, the federation control plane creates
a job in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get job myjob
```
The previous example assumes that you have a context named `gce-asia-east1a`
configured in your client for your cluster in that zone.
The jobs in the underlying clusters match the federated job
except in the number of parallelism and completions. The federation control plane ensures that the
sum of the parallelism and completions in each cluster matches the desired number of parallelism and completions in the
federated job.
### Spreading job tasks in underlying clusters
By default, parallelism and completions are spread equally in all underlying clusters. For example:
if you have 3 registered clusters and you create a federated job with
`spec.parallelism = 9` and `spec.completions = 18`, then each job in the 3 clusters has
`spec.parallelism = 3` and `spec.completions = 6`.
To modify the number of parallelism and completions in each cluster, you can specify
[ReplicaAllocationPreferences](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go)
as an annotation with key `federation.kubernetes.io/job-preferences`
on the federated job.
## Updating a federated job
You can update a federated job as you would update a Kubernetes
job; however, for a federated job, you must send the request to
the federation API server instead of sending it to a specific Kubernetes cluster.
The federation control plane ensures that whenever the federated job is
updated, it updates the corresponding job in all underlying clusters to
match it.
If your update includes a change in number of parallelism and completions, the federation
control plane changes the number of parallelism and completions in underlying clusters to
ensure that their sum remains equal to the number of desired parallelism and completions in
federated job.
## Deleting a federated job
You can delete a federated job as you would delete a Kubernetes
job; however, for a federated job, you must send the request to
the federation API server instead of sending it to a specific Kubernetes cluster.
For example, with kubectl:
```shell
kubectl --context=federation-cluster delete job myjob
```
**Note:** Deleting a federated job will not delete the
corresponding jobs from underlying clusters.
You must delete the underlying jobs manually.
{: .note}
{% endcapture %}
{% include templates/task.md %}
docs/tasks/configure-pod-container/opaque-integer-resource.md
+1 -1
View File
@@ -6,7 +6,7 @@ title: Assign Opaque Integer Resources to a Container
This page shows how to assign opaque integer resources to a Container.
{% include feature-state-alpha.md %}
{% include feature-state-deprecated.md %}
{% endcapture %}
docs/tasks/configure-pod-container/security-context-2.yaml
+2 -1
View File
@@ -1,7 +1,7 @@
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo-2
name: security-context-demo-2
spec:
securityContext:
runAsUser: 1000
@@ -10,3 +10,4 @@ spec:
image: gcr.io/google-samples/node-hello:1.0
securityContext:
runAsUser: 2000
allowPrivilegeEscalation: false
docs/tasks/configure-pod-container/security-context.md
+4
View File
@@ -24,6 +24,8 @@ a Pod or Container. Security context settings include:
* [Seccomp](https://en.wikipedia.org/wiki/Seccomp): Limit a process's access to open file descriptors.
* AllowPrivilegeEscalation: Controls whether a process can gain more privileges than its parent process. This bool directly controls whether the [`no_new_privs`](https://www.kernel.org/doc/Documentation/prctl/no_new_privs.txt) flag gets set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged OR 2) has `CAP_SYS_ADMIN`.
For more information about security mechanisms in Linux, see
[Overview of Linux Kernel Security Features](https://www.linux.com/learn/overview-linux-kernel-security-features)
@@ -345,6 +347,8 @@ label given to all Containers in the Pod as well as the Volumes.
* [Security Contexts design document](https://git.k8s.io/community/contributors/design-proposals/auth/security_context.md)
* [Ownership Management design document](https://git.k8s.io/community/contributors/design-proposals/storage/volume-ownership-management.md)
* [Pod Security Policies](/docs/concepts/policy/pod-security-policy/)
* [AllowPrivilegeEscalation design
document](https://git.k8s.io/community/contributors/design-proposals/auth/no-new-privs.md)
{% endcapture %}
docs/tasks/configure-pod-container/security-context.yaml
+2 -1
View File
@@ -1,7 +1,7 @@
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
name: security-context-demo
spec:
securityContext:
runAsUser: 1000
@@ -15,3 +15,4 @@ spec:
volumeMounts:
- name: sec-ctx-vol
mountPath: /data/demo
allowPrivilegeEscalation: false
docs/tasks/debug-application-cluster/audit.md
+260 -58
View File
@@ -22,7 +22,7 @@ answer the following questions:
- from where was it initiated?
- to where was it going?
## Audit logs
## Legacy Audit
Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests
processed by the server. Each audit log entry contains two lines:
@@ -37,8 +37,13 @@ Example output for `admin` user listing pods in the `default` namespace:
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"
```
Note that this format changes when enabling the `AdvancedAuditing` feature discussed
later in this document.
Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default.
To fallback to this legacy audit, disable the advanced auditing feature
using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]:
```
--feature-gates=AdvancedAuditing=false
```
### Configuration
@@ -62,18 +67,12 @@ and `audit-log-maxage` options.
## Advanced audit
Kubernetes 1.7 expands auditing with experimental functionality such as event
filtering and a webhook for integration with external systems. The rest of this
document covers features that are __alpha__ and may change in backward incompatible
ways.
filtering and a webhook for integration with external systems. Kubernetes 1.8
upgrades the advanced audit feature to beta, and some backward incompatible changes
have been committed.
Enable the alpha auditing features using the `AdvancedAuditing` feature gate on
the [kube-apiserver][kube-apiserver]:
```
--feature-gates=AdvancedAuditing=true
```
`AdvancedAuditing`is customizable in two ways. Policy, which determines what's recorded,
`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded,
and backends, which persist records. Backend implementations include logs files and
webhooks.
@@ -110,10 +109,17 @@ The policy file holds rules that determine the level of an event. Known audit le
When an event is processed, it's compared against the list of rules in order.
The first matching rule sets the audit level of the event. The audit policy is
defined by the [`audit.k8s.io` API group][audit-api].
Some new fields are supported in beta version, like `resourceNames` and `omitStages`.
An example audit policy file:
In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in
the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide
a valid `apiVersion` and `kind` value will be treated as illgal.
Some example audit policy files:
```yaml
apiVersion: audit.k8s.io/v1beta1 #this is required in Kubernetes 1.8
kind: Policy
rules:
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
@@ -155,10 +161,46 @@ rules:
- level: Metadata
```
The next audit policy file shows new features introduced in Kubernetes 1.8:
```yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# Log pod changes at Request level
- level: Request
resources:
- group: ""
# Resource "pods" no longer matches requests to any subresource of pods,
# This behavior is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# A catch-all rule to log all other requests at the Metadata level.
# For this rule we use "omitStages" to omit events at "ReqeustReceived" stage.
# Events in this stage will not be sent to backend.
- level: Metadata
omitStages:
- "RequestReceived"
```
You can use a minimal audit policy file to log all requests at the `Metadata` level:
```yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
```
@@ -181,18 +223,35 @@ API can be found [here][audit-api] with more details about the exact fields capt
#### Log backend
The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing`
feature flag. This includes the cleanups discussed above, such as changes to the `method`
values and the introduction of a "stage" for each event. As before, the `id` field of
the log line indicates which events were generated from the same request. Events are
formatted as follows:
feature flag. All generated events defined by `--audit-policy-file` are recorded in structured
json format:
```
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="<deferred>"
2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="200"
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}}
```
In alpha version, objectRef.apiVersion holds both the api group and version.
In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion.
Starting from Kubernetes 1.8, structured json format is used for log backend by default.
Use the following option to switch log to legacy format:
```
--audit-log-format=legacy
```
With legacy format, events are formatted as follows:
```
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="<deferred>"
2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200"
```
Logged events omit the request and response bodies. The `Request` and
`RequestResponse` levels are equivalent to `Metadata` for this backend.
`RequestResponse` levels are equivalent to `Metadata` for legacy format. This legacy format
of advanced audit is different from the [Legacy Audit](# Legacy Audit) discussed above, such
as changes to the method values and the introduction of a "stage" for each event.
#### Webhook backend
@@ -241,44 +300,187 @@ Events are POSTed as a JSON serialized `EventList`. An example payload:
```json
{
"kind": "EventList",
"apiVersion": "audit.k8s.io/v1alpha1",
"items": [
{
"metadata": {
"creationTimestamp": null
},
"level": "Metadata",
"timestamp": "2017-06-15T23:07:40Z",
"auditID": "4faf711a-9094-400f-a876-d9188ceda548",
"stage": "ResponseComplete",
"requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer",
"verb": "get",
"user": {
"username": "system:apiserver",
"uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e",
"groups": [
"system:masters"
]
},
"sourceIPs": [
"127.0.0.1"
],
"objectRef": {
"resource": "rolebindings",
"namespace": "kube-public",
"name": "system:controller:bootstrap-signer",
"apiVersion": "rbac.authorization.k8s.io/v1beta1"
},
"responseStatus": {
"metadata": {},
"code": 200
}
}
]
"apiVersion": "audit.k8s.io/v1beta1",
"items": [
{
"auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886",
"level": "Metadata",
"metadata": {
"creationTimestamp": null
},
"objectRef": {
"apiGroup": "rbac.authorization.k8s.io",
"apiVersion": "v1",
"name": "jane",
"namespace": "default",
"resource": "roles"
},
"requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane",
"responseStatus": {
"code": 200,
"metadata": {}
},
"sourceIPs": [
"172.16.116.128"
],
"stage": "ResponseComplete",
"timestamp": "2017-09-05T10:20:24Z",
"user": {
"groups": [
"system:masters",
"system:authenticated"
],
"username": "kubecfg"
},
"verb": "get"
}
],
"kind": "EventList",
"metadata": {}
}
```
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1alpha1/types.go
### Audit-Id
Audit-Id is a unique ID for each http request to kube-apiserver. The ID of events will be the
same if they were generated from the same request. Starting from Kubernetes 1.8, if an audit
event is generated for the request, kube-apiserver will respond with an Audit-Id in the HTTP header.
Note that for some special requests like `kubectl exec`, `kubectl attach`, kube-apiserver works
like a proxy, no Audit-Id will be returned even if audit events are recorded.
### Log Collector Examples
#### Use fluentd to collect and distribute audit events from log file
[Fluentd][fluentd] is an open source data collector for unified logging layer.
In this example, we will use fluentd to split audit events by different namespaces.
Note that this example requries json format output support in Kubernetes 1.8.
1. install [fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter][fluentd_install_doc] in the kube-apiserver node
1. create a config file for fluentd
$ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
rewriterule1 namespace ^(.+) ${tag}.$1
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
1. start fluentd
$ fluentd -c /etc/fluentd/config -vv
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
1. check audits for different namespaces in /var/log/audit-*.log
#### Use logstash to collect and distribute audit events from webhook backend
[Logstash][logstash] is an open source, server-side data processing tool. In this example,
we will use logstash to collect audit events from webhook backend, and save events of
different users into different files.
1. install [logstash][logstash_install_doc]
1. create config file for logstash
$ cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
1. start logstash
$ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
1. create a [kubeconfig file](/docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/) for kube-apiserver webhook audit backend
$ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
1. check audits in logstash node's directories /var/log/kube-audit-*/audit
Note that in addition to file output plugin, logstash has a variety of outputs that
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go
[kube-apiserver]: /docs/admin/kube-apiserver
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.7.0/cluster/gce/gci/configure-helper.sh#L490
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd
[logstash]: https://www.elastic.co/products/logstash
[logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
docs/tasks/debug-application-cluster/core-metrics-pipeline.md
+44
View File
@@ -0,0 +1,44 @@
---
approvers:
- fgrzadkowski
- piosz
title: Core metrics pipeline
---
Starting from Kubernetes 1.8, resource usage metrics, such as container CPU and memory usage,
are available in Kubernetes through the Metrics API. These metrics can be either accessed directly
by user, for example by using `kubectl top` command, or used by a controller in the cluster, e.g.
Horizontal Pod Autoscaler, to make decisions.
## The Metrics API
Through the Metrics API you can get the amount of resource currently used
by a given node or a given pod. This API doesn't store the metric values,
so it's not possible for example to get the amount of resources used by a
given node 10 minutes ago.
The API no different from any other API:
- it is discoverable through the same endpoint as the other Kubernetes APIs under `/apis/metrics.k8s.io/` path
- it offers the same security, scalability and reliability guarantees
The API is defined in [k8s.io/metrics](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go)
repository. You can find more information about the API there.
**Note:** The API requires metrics server to be deployed in the cluster. Otherwise it will be not available.
## Metrics Server
[Metrics Server](https://github.com/kubernetes-incubator/metrics-server) is a cluster-wide aggregator of resource usage data.
Starting from Kubernetes 1.8 it's deployed by default in clusters created by `kube-up.sh` script
as a Deployment object. If you use a different Kubernetes setup mechanism you can deploy it using the provided
[deployment yamls](https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy).
It's supported in Kubernetes 1.7+ (see details below).
Metric server collects metrics from the Summary API, exposed by [Kubelet](/docs/admin/kubelet/) on each node.
Metrics Server registered in the main API server through
[Kubernetes aggregator](https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/),
which was introduced in Kubernetes 1.7.
Learn more about the metrics server in [the design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md).
docs/tasks/debug-application-cluster/debug-stateful-set.md
+7 -42
View File
@@ -18,7 +18,6 @@ This task shows you how to debug a StatefulSet.
{% capture prerequisites %}
* You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
* You should have a StatefulSet running that you want to investigate.
@@ -28,52 +27,18 @@ This task shows you how to debug a StatefulSet.
## Debugging a StatefulSet
In order to list all the pods which belong to a StatefulSet, which have a label `app=myapp` set on them, you can use the following:
In order to list all the pods which belong to a StatefulSet, which have a label `app=myapp` set on them,
you can use the following:
```shell
kubectl get pods -l app=myapp
```
If you find that any Pods listed are in `Unknown` or `Terminating` state for an extended period of time, refer to the [Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) task for instructions on how to deal with them. You can debug individual Pods in a StatefulSet using the
[Debugging Pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods) guide.
StatefulSets provide a debug mechanism to pause all controller operations on Pods using an annotation. Setting the `pod.alpha.kubernetes.io/initialized` annotation to `"false"` on any StatefulSet Pod will *pause* all operations of the StatefulSet. When paused, the StatefulSet will not perform any scaling operations. Once the debug hook is set, you can execute commands within the containers of StatefulSet pods without interference from scaling operations. You can set the annotation to `"false"` by executing the following:
```shell
kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="false" --overwrite
```
When the annotation is set to `"false"`, the StatefulSet will not respond to its Pods becoming unhealthy or unavailable. It will not create replacement Pods till the annotation is removed or set to `"true"` on each StatefulSet Pod.
### Step-wise Initialization
You can also use the same annotation to debug race conditions during bootstrapping of the StatefulSet by setting the `pod.alpha.kubernetes.io/initialized` annotation to `"false"` in the `.spec.template.metadata.annotations` field of the StatefulSet prior to creating it.
```yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: my-app
spec:
serviceName: "my-app"
replicas: 3
template:
metadata:
labels:
app: my-app
annotations:
pod.alpha.kubernetes.io/initialized: "false"
...
...
...
```
After setting the annotation, if you create the StatefulSet, you can wait for each Pod to come up and verify that it has initialized correctly. The StatefulSet will not create any subsequent Pods till the debug annotation is set to `"true"` (or removed) on each Pod that has already been created. You can set the annotation to `"true"` by executing the following:
```shell
kubectl annotate pods <pod-name> pod.alpha.kubernetes.io/initialized="true" --overwrite
```
If you find that any Pods listed are in `Unknown` or `Terminating` state for an extended period of time,
refer to the [Deleting StatefulSet Pods](/docs/tasks/manage-stateful-set/delete-pods/) task for
instructions on how to deal with them.
You can debug individual Pods in a StatefulSet using the
[Debugging Pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/) guide.
{% endcapture %}
docs/tasks/debug-application-cluster/event-exporter-deploy.yaml
+1 -1
View File
@@ -6,7 +6,7 @@ metadata:
labels:
app: event-exporter
---
apiVersion: rbac.authorization.k8s.io/v1beta1
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: event-exporter-rb
docs/tasks/debug-application-cluster/logging-stackdriver.md
+12 -8
View File
@@ -245,15 +245,19 @@ In this case you need to be able to change the parameters of `DaemonSet` and `Co
### Prerequisites
If you're on GKE and Stackdriver Logging is enabled in your cluster, you cannot change its
parameters. Likewise, if you're not on GKE, but Stackdriver Logging is installed as an addon,
you won't be able to change deployment parameters using Kubernetes API. To make it possible
to change parameters of Stackdriver Logging agents, you should switch to the API object
deployment, when Stackdriver Logging is installed into a running cluster that didn't have any
cluster logging solutions installed before that.
If you're using GKE and Stackdriver Logging is enabled in your cluster, you
cannot change its configuration, because it's managed and supported by GKE.
However, you can disable the default integration and deploy your own. Note,
that you will have to support and maintain a newly deployed configuration
yourself: update the image and configuration, adjust the resources and so on.
To disable the default logging integration, use the following command:
You can find notes on how to install Stackdriver Logging agents into a running cluster in the
[Deploying section](#deploying).
```
gcloud beta container clusters update --logging-service=none CLUSTER
```
You can find notes on how to then install Stackdriver Logging agents into
a running cluster in the [Deploying section](#deploying).
### Changing `DaemonSet` parameters
docs/tasks/extend-kubectl/kubectl-plugins.md
+137
View File
@@ -0,0 +1,137 @@
---
title: Extend kubectl with plugins
approvers:
- fabianofranz
description: With kubectl plugins, you can extend the functionality of the kubectl command by adding new subcommands.
---
{% capture overview %}
{% include feature-state-alpha.md %}
This guide shows you how to install and write extensions for [kubectl](/docs/user-guide/kubectl). Usually called *plugins* or *binary extensions*, this feature allows you to extend the default set of commands available in `kubectl` by adding new subcommands to perform new tasks and extend the set of features available in the main distribution of `kubectl`.
{% endcapture %}
{% capture prerequisites %}
You need to have a working `kubectl` binary installed. Note that plugins were officially introduced as an alpha feature in the v1.8.0 release. So, while some parts of the plugins feature were already available in previous versions, a `kubectl` version of 1.8.0 or later is recommended.
Until a GA version is released, plugins will only be available under the `kubectl plugin` subcommand.
{% endcapture %}
{% capture steps %}
## Installing kubectl plugins
A plugin is nothing more than a set of files: at least a **plugin.yaml** descriptor, and likely one or more binary, script, or assets files. To install a plugin, copy those files to one of the locations in the filesystem where `kubectl` searches for plugins.
Note that Kubernetes does not provide a package manager or something similar to install or update plugins, so it's your responsibility to place the plugin files in the correct location. We recommend that each plugin is located on its own directory, so installing a plugin that is distributed as a compressed file is as simple as extracting it to one of the locations specified in the [Plugin loader](#plugin-loader) section.
### Plugin loader
The plugin loader is responsible for searching plugin files in the filesystem locations specified below, and checking if the plugin provides the minimum amount of information required for it to run. Files placed in the right location that don't provide the minimum amount of information, for example an incomplete *plugin.yaml* descriptor, are ignored.
#### Search order
The plugin loader uses the following search order:
1. `${KUBECTL_PLUGINS_PATH}` If specified, the search stops here.
2. `${XDG_DATA_DIRS}/kubectl/plugins`
3. `~/.kube/plugins`
If the `KUBECTL_PLUGINS_PATH` environment variable is present, the loader uses it as the only location to look for plugins.
The `KUBECTL_PLUGINS_PATH` environment variable is a list of directories. In Linux and Mac, the list is colon-delimited. In
Windows, the list is semicolon-delimited.
If `KUBECTL_PLUGINS_PATH` is not present, the loader searches these additional locations:
First, one or more directories specified according to the
[XDG System Directory Structure](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
specification. Specifically, the loader locates the directories specified by the `XDG_DATA_DIRS` environment variable,
and then searches `kubectl/plugins` directory inside of those.
If `XDG_DATA_DIRS` is not specified, it defaults to `/usr/local/share:/usr/share`.
Second, the `plugins` directory under the user's kubeconfig dir. In most cases, this is `~/.kube/plugins`.
```shell
# Loads plugins from both /path/to/dir1 and /path/to/dir2
KUBECTL_PLUGINS_PATH=/path/to/dir1:/path/to/dir2 kubectl plugin -h
```
## Writing kubectl plugins
You can write a plugin in any programming language or script that allows you to write command-line commands.
A plugin does not necessarily need to have a binary component. It could rely entirely on operating system utilities
like `echo`, `sed`, or `grep`. Or it could rely on the `kubectl` binary.
The only strong requirement for a `kubectl` plugin is the `plugin.yaml` descriptor file. This file is responsible for declaring at least the minimum attributes required to register a plugin and must be located under one of the locations specified in the [Search order](#search-order) section.
### The plugin.yaml descriptor
The descriptor file supports the following attributes:
```
name: "targaryen" # REQUIRED: the plugin command name, to be invoked under 'kubectl'
shortDesc: "Dragonized plugin" # REQUIRED: the command short description, for help
longDesc: "" # the command long description, for help
example: "" # command example(s), for help
command: "./dracarys" # REQUIRED: the command, binary, or script to invoke when running the plugin
flags: # flags supported by the plugin
- name: "heat" # REQUIRED for each flag: flag name
shorthand: "h" # short version of the flag name
desc: "Fire heat" # REQUIRED for each flag: flag description
defValue: "extreme" # default value of the flag
tree: # allows the declaration of subcommands
- ... # subcommands support the same set of attributes
```
The preceding descriptor declares the `kubectl plugin targaryen` plugin, which has one flag named `-h | --heat`.
When the plugin is invoked, it calls the `dracarys` binary or script, which is located in the same directory as the descriptor file. The [Accessing runtime attributes](#accessing-runtime-attributes) section describes how the `dracarys` command accesses the flag value and other runtime context.
### Recommended directory structure
It is recommended that each plugin has its own subdirectory in the filesystem, preferably with the same name as the plugin command. The directory must contain the `plugin.yaml` descriptor and any binary, script, asset, or other dependency it might require.
For example, the directory structure for the `targaryen` plugin could look like this:
```
~/.kube/plugins/
└── targaryen
├── plugin.yaml
└── dracarys
```
### Accessing runtime attributes
In most use cases, the binary or script file you write to support the plugin must have access to some contextual information provided by the plugin framework. For example, if you declared flags in the descriptor file, your plugin must have access to the user-provided flag values at runtime. The same is true for global flags. The plugin framework is responsible for doing that, so plugin writers don't need to worry about parsing arguments. This also ensures the best level of consistency between plugins and regular `kubectl` commands.
Plugins have access to runtime context attributes through environment variables. So to access the value provided through a flag, for example, just look for the value of the proper environment variable using the appropriate function call for your binary or script.
The supported environment variables are:
* `KUBECTL_PLUGINS_CALLER`: The full path to the `kubectl` binary that was used in the current command invocation.
As a plugin writer, you don't have to implement logic to authenticate and access the Kubernetes API. Instead, you can invoke `kubectl` to obtain the information you need, through something like `kubectl get --raw=/apis`.
* `KUBECTL_PLUGINS_CURRENT_NAMESPACE`: The current namespace that is the context for this call. This is the actual namespace to be used, meaning it was already processed in terms of the precedence between what was provided through the kubeconfig, the `--namespace` global flag, environment variables, and so on.
* `KUBECTL_PLUGINS_DESCRIPTOR_*`: One environment variable for every attribute declared in the `plugin.yaml` descriptor.
For example, `KUBECTL_PLUGINS_DESCRIPTOR_NAME`, `KUBECTL_PLUGINS_DESCRIPTOR_COMMAND`.
* `KUBECTL_PLUGINS_GLOBAL_FLAG_*`: One environment variable for every global flag supported by `kubectl`.
For example, `KUBECTL_PLUGINS_GLOBAL_FLAG_NAMESPACE`, `KUBECTL_PLUGINS_GLOBAL_FLAG_V`.
* `KUBECTL_PLUGINS_LOCAL_FLAG_*`: One environment variable for every local flag declared in the `plugin.yaml` descriptor. For example, `KUBECTL_PLUGINS_LOCAL_FLAG_HEAT` in the preceding `targaryen` example.
{% endcapture %}
{% capture whatsnext %}
* Check the repository for [some more examples](https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl/plugins/examples) of plugins.
* In case of any questions, feel free to reach out to the [CLI SIG team](https://github.com/kubernetes/community/tree/master/sig-cli).
* Binary plugins is still an alpha feature, so this is the time to contribute ideas and improvements to the codebase. We're also excited to hear about what you're planning to implement with plugins, so [let us know](https://github.com/kubernetes/community/tree/master/sig-cli)!
{% endcapture %}
{% include templates/task.md %}
docs/tasks/index.md
+5 -1
View File
@@ -52,7 +52,7 @@ Configure components in a cluster federation.
#### Managing Stateful Applications
Perform common tasks for managing Stateful applications, including upgrading from PetSets and scaling, deleting, and debugging StatefulSets.
Perform common tasks for managing Stateful applications, including scaling, deleting, and debugging StatefulSets.
#### Cluster Daemons
@@ -62,6 +62,10 @@ Perform common tasks for managing a DaemonSet, such as performing a rolling upda
Configure and schedule NVIDIA GPUs for use as a resource by nodes in a cluster.
#### Managing HugePages
Configure and schedule huge pages as a schedulable resource in a cluster.
### What's next
If you would like to write a task page, see
docs/tasks/manage-hugepages/scheduling-hugepages.md
+81
View File
@@ -0,0 +1,81 @@
---
approvers:
- derekwaynecarr
title: Manage HugePages
---
{% capture overview %}
{% include feature-state-alpha.md %}
Kubernetes supports the allocation and consumption of pre-allocated huge pages
by applications in a Pod as an **alpha** feature. This page describes how users
can consume huge pages and the current limitations.
{% endcapture %}
{% capture prerequisites %}
1. Kubernetes nodes must pre-allocate huge pages in order for the node to report
its huge page capacity. A node may only pre-allocate huge pages for a single
size.
1. A special **alpha** feature gate `HugePages` has to be set to true across the
system: `--feature-gates="HugePages=true"`.
The nodes will automatically discover and report all huge page resources as a
schedulable resource.
{% endcapture %}
{% capture steps %}
## API
Huge pages can be consumed via container level resource requirements using the
resource name `hugepages-<size>`, where size is the most compact binary notation
using integer values supported on a particular node. For example, if a node
supports 2048KiB page sizes, it will expose a schedulable resource
`hugepages-2Mi`. Unlike CPU or memory, huge pages do not support overcommit.
```yaml
apiVersion: v1
kind: Pod
metadata:
generateName: hugepages-volume-
spec:
containers:
- image: fedora:latest
command:
- sleep
- inf
name: example
volumeMounts:
- mountPath: /hugepages
name: hugepage
resources:
limits:
hugepages-2Mi: 100Mi
volumes:
- name: hugepage
emptyDir:
medium: HugePages
```
- Huge page requests must equal the limits. This is the default if limits are
specified, but requests are not.
- Huge pages are isolated at a pod scope, container isolation is planned in a
future iteration.
- EmptyDir volumes backed by huge pages may not consume more huge page memory
than the pod request.
- Applications that consume huge pages via `shmget()` with `SHM_HUGETLB` must
run with a supplemental group that matches `proc/sys/vm/hugetlb_shm_group`
## Future
- Support container isolation of huge pages in addition to pod isolation.
- NUMA locality guarnatees as a feature of quality of service.
- ResourceQuota support.
- LimitRange support.
{% endcapture %}
{% include templates/task.md %}
docs/tasks/run-application/deployment-patch-demo.yaml
+7 -1
View File
@@ -1,9 +1,15 @@
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: patch-demo
spec:
selector:
matchLabels:
app: nginx
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
docs/tasks/run-application/deployment-scale.yaml
+4 -1
View File
@@ -1,8 +1,11 @@
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 4 # Update the replicas from 2 to 4
template:
metadata:
docs/tasks/run-application/deployment-update.yaml
+4 -1
View File
@@ -1,8 +1,11 @@
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
docs/tasks/run-application/deployment.yaml
+4 -1
View File
@@ -1,8 +1,11 @@
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template: # create pods using pod definition in this template
metadata:
docs/tasks/run-application/gce-volume.yaml
-12
View File
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: mysql-disk
fsType: ext4
docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
+8 -7
View File
@@ -9,7 +9,7 @@ title: Horizontal Pod Autoscaling Walkthrough
Horizontal Pod Autoscaling automatically scales the number of pods
in a replication controller, deployment or replica set based on observed CPU utilization
(or, with alpha support, on some other, application-provided metrics).
(or, with beta support, on some other, application-provided metrics).
This document walks you through an example of enabling Horizontal Pod Autoscaling for the php-apache server. For more information on how Horizontal Pod Autoscaling behaves, see the [Horizontal Pod Autoscaling user guide](/docs/tasks/run-application/horizontal-pod-autoscale/).
@@ -129,18 +129,18 @@ Here CPU utilization dropped to 0, and so HPA autoscaled the number of replicas
## Autoscaling on multiple metrics and custom metrics
You can introduce additional metrics to use when autoscaling the `php-apache` Deployment
by making use of the `autoscaling/v2alpha1` API version.
by making use of the `autoscaling/v2beta1` API version.
First, get the YAML of your HorizontalPodAutoscaler in the `autoscaling/v2alpha1` form:
First, get the YAML of your HorizontalPodAutoscaler in the `autoscaling/v2beta1` form:
```shell
$ kubectl get hpa.v2alpha1.autoscaling -o yaml > /tmp/hpa-v2.yaml
$ kubectl get hpa.v2beta1.autoscaling -o yaml > /tmp/hpa-v2.yaml
```
Open the `/tmp/hpa-v2.yaml` file in an editor, and you should see YAML which looks like this:
```yaml
apiVersion: autoscaling/v2alpha1
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
@@ -189,6 +189,7 @@ are averaged together across pods and compared with a target value to determine
They work much like resource metrics, except that they *only* have the `targetAverageValue` field.
Pod metrics are specified using a metric block like this:
```yaml
type: Pods
pods:
@@ -220,7 +221,7 @@ For example, if you had your monitoring system collecting metrics about network
you could update the definition above using `kubectl edit` to look like this:
```yaml
apiVersion: autoscaling/v2alpha1
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
@@ -268,7 +269,7 @@ Ingress were serving a total of 10000 requests per second.
## Appendix: Horizontal Pod Autoscaler Status Conditions
When using the `autoscaling/v2alpha1` form of the HorizontalPodAutoscaler, you will be able to see
When using the `autoscaling/v2beta1` form of the HorizontalPodAutoscaler, you will be able to see
*status conditions* set by Kubernetes on the HorizontalPodAutoscaler. These status conditions indicate
whether or not the HorizontalPodAutoscaler is able to scale, and whether or not it is currently restricted
in any way.
docs/tasks/run-application/horizontal-pod-autoscale.md
+6 -6
View File
@@ -12,7 +12,7 @@ This document describes the current state of Horizontal Pod Autoscaling in Kuber
With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods
in a replication controller, deployment or replica set based on observed CPU utilization
(or, with alpha support, on some other, application-provided metrics). Note that Horizontal
(or, with beta support, on some other, application-provided metrics). Note that Horizontal
Pod Autoscaling does not apply to objects that can't be scaled, for example, DaemonSet.
The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller.
@@ -73,8 +73,8 @@ The Horizontal Pod Autoscaler is an API resource in the Kubernetes `autoscaling`
The current stable version, which only includes support for CPU autoscaling,
can be found in the `autoscaling/v1` API version.
The alpha version, which includes support for scaling on memory and custom metrics,
can be found in `autoscaling/v2alpha1`. The new fields introduced in `autoscaling/v2alpha1`
The beta version, which includes support for scaling on memory and custom metrics,
can be found in `autoscaling/v2beta1`. The new fields introduced in `autoscaling/v2beta1`
are preserved as annotations when working with `autoscaling/v1`.
More details about the API object can be found at
@@ -108,7 +108,7 @@ the Horizontal Pod Autoscaler will not be bound to the new replication controlle
## Support for multiple metrics
Kubernetes 1.6 adds support for scaling based on multiple metrics. You can use the `autoscaling/v2alpha1` API
Kubernetes 1.6 adds support for scaling based on multiple metrics. You can use the `autoscaling/v2beta1` API
version to specify multiple metrics for the Horizontal Pod Autoscaler to scale on. Then, the Horizontal Pod
Autoscaler controller will evaluate each metric, and propose a new scale based on that metric. The largest of the
proposed scales will be used as the new scale.
@@ -116,12 +116,12 @@ proposed scales will be used as the new scale.
## Support for custom metrics
**Note**: Kubernetes 1.2 added alpha support for scaling based on application-specific metrics using special annotations.
Support for these annotations was removed in Kubernetes 1.6 in favor of the `autoscaling/v2alpha1` API. While the old method for collecting
Support for these annotations was removed in Kubernetes 1.6 in favor of the new autoscaling API. While the old method for collecting
custom metrics is still available, these metrics will not be available for use by the Horizontal Pod Autoscaler, and the former
annotations for specifying which custom metrics to scale on are no longer honored by the Horizontal Pod Autoscaler controller.
Kubernetes 1.6 adds support for making use of custom metrics in the Horizontal Pod Autoscaler.
You can add custom metrics for the Horizontal Pod Autoscaler to use in the `autoscaling/v2alpha1` API.
You can add custom metrics for the Horizontal Pod Autoscaler to use in the `autoscaling/v2beta1` API.
Kubernetes then queries the new custom metrics API to fetch the values of the appropriate custom metrics.
### Requirements
docs/tasks/run-application/mysql-deployment.yaml
+4 -2
View File
@@ -16,16 +16,18 @@ metadata:
spec:
accessModes:
- ReadWriteOnce
storageClassName: ""
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
docs/tasks/run-application/mysql-statefulset.yaml
+55 -52
View File
@@ -1,62 +1,66 @@
apiVersion: apps/v1beta1
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
serviceName: mysql
replicas: 3
template:
metadata:
labels:
app: mysql
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "init-mysql",
"image": "mysql:5.7",
"command": ["bash", "-c", "
set -ex\n
# Generate mysql server-id from pod ordinal index.\n
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1\n
ordinal=${BASH_REMATCH[1]}\n
echo [mysqld] > /mnt/conf.d/server-id.cnf\n
# Add an offset to avoid reserved server-id=0 value.\n
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf\n
# Copy appropriate conf.d files from config-map to emptyDir.\n
if [[ $ordinal -eq 0 ]]; then\n
cp /mnt/config-map/master.cnf /mnt/conf.d/\n
else\n
cp /mnt/config-map/slave.cnf /mnt/conf.d/\n
fi\n
"],
"volumeMounts": [
{"name": "conf", "mountPath": "/mnt/conf.d"},
{"name": "config-map", "mountPath": "/mnt/config-map"}
]
},
{
"name": "clone-mysql",
"image": "gcr.io/google-samples/xtrabackup:1.0",
"command": ["bash", "-c", "
set -ex\n
# Skip the clone if data already exists.\n
[[ -d /var/lib/mysql/mysql ]] && exit 0\n
# Skip the clone on master (ordinal index 0).\n
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1\n
ordinal=${BASH_REMATCH[1]}\n
[[ $ordinal -eq 0 ]] && exit 0\n
# Clone data from previous peer.\n
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql\n
# Prepare the backup.\n
xtrabackup --prepare --target-dir=/var/lib/mysql\n
"],
"volumeMounts": [
{"name": "data", "mountPath": "/var/lib/mysql", "subPath": "mysql"},
{"name": "conf", "mountPath": "/etc/mysql/conf.d"}
]
}
]'
spec:
initContainers:
- name: init-mysql
image: mysql:5.7
command:
- bash
- "-c"
- |
set -ex
# Generate mysql server-id from pod ordinal index.
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# Add an offset to avoid reserved server-id=0 value.
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# Copy appropriate conf.d files from config-map to emptyDir.
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: conf
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql
image: gcr.io/google-samples/xtrabackup:1.0
command:
- bash
- "-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] && exit 0
# Skip the clone on master (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# Clone data from previous peer.
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql
image: mysql:5.7
@@ -74,18 +78,20 @@ spec:
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 1
cpu: 500m
memory: 1Gi
livenessProbe:
exec:
command: ["mysqladmin", "ping"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
# Check we can execute queries over TCP (skip-networking is off).
command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
- name: xtrabackup
image: gcr.io/google-samples/xtrabackup:1.0
@@ -154,11 +160,8 @@ spec:
volumeClaimTemplates:
- metadata:
name: data
annotations:
volume.alpha.kubernetes.io/storage-class: default
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
docs/tasks/run-application/patch-file.json
-14
View File
@@ -1,14 +0,0 @@
{
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "patch-demo-ctr-2",
"image": "redis"
}
]
}
}
}
}
docs/tasks/run-application/run-replicated-stateful-application.md
+1 -2
View File
@@ -1,6 +1,5 @@
---
approvers:
- bprashanth
- enisoc
- erictune
- foxish
@@ -225,7 +224,7 @@ by running a temporary container with the `mysql:5.7` image and running the
`mysql` client binary.
```shell
kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\
kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\
mysql -h mysql-0.mysql <<EOF
CREATE DATABASE test;
CREATE TABLE test.messages (message VARCHAR(250));
docs/tasks/run-application/run-single-instance-stateful-application.md
+10 -67
View File
@@ -24,48 +24,13 @@ application is MySQL.
* {% include task-tutorial-prereqs.md %}
* For data persistence we will create a Persistent Volume that
references a disk in your
environment. See
[here](/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes) for
the types of environments supported. This Tutorial will demonstrate
`GCEPersistentDisk` but any type will work. `GCEPersistentDisk`
volumes only work on Google Compute Engine.
* {% include default-storage-class-prereqs.md %}
{% endcapture %}
{% capture lessoncontent %}
## Set up a disk in your environment
You can use any type of persistent volume for your stateful app. See
[Types of Persistent Volumes](/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes)
for a list of supported environment disks. For Google Compute Engine, run:
```
gcloud compute disks create --size=20GB mysql-disk
```
Next create a PersistentVolume that points to the `mysql-disk`
disk just created. Here is a configuration file for a PersistentVolume
that points to the Compute Engine disk above:
{% include code.html language="yaml" file="gce-volume.yaml" ghlink="/docs/tasks/run-application/gce-volume.yaml" %}
Notice that the `pdName: mysql-disk` line matches the name of the disk
in the Compute Engine environment. See the
[Persistent Volumes](/docs/concepts/storage/persistent-volumes/)
for details on writing a PersistentVolume configuration file for other
environments.
Create the persistent volume:
```
kubectl create -f https://k8s.io/docs/tasks/run-application/gce-volume.yaml
```
## Deploy MySQL
You can run a stateful application by creating a Kubernetes Deployment
@@ -74,8 +39,8 @@ PersistentVolumeClaim. For example, this YAML file describes a
Deployment that runs MySQL and references the PersistentVolumeClaim. The file
defines a volume mount for /var/lib/mysql, and then creates a
PersistentVolumeClaim that looks for a 20G volume. This claim is
satisfied by any volume that meets the requirements, in this case, the
volume created above.
satisfied by any existing volume that meets the requirements,
or by a dynamic provisioner.
Note: The password is defined in the config yaml, and this is insecure. See
[Kubernetes Secrets](/docs/concepts/configuration/secret/)
@@ -134,28 +99,6 @@ for a secure solution.
NAME READY STATUS RESTARTS AGE
mysql-63082529-2z3ki 1/1 Running 0 3m
1. Inspect the Persistent Volume:
kubectl describe pv mysql-pv
Name: mysql-pv
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller=yes
StorageClass:
Status: Bound
Claim: default/mysql-pv-claim
Reclaim Policy: Retain
Access Modes: RWO
Capacity: 20Gi
Message:
Source:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mysql-disk
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
1. Inspect the PersistentVolumeClaim:
kubectl describe pvc mysql-pv-claim
@@ -183,7 +126,7 @@ behind a Service and you don't intend to increase the number of Pods.
Run a MySQL client to connect to the server:
```
kubectl run -it --rm --image=mysql:5.6 mysql-client -- mysql -h <pod-ip> -p <password>
kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
```
This command creates a new Pod in the cluster running a MySQL client
@@ -220,14 +163,14 @@ Delete the deployed objects by name:
```
kubectl delete deployment,svc mysql
kubectl delete pvc mysql-pv-claim
kubectl delete pv mysql-pv
```
Also, if you are using Compute Engine disks:
```
gcloud compute disks delete mysql-disk
```
If you manually provisioned a PersistentVolume, you also need to manually
delete it, as well as release the underlying resource.
If you used a dynamic provisioner, it automatically deletes the
PersistentVolume when it sees that you deleted the PersistentVolumeClaim.
Some dynamic provisioners (such as those for EBS and PD) also release the
underlying resource upon deleting the PersistentVolume.
{% endcapture %}
docs/tasks/run-application/run-stateless-application-deployment.md
+1 -1
View File
@@ -38,7 +38,7 @@ a Deployment that runs the nginx:1.7.9 Docker image:
1. Create a Deployment based on the YAML file:
kubectl create -f https://k8s.io/docs/tasks/run-application/deployment.yaml
kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml
1. Display information about the Deployment:
docs/tasks/run-application/update-api-object-kubectl-patch.md
+6 -6
View File
@@ -81,14 +81,14 @@ The output shows that the PodSpec in the Deployment has two Containers:
```shell
containers:
- image: nginx
imagePullPolicy: Always
name: patch-demo-ctr
...
- image: redis
imagePullPolicy: Always
name: patch-demo-ctr-2
...
- image: nginx
imagePullPolicy: Always
name: patch-demo-ctr
...
```
View the Pods associated with your patched Deployment:
@@ -118,10 +118,10 @@ The output shows that the Pod has two Containers: one running nginx and one runn
```
containers:
- image: nginx
...
- image: redis
...
- image: nginx
...
```
### Notes on the strategic merge patch
docs/tasks/run-application/upgrade-pet-set-to-stateful-set.md
+14 -2
View File
@@ -30,7 +30,12 @@ Here are some notable changes:
* **StatefulSet is the new PetSet**: PetSet is no longer available in Kubernetes release 1.5 or later. It becomes beta StatefulSet. To understand why the name was changed, see this [discussion thread](https://github.com/kubernetes/kubernetes/issues/27430).
* **StatefulSet guards against split brain**: StatefulSets guarantee at most one Pod for a given ordinal index can be running anywhere in a cluster, to guard against split brain scenarios with distributed applications. *TODO: Link to doc about fencing.*
* **Flipped debug annotation behavior**: The default value of the debug annotation (`pod.alpha.kubernetes.io/initialized`) is now `true`. The absence of this annotation will pause PetSet operations, but will NOT pause StatefulSet operations. In most cases, you no longer need this annotation in your StatefulSet manifests.
* **Flipped debug annotation behavior**:
The default value of the debug annotation (`pod.alpha.kubernetes.io/initialized`) is `true` in 1.5 through 1.7.
The annotation is completely ignored in 1.8 and above, which always behave as if it were `true`.
The absence of this annotation will pause PetSet operations, but will NOT pause StatefulSet operations.
In most cases, you no longer need this annotation in your StatefulSet manifests.
## Upgrading from PetSets to StatefulSets
@@ -66,7 +71,14 @@ Now, for every PetSet manifest you have, prepare a corresponding StatefulSet man
1. Change `apiVersion` from `apps/v1alpha1` to `apps/v1beta1`.
2. Change `kind` from `PetSet` to `StatefulSet`.
3. If you have the debug hook annotation `pod.alpha.kubernetes.io/initialized` set to `true`, you can remove it because it's redundant. If you don't have this annotation, you should add one, with the value set to `false`, to pause StatefulSets operations.
3. If you have the debug hook annotation `pod.alpha.kubernetes.io/initialized` set to `true`,
you can remove it because it's redundant.
If you don't have this annotation or have it set to `false`,
be aware that StatefulSet operations might resume after the upgrade.
If you are upgrading to 1.6 or 1.7, you can set the annotation explicitly to `false` to maintain
the paused behavior.
If you are upgrading to 1.8 or above, there's no longer any debug annotation to pause StatefulSets.
It's recommended that you keep both PetSet manifests and StatefulSet manifests, so that you can safely roll back and recreate your PetSets,
if you decide not to upgrade your cluster.
docs/tasks/tls/certificate-rotation.md
+81
View File
@@ -0,0 +1,81 @@
---
approvers:
- jcbsmpsn
- mikedanese
title: Certificate Rotation
---
{% capture overview %}
This page shows how to enable and configure certificate rotation for the kubelet.
{% endcapture %}
{% capture prerequisites %}
* Kubernetes version 1.8.0 or later is required
* Kubelet certificate rotation is beta in 1.8.0 which means it may change without notice.
{% endcapture %}
{% capture steps %}
## Overview
The kubelet uses certificates for authenticating to the Kubernetes API. By
default, these certificates are issued with one year expiration so that they do
not need to be renewed too frequently.
Kubernetes 1.8 contains [kubelet certificate
rotation](/docs/tasks/administer-cluster/certificate-rotation/), a beta feature
that will automatically generate a new key and request a new certificate from
the Kubernetes API as the current certificate approaches expiration. Once the
new certificate is available, it will be used for authenticating connections to
the Kubernetes API.
## Enabling client certificate rotation
The `kubelet` process accepts an argument `--rotate-certificates` that controls
if the kubelet will automatically request a new certificate as the expiration of
the certificate currently in use approaches. Since certificate rotation is a
beta feature, the feature flag must also be enabled with
`--feature-gates=RotateKubeletClientCertificate=true`.
The `kube-controller-manager` process accepts an argument
`--experimental-cluster-signing-duration` that controls how long certificates
will be issued for.
## Understanding the certificate rotation configuration
When a kubelet starts up, if it is configured to bootstrap (using the
`--bootstrap-kubeconfig` flag), it will use its initial certificate to connect
to the Kubernetes API and issue a certificate signing request. You can view the
status of certificate signing requests using:
```sh
kubectl get csr
```
Initially a certificate signing request from the kubelet on a node will have a
status of `Pending`. If the certificate signing requests meets specific
criteria, it will be auto approved by the controller manager, then it will have
a status of `Approved`. Next, the controller manager will sign a certificate,
issued for the duration specified by the
`--experimental-cluster-signing-duration` parameter, and the signed certificate
will be attached to the certificate signing requests.
The kubelet will retrieve the signed certificate from the Kubernetes API and
write that to disk, in the location specified by `--cert-dir`. Then the kubelet
will use the new certificate to connect to the Kubernetes API.
As the expiration of the signed certificate approaches, the kubelet will
automatically issue a new certificate signing request, using the Kubernetes
API. Again, the controller manager will automatically approve the certificate
request and attach a signed certificate to the certificate signing request. The
kubelet will retrieve the new signed certificate from the Kubernetes API and
write that to disk. Then it will update the connections it has to the
Kubernetes API to reconnect using the new certificate.
{% endcapture %}
{% include templates/task.md %}