whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into release-1.6

Browse Source
This commit is contained in:
Devin Donnelly
2017-03-20 13:51:51 -07:00
parent d4383a4b6a 7cc984d66b
commit 4956e43fb1
131 changed files with 7647 additions and 6853 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/user-guide/config-best-practices.md
+2 -114
View File
@@ -1,119 +1,7 @@
---
assignees:
- mikedanese
title: Best Practices for Configuration
---
This document is meant to highlight and consolidate in one place configuration best practices that are introduced throughout the user-guide and getting-started documentation and examples. This is a living document so if you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR.
## General Config Tips
- When defining configurations, specify the latest stable API version (currently v1).
- Configuration files should be stored in version control before being pushed to the cluster. This allows a configuration to be quickly rolled back if needed, and will aid with cluster re-creation and restoration if necessary.
- Write your configuration files using YAML rather than JSON. They can be used interchangeably in almost all scenarios, but YAML tends to be more user-friendly for config.
- Group related objects together in a single file where this makes sense. This format is often easier to manage than separate files. See the [guestbook-all-in-one.yaml](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/all-in-one/guestbook-all-in-one.yaml) file as an example of this syntax.
(Note also that many `kubectl` commands can be called on a directory, and so you can also call
`kubectl create` on a directory of config files— see below for more detail).
- Don't specify default values unnecessarily, in order to simplify and minimize configs, and to
reduce error. For example, omit the selector and labels in a `ReplicationController` if you want
them to be the same as the labels in its `podTemplate`, since those fields are populated from the
`podTemplate` labels by default. See the [guestbook app's](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) .yaml files for some [examples](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/frontend-deployment.yaml) of this.
- Put an object description in an annotation to allow better introspection.
## "Naked" Pods vs Replication Controllers and Jobs
- If there is a viable alternative to naked pods (i.e., pods not bound to a [replication controller
](/docs/user-guide/replication-controller)), go with the alternative. Naked pods will not be rescheduled in the
event of node failure.
Replication controllers are almost always preferable to creating pods, except for some explicit
[`restartPolicy: Never`](/docs/user-guide/pod-states/#restartpolicy) scenarios. A
[Job](/docs/user-guide/jobs/) object (currently in Beta), may also be appropriate.
## Services
- It's typically best to create a [service](/docs/user-guide/services/) before corresponding [replication
controllers](/docs/user-guide/replication-controller/), so that the scheduler can spread the pods comprising the
service. You can also create a replication controller without specifying replicas (this will set
replicas=1), create a service, then scale up the replication controller. This can be useful in
ensuring that one replica works before creating lots of them.
- Don't use `hostPort` (which specifies the port number to expose on the host) unless absolutely
necessary, e.g., for a node daemon. When you bind a Pod to a `hostPort`, there are a limited
number of places that pod can be scheduled, due to port conflicts— you can only schedule as many
such Pods as there are nodes in your Kubernetes cluster.
If you only need access to the port for debugging purposes, you can use the [kubectl proxy and apiserver proxy](/docs/user-guide/connecting-to-applications-proxy/) or [kubectl port-forward](/docs/user-guide/connecting-to-applications-port-forward/).
You can use a [Service](/docs/user-guide/services/) object for external service access.
If you do need to expose a pod's port on the host machine, consider using a [NodePort](/docs/user-guide/services/#type-nodeport) service before resorting to `hostPort`.
- Avoid using `hostNetwork`, for the same reasons as `hostPort`.
- Use _headless services_ for easy service discovery when you don't need kube-proxy load balancing.
See [headless services](/docs/user-guide/services/#headless-services).
## Using Labels
- Define and use [labels](/docs/user-guide/labels/) that identify __semantic attributes__ of your application or
deployment. For example, instead of attaching a label to a set of pods to explicitly represent
some service (e.g., `service: myservice`), or explicitly representing the replication
controller managing the pods (e.g., `controller: mycontroller`), attach labels that identify
semantic attributes, such as `{ app: myapp, tier: frontend, phase: test, deployment: v3 }`. This
will let you select the object groups appropriate to the context— e.g., a service for all "tier:
frontend" pods, or all "test" phase components of app "myapp". See the
[guestbook](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/) app for an example of this approach.
A service can be made to span multiple deployments, such as is done across [rolling updates](/docs/user-guide/kubectl/kubectl_rolling-update/), by simply omitting release-specific labels from its selector, rather than updating a service's selector to match the replication controller's selector fully.
- To facilitate rolling updates, include version info in replication controller names, e.g. as a
suffix to the name. It is useful to set a 'version' label as well. The rolling update creates a
new controller as opposed to modifying the existing controller. So, there will be issues with
version-agnostic controller names. See the [documentation](/docs/user-guide/kubectl/kubectl_rolling-update/) on
the rolling-update command for more detail.
Note that the [Deployment](/docs/user-guide/deployments/) object obviates the need to manage replication
controller 'version names'. A desired state of an object is described by a Deployment, and if
changes to that spec are _applied_, the deployment controller changes the actual state to the
desired state at a controlled rate. (Deployment objects are currently part of the [`extensions`
API Group](/docs/api/#api-groups).)
- You can manipulate labels for debugging. Because Kubernetes replication controllers and services
match to pods using labels, this allows you to remove a pod from being considered by a
controller, or served traffic by a service, by removing the relevant selector labels. If you
remove the labels of an existing pod, its controller will create a new pod to take its place.
This is a useful way to debug a previously "live" pod in a quarantine environment. See the
[`kubectl label`](/docs/user-guide/kubectl/kubectl_label/) command.
## Container Images
- The [default container image pull policy](/docs/user-guide/images/) is `IfNotPresent`, which causes the
[Kubelet](/docs/admin/kubelet/) to not pull an image if it already exists. If you would like to
always force a pull, you must specify a pull image policy of `Always` in your .yaml file
(`imagePullPolicy: Always`) or specify a `:latest` tag on your image.
That is, if you're specifying an image with other than the `:latest` tag, e.g. `myimage:v1`, and
there is an image update to that same tag, the Kubelet won't pull the updated image. You can
address this by ensuring that any updates to an image bump the image tag as well (e.g.
`myimage:v2`), and ensuring that your configs point to the correct version.
**Note:** you should avoid using `:latest` tag when deploying containers in production, because this makes it hard
to track which version of the image is running and hard to roll back.
## Using kubectl
- Use `kubectl create -f <directory>` where possible. This looks for config objects in all `.yaml`, `.yml`, and `.json` files in `<directory>` and passes them to `create`.
- Use `kubectl delete` rather than `stop`. `Delete` has a superset of the functionality of `stop`, and `stop` is deprecated.
- Use kubectl bulk operations (via files and/or labels) for get and delete. See [label selectors](/docs/user-guide/labels/#label-selectors) and [using labels effectively](/docs/user-guide/managing-deployments/#using-labels-effectively).
- Use `kubectl run` and `expose` to quickly create and expose single container Deployments. See the [quick start guide](/docs/user-guide/quick-start/) for an example.
{% include user-guide-content-moved.md %}
[Configuration Overview](/docs/concepts/configuration/overview/)
docs/user-guide/cron-jobs.md
+3 -3
View File
@@ -11,7 +11,7 @@ title: Cron Jobs
## What is a cron job?
A _Cron Job_ manages time based [Jobs](/docs/user-guide/jobs/), namely:
A _Cron Job_ manages time based [Jobs](/docs/concepts/jobs/run-to-completion-finite-workloads/), namely:
* Once at a specified point in time
* Repeatedly at a specified point in time
@@ -159,8 +159,8 @@ string, e.g. `0 * * * *` or `@hourly`, as schedule time of its jobs to be create
### Job Template
The `.spec.jobTemplate` is another required field of the `.spec`. It is a job template. It has exactly the same schema
as a [Job](/docs/user-guide/jobs), except it is nested and does not have an `apiVersion` or `kind`, see
[Writing a Job Spec](/docs/user-guide/jobs/#writing-a-job-spec).
as a [Job](/docs/concepts/jobs/run-to-completion-finite-workloads/), except it is nested and does not have an `apiVersion` or `kind`, see
[Writing a Job Spec](/docs/concepts/jobs/run-to-completion-finite-workloads/#writing-a-job-spec).
### Starting Deadline Seconds
docs/user-guide/deployments.md
+2 -2
View File
@@ -719,7 +719,7 @@ to a previous revision, or even pause it if you need to apply multiple tweaks in
### Canary Deployment
If you want to roll out releases to a subset of users or servers using the Deployment, you can create multiple Deployments, one for each release,
following the canary pattern described in [managing resources](/docs/user-guide/managing-deployments/#canary-deployments).
following the canary pattern described in [managing resources](/docs/concepts/cluster-administration/manage-deployment/#canary-deployments).
## Writing a Deployment Spec
@@ -779,7 +779,7 @@ All existing Pods are killed before new ones are created when
#### Rolling Update Deployment
The Deployment updates Pods in a [rolling update](/docs/user-guide/update-demo/) fashion
The Deployment updates Pods in a [rolling update](/docs/tasks/run-application/rolling-update-replication-controller/) fashion
when `.spec.strategy.type==RollingUpdate`.
You can specify `maxUnavailable` and `maxSurge` to control
the rolling update process.
docs/user-guide/federation/configmap.md
+2 -82
View File
@@ -2,86 +2,6 @@
title: Federated ConfigMap
---
This guide explains how to use ConfigMaps in a Federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you).
Other tutorials, such as Kelsey Hightower's
[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation),
might also help you create a Federated Kubernetes cluster.
You should also have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [ConfigMaps](/docs/user-guide/ConfigMaps/) in particular.
## Overview
Federated ConfigMaps are very similar to the traditional [Kubernetes
ConfigMaps](/docs/user-guide/configmap/) and provide the same functionality.
Creating them in the federation control plane ensures that they are synchronized
across all the clusters in federation.
## Creating a Federated ConfigMap
The API for Federated ConfigMap is 100% compatible with the
API for traditional Kubernetes ConfigMap. You can create a ConfigMap by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f myconfigmap.yaml
```
The `--context=federation-cluster` flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a Federated ConfigMap is created, the federation control plane will create
a matching ConfigMap in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get configmap myconfigmap
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone.
These ConfigMaps in underlying clusters will match the Federated ConfigMap.
## Updating a Federated ConfigMap
You can update a Federated ConfigMap as you would update a Kubernetes
ConfigMap; however, for a Federated ConfigMap, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
The federation control plane ensures that whenever the Federated ConfigMap is
updated, it updates the corresponding ConfigMaps in all underlying clusters to
match it.
## Deleting a Federated ConfigMap
You can delete a Federated ConfigMap as you would delete a Kubernetes
ConfigMap; however, for a Federated ConfigMap, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete configmap
```
Note that at this point, deleting a Federated ConfigMap will not delete the
corresponding ConfigMaps from underlying clusters.
You must delete the underlying ConfigMaps manually.
We intend to fix this in the future.
[Federated ConfigMap](/docs/tasks/administer-federation/configmap/)
docs/user-guide/federation/daemonsets.md
+2 -78
View File
@@ -2,82 +2,6 @@
title: Federated DaemonSet
---
This guide explains how to use DaemonSets in a federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you).
Other tutorials, such as Kelsey Hightower's
[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation),
might also help you create a Federated Kubernetes cluster.
You should also have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and DaemonSets in particular.
## Overview
DaemonSets in federation control plane ("Federated Daemonsets" in
this guide) are very similar to the traditional [Kubernetes
DaemonSets](/docs/user-guide/DaemonSets/) and provide the same functionality.
Creating them in the federation control plane ensures that they are synchronized
across all the clusters in federation.
## Creating a Federated Daemonset
The API for Federated Daemonset is 100% compatible with the
API for traditional Kubernetes DaemonSet. You can create a DaemonSet by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f mydaemonset.yaml
```
The `--context=federation-cluster` flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a Federated Daemonset is created, the federation control plane will create
a matching DaemonSet in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get daemonset mydaemonset
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone.
These DaemonSets in underlying clusters will match the Federated Daemonset.
## Updating a Federated Daemonset
You can update a Federated Daemonset as you would update a Kubernetes
DaemonSet; however, for a Federated Daemonset, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
The federation control plane ensures that whenever the Federated Daemonset is
updated, it updates the corresponding DaemonSets in all underlying clusters to
match it.
## Deleting a Federated Daemonset
You can delete a Federated Daemonset as you would delete a Kubernetes
DaemonSet; however, for a Federated Daemonset, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete daemonset mydaemonset
```
[Federated DaemonSet](/docs/tasks/administer-federation/daemonset/)
docs/user-guide/federation/deployment.md
+2 -103
View File
@@ -2,107 +2,6 @@
title: Federated Deployment
---
This guide explains how to use Deployments in the Federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you).
Other tutorials, such as Kelsey Hightower's
[Federated Kubernetes Tutorial](https://github.com/kelseyhightower/kubernetes-cluster-federation),
might also help you create a Federated Kubernetes cluster.
You should also have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [Deployment](/docs/user-guide/deployments) in particular.
## Overview
Deployments in federation control plane (referred to as "Federated Deployments" in
this guide) are very similar to the traditional [Kubernetes
Deployment](/docs/user-guide/deployments/), and provide the same functionality.
Creating them in the federation control plane ensures that the desired number of
replicas exist across the registered clusters.
**As of Kubernetes version 1.5, Federated Deployment is an Alpha feature. The core
functionality of Deployment is present, but some features
(such as full rollout compatibility) are still in development.**
## Creating a Federated Deployment
The API for Federated Deployment is compatible with the
API for traditional Kubernetes Deployment. You can create a Deployment by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f mydeployment.yaml
```
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a Federated Deployment is created, the federation control plane will create
a Deployment in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get deployment mydep
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone.
These Deployments in underlying clusters will match the federation Deployment
_except_ in the number of replicas and revision-related annotations.
Federation control plane ensures that the
sum of replicas in each cluster combined matches the desired number of replicas in the
Federated Deployment.
### Spreading Replicas in Underlying Clusters
By default, replicas are spread equally in all the underlying clusters. For ex:
if you have 3 registered clusters and you create a Federated Deployment with
`spec.replicas = 9`, then each Deployment in the 3 clusters will have
`spec.replicas=3`.
To modify the number of replicas in each cluster, you can specify
[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go)
as an annotation with key `federation.kubernetes.io/deployment-preferences`
on Federated Deployment.
## Updating a Federated Deployment
You can update a Federated Deployment as you would update a Kubernetes
Deployment; however, for a Federated Deployment, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
The federation control plane ensures that whenever the Federated Deployment is
updated, it updates the corresponding Deployments in all underlying clusters to
match it. So if the rolling update strategy was chosen then the underlying
cluster will do the rolling update independently and `maxSurge` and `maxUnavailable`
will apply only to individual clusters. This behavior may change in the future.
If your update includes a change in number of replicas, the federation
control plane will change the number of replicas in underlying clusters to
ensure that their sum remains equal to the number of desired replicas in
Federated Deployment.
## Deleting a Federated Deployment
You can delete a Federated Deployment as you would delete a Kubernetes
Deployment; however, for a Federated Deployment, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete deployment mydep
```
[Federated Deployment](/docs/tasks/administer-federation/deployment/)
docs/user-guide/federation/events.md
+2 -35
View File
@@ -2,39 +2,6 @@
title: Federated Events
---
This guide explains how to use events in federation control plane to help in debugging.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you). Other tutorials, for example
[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation)
by Kelsey Hightower, are also available to help you.
You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general.
## Overview
Events in federation control plane (referred to as "federation events" in
this guide) are very similar to the traditional Kubernetes
Events providing the same functionality.
Federation Events are stored only in federation control plane and are not passed on to the underlying Kubernetes clusters.
Federation controllers create events as they process API resources to surface to the
user, the state that they are in.
You can get all events from federation apiserver by running:
```shell
kubectl --context=federation-cluster get events
```
The standard kubectl get, update, delete commands will all work.
[Federated Evemts](/docs/tasks/administer-federation/events/)
docs/user-guide/federation/federated-ingress.md
+2 -351
View File
@@ -2,355 +2,6 @@
title: Federated Ingress
---
This guide explains how to use Kubernetes Federated Ingress to deploy
a common HTTP(S) virtual IP load balancer across a federated service running in
multiple Kubernetes clusters. As of v1.4, clusters hosted in Google
Cloud (both GKE and GCE, or both) are supported. This makes it
easy to deploy a service that reliably serves HTTP(S) traffic
originating from web clients around the globe on a single, static IP
address. Low
network latency, high fault tolerance and easy administration are
ensured through intelligent request routing and automatic replica
relocation (using [Federated ReplicaSets](docs/user-guide/federation/federated-replicaset.md)).
Clients are automatically routed, via the shortest network path, to
the cluster closest to them with available capacity (despite the fact
that all clients use exactly the same static IP address). The load balancer
automatically checks the health of the pods comprising the service,
and avoids sending requests to unresponsive or slow pods (or entire
unresponsive clusters).
{% include user-guide-content-moved.md %}
Federated Ingress is released as an alpha feature, and supports Google Cloud Platform (GKE,
GCE and hybrid scenarios involving both) in Kubernetes v1.4. Work is under way to support other cloud
providers such as AWS, and other hybrid cloud scenarios (e.g. services
spanning private on-premise as well as public cloud Kubernetes
clusters). We welcome your feedback.
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you). Other tutorials, for example
[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation)
by Kelsey Hightower, are also available to help you.
You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general, and [Ingress](/docs/user-guide/ingress/) in particular.
## Overview
Federated Ingresses are created in much that same way as traditional
[Kubernetes Ingresses](/docs/user-guide/ingress/): by making an API
call which specifies the desired properties of your logical ingress point. In the
case of Federated Ingress, this API call is directed to the
Federation API endpoint, rather than a Kubernetes cluster API
endpoint. The API for Federated Ingress is 100% compatible with the
API for traditional Kubernetes Services.
Once created, the Federated Ingress automatically:
1. creates matching Kubernetes Ingress objects in every cluster
underlying your Cluster Federation,
2. ensures that all of these in-cluster ingress objects share the same
logical global L7 (i.e. HTTP(S)) load balancer and IP address.
3. monitors the health and capacity of the service "shards" (i.e. your
pods) behind this ingress in each cluster
4. ensures that all client connections are routed to an appropriate
healthy backend service endpoint at all times, even in the event of
pod, cluster,
availability zone or regional outages.
Note that in the case of Google Cloud, the logical L7 load balancer is
not a single physical device (which would present both a single point
of failure, and a single global network routing choke point), but
rather a
[truly global, highly available load balancing managed service](https://cloud.google.com/load-balancing/),
globally reachable via a single, static IP address.
Clients inside your federated Kubernetes clusters (i.e. Pods) will be
automatically routed to the cluster-local shard of the Federated Service
backing the Ingress in their
cluster if it exists and is healthy, or the closest healthy shard in a
different cluster if it does not. Note that this involves a network
trip to the HTTP(s) load balancer, which resides outside your local
Kubernetes cluster but inside the same GCP region.
## Creating a federated ingress
You can create a federated ingress in any of the usual ways, for example using kubectl:
``` shell
kubectl --context=federation-cluster create -f myingress.yaml
```
For example ingress YAML configurations, see the [Ingress User Guide](/docs/user-guide/ingress/)
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation API endpoint, with the appropriate
credentials. If you have not yet configured such a context, visit the
[federation admin guide](/docs/admin/federation/) or one of the
[administration tutorials](https://github.com/kelseyhightower/kubernetes-cluster-federation)
to find out how to do so.
As described above, the Federated Ingress will automatically create
and maintain matching Kubernetes ingresses in all of the clusters
underlying your federation. These cluster-specific ingresses (and
their associated ingress controllers) configure and manage the load
balancing and health checking infrastructure that ensures that traffic
is load balanced to each cluster appropriately.
You can verify this by checking in each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get ingress myingress
NAME HOSTS ADDRESS PORTS AGE
myingress * 130.211.5.194 80, 443 1m
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone. The name and
namespace of the underlying ingress will automatically match those of
the Federated Ingress that you created above (and if you happen to
have had ingresses of the same name and namespace already existing in
any of those clusters, they will be automatically adopted by the
Federation and updated to conform with the specification of your
Federated Ingress - either way, the end result will be the same).
The status of your Federated Ingress will automatically reflect the
real-time status of the underlying Kubernetes ingresses, for example:
``` shell
$kubectl --context=federation-cluster describe ingress myingress
Name: myingress
Namespace: default
Address: 130.211.5.194
TLS:
tls-secret terminates
Rules:
Host Path Backends
---- ---- --------
* * echoheaders-https:80 (10.152.1.3:8080,10.152.2.4:8080)
Annotations:
https-target-proxy: k8s-tps-default-myingress--ff1107f83ed600c0
target-proxy: k8s-tp-default-myingress--ff1107f83ed600c0
url-map: k8s-um-default-myingress--ff1107f83ed600c0
backends: {"k8s-be-30301--ff1107f83ed600c0":"Unknown"}
forwarding-rule: k8s-fw-default-myingress--ff1107f83ed600c0
https-forwarding-rule: k8s-fws-default-myingress--ff1107f83ed600c0
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
3m 3m 1 {loadbalancer-controller } Normal ADD default/myingress
2m 2m 1 {loadbalancer-controller } Normal CREATE ip: 130.211.5.194
```
Note that:
1. the address of your Federated Ingress
corresponds with the address of all of the
underlying Kubernetes ingresses (once these have been allocated - this
may take up to a few minutes).
2. we have not yet provisioned any backend Pods to receive
the network traffic directed to this ingress (i.e. 'Service
Endpoints' behind the service backing the Ingress), so the Federated Ingress does not yet consider these to
be healthy shards and will not direct traffic to any of these clusters.
3. the federation control system will
automatically reconfigure the load balancer controllers in all of the
clusters in your federation to make them consistent, and allow
them to share global load balancers. But this reconfiguration can
only complete successfully if there are no pre-existing Ingresses in
those clusters (this is a safety feature to prevent accidental
breakage of existing ingresses). So to ensure that your federated
ingresses function correctly, either start with new, empty clusters, or make
sure that you delete (and recreate if necessary) all pre-existing
Ingresses in the clusters comprising your federation.
#Adding backend services and pods
To render the underlying ingress shards healthy, we need to add
backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but
the easiest is to create a [Federated Service](federated-services.md) and
[Federated Replicaset](federated-replicasets.md). Details of how those
work are covered in the aforementioned user guides - here we'll simply use them, to
create appropriately labelled pods and services in the 13 underlying clusters of
our federation:
``` shell
kubectl --context=federation-cluster create -f services/nginx.yaml
```
``` shell
kubectl --context=federation-cluster create -f myreplicaset.yaml
```
Note that in order for your federated ingress to work correctly on
Google Cloud, the node ports of all of the underlying cluster-local
services need to be identical. If you're using a federated service
this is easy to do. Simply pick a node port that is not already
being used in any of your clusters, and add that to the spec of your
federated service. If you do not specify a node port for your
federated service, each cluster will choose it's own node port for
its cluster-local shard of the service, and these will probably end
up being different, which is not what you want.
You can verify this by checking in each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get services nginx
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx 10.63.250.98 104.199.136.89 80/TCP 9m
```
## Hybrid cloud capabilities
Federations of Kubernetes Clusters can include clusters running in
different cloud providers (e.g. Google Cloud, AWS), and on-premises
(e.g. on OpenStack). However, in Kubernetes v1.4, Federated Ingress is only
supported across Google Cloud clusters. In future versions we intend
to support hybrid cloud Ingress-based deployments.
## Discovering a federated ingress
Ingress objects (in both plain Kubernets clusters, and in federations
of clusters) expose one or more IP addresses (via
the Status.Loadbalancer.Ingress field) that remains static for the lifetime
of the Ingress object (in future, automatically managed DNS names
might also be added). All clients (whether internal to your cluster,
or on the external network or internet) should connect to one of these IP
or DNS addresses. As mentioned above, all client requests are automatically
routed, via the shortest network path, to a healthy pod in the
closest cluster to the origin of the request. So for example, HTTP(S)
requests from internet
users in Europe will be routed directly to the closest cluster in
Europe that has available capacity. If there are no such clusters in
Europe, the request will be routed to the next closest cluster
(typically in the U.S.).
## Handling failures of backend pods and whole clusters
Ingresses are backed by Services, which are typically (but not always)
backed by one or more ReplicaSets. For Federated Ingresses, it is
common practise to use the federated variants of Services and
ReplicaSets (see [Federated Services](federated-services.md) and
[Federated ReplicaSets](federated-replicasets.md)) for this purpose, as
described above.
In particular, Federated ReplicaSets ensure that the desired number of
pods are kept running in each cluster, even in the event of node
failures. In the event of entire cluster or availability zone
failures, Federated ReplicaSets automatically place additional
replacas in the other available clusters in the federation to accommodate the
traffic which was previously being served by the now unavailable
cluster. While the Federated ReplicaSet ensures that sufficient replicas are
kept running, the Federated Ingress ensures that user traffic is
automatically redirected away from the failed cluster to other
available clusters.
## Known issue
GCE L7 load balancer back-ends and health checks are known to "flap"; this is due
to conflicting firewall rules in the federation's underlying clusters, which might override one another. To work around this problem, you can
install the firewall rules manually to expose the targets of all the
underlying clusters in your federation for each Federated Ingress
object. This way, the health checks can consistently pass and the GCE L7 load balancer
can remain stable. You install the rules using the
[`gcloud`](https://cloud.google.com/sdk/gcloud/) command line tool,
[Google Cloud Console](https://console.cloud.google.com) or the
[Google Compute Engine APIs](https://cloud.google.com/compute/docs/reference/latest/).
You can install these rules using
[`gcloud`](https://cloud.google.com/sdk/gcloud/) as follows:
```shell
gcloud compute firewall-rules create <firewall-rule-name> \
--source-ranges 130.211.0.0/22 --allow [<service-nodeports>] \
--target-tags [<target-tags>] \
--network <network-name>
```
where:
1. `firewall-rule-name` can be any name.
2. `[<service-nodeports>]` is the comma separated list of node ports corresponding to the services that back the Federated Ingress.
3. [<target-tags>] is the comma separated list of the target tags assigned to the nodes in a Kubernetes cluster.
4. <network-name> is the name of the network where the firewall rule must be installed.
Example:
```shell
gcloud compute firewall-rules create my-federated-ingress-firewall-rule \
--source-ranges 130.211.0.0/22 --allow tcp:30301, tcp:30061, tcp:34564 \
--target-tags my-cluster-1-minion, my-cluster-2-minion \
--network default
```
## Troubleshooting
#### I cannot connect to my cluster federation API
Check that your
1. Client (typically kubectl) is correctly configured (including API endpoints and login credentials), and
2. Cluster Federation API server is running and network-reachable.
See the [federation admin guide](/docs/admin/federation/) to learn
how to bring up a cluster federation correctly (or have your cluster administrator do this for you), and how to correctly configure your client.
#### I can create a federated ingress/service/replicaset successfully against the cluster federation API, but no matching ingresses/services/replicasets are created in my underlying clusters
Check that:
1. Your clusters are correctly registered in the Cluster Federation API (`kubectl describe clusters`)
2. Your clusters are all 'Active'. This means that the cluster
Federation system was able to connect and authenticate against the
clusters' endpoints. If not, consult the event logs of the federation-controller-manager pod to ascertain what the failure might be. (`kubectl --namespace=federation logs $(kubectl get pods --namespace=federation -l module=federation-controller-manager -oname`)
3. That the login credentials provided to the Cluster Federation API
for the clusters have the correct authorization and quota to create
ingresses/services/replicasets in the relevant namespace in the
clusters. Again you should see associated error messages providing
more detail in the above event log file if this is not the case.
4. Whether any other error is preventing the service creation
operation from succeeding (look for `ingress-controller`,
`service-controller` or `replicaset-controller`,
errors in the output of `kubectl logs federation-controller-manager --namespace federation`).
#### I can create a federated ingress successfully, but request load is not correctly distributed across the underlying clusters
Check that:
1. the services underlying your federated ingress in each cluster have
identical node ports. See [above](#creating_a_federated_ingress) for further explanation.
2. the load balancer controllers in each of your clusters are of the
correct type ("GLBC") and have been correctly reconfigured by the
federation control plane to share a global GCE load balancer (this
should happen automatically). If they of the correct type, and
have been correctly reconfigured, the UID data item in the GLBC
configmap in each cluster will be identical across all clusters.
See
[the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid)
for further details.
If this is not the case, check the logs of your federation
controller manager to determine why this automated reconfiguration
might be failing.
3. no ingresses have been manually created in any of your clusters before the above
reconfiguration of the load balancer controller completed
successfully. Ingresses created before the reconfiguration of
your GLBC will interfere with the behavior of your federated
ingresses created after the reconfiguration (see
[the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid)
for further information. To remedy this,
delete any ingresses created before the cluster joined the
federation (and had it's GLBC reconfigured), and recreate them if
necessary.
#### This troubleshooting guide did not help me solve my problem
Please use one of our [support channels](http://kubernetes.io/docs/troubleshooting/) to seek assistance.
## For more information
* [Federation proposal](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/proposals/federation.md) details use cases that motivated this work.
[Federated Ingress](/docs/tasks/administer-federation/ingress/)
docs/user-guide/federation/index.md
+2 -132
View File
@@ -2,136 +2,6 @@
title: Federation User Guide
---
This guide explains why and how to manage multiple Kubernetes clusters using
federation.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Why federation
Federation makes it easy to manage multiple clusters. It does so by providing 2
major building blocks:
* Sync resources across clusters: Federation provides the ability to keep
resources in multiple clusters in sync. This can be used, for example, to
ensure that the same deployment exists in multiple clusters.
* Cross cluster discovery: It provides the ability to auto-configure DNS
servers and load balancers with backends from all clusters. This can be used,
for example, to ensure that a global VIP or DNS record can be used to access
backends from multiple clusters.
Some other use cases that federation enables are:
* High Availability: By spreading load across clusters and auto configuring DNS
servers and load balancers, federation minimises the impact of cluster
failure.
* Avoiding provider lock-in: By making it easier to migrate applications across
clusters, federation prevents cluster provider lock-in.
Federation is not helpful unless you have multiple clusters. Some of the reasons
why you might want multiple clusters are:
* Low latency: Having clusters in multiple regions minimises latency by serving
users from the cluster that is closest to them.
* Fault isolation: It might be better to have multiple small clusters rather
than a single large cluster for fault isolation (for example: multiple
clusters in different availability zones of a cloud provider).
[Multi cluster guide](/docs/admin/multi-cluster) has more details on this.
* Scalability: There are scalability limits to a single kubernetes cluster (this
should not be the case for most users. For more details:
[Kubernetes Scaling and Performance Goals](https://github.com/kubernetes/community/blob/master/sig-scalability/goals.md)).
* Hybrid cloud: You can have multiple clusters on different cloud providers or
on-premises data centers.
### Caveats
While there are a lot of attractive use cases for federation, there are also
some caveats.
* Increased network bandwidth and cost: The federation control plane watches all
clusters to ensure that the current state is as expected. This can lead to
significant network cost if the clusters are running in different regions on
a cloud provider or on different cloud providers.
* Reduced cross cluster isolation: A bug in the federation control plane can
impact all clusters. This is mitigated by keeping the logic in federation
control plane to a minimum. It mostly delegates to the control plane in
kubernetes clusters whenever it can. The design and implementation also errs
on the side of safety and avoiding multicluster outage.
* Maturity: The federation project is relatively new and is not very mature.
Not all resources are available and many are still alpha. [Issue
38893](https://github.com/kubernetes/kubernetes/issues/38893) ennumerates
known issues with the system that the team is busy solving.
## Setup
To be able to federate multiple clusters, we first need to setup a federation
control plane.
Follow the [setup guide](/docs/admin/federation/) to setup the
federation control plane.
## Hybrid cloud capabilities
Federations of Kubernetes Clusters can include clusters running in
different cloud providers (e.g. Google Cloud, AWS), and on-premises
(e.g. on OpenStack). Simply create all of the clusters that you
require, in the appropriate cloud providers and/or locations, and
register each cluster's API endpoint and credentials with your
Federation API Server (See the
[federation admin guide](/docs/admin/federation/) for details).
Thereafter, your API resources can span different clusters
and cloud providers.
## API resources
Once we have the control plane setup, we can start creating federation API
resources.
The following guides explain some of the resources in detail:
* [ConfigMap](https://kubernetes.io/docs/user-guide/federation/configmap/)
* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/)
* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/)
* [Events](https://kubernetes.io/docs/user-guide/federation/events/)
* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/)
* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/)
* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/)
* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/)
* [Services](https://kubernetes.io/docs/user-guide/federation/federated-services/)
[API reference docs](/docs/federation/api-reference/) lists all the
resources supported by federation apiserver.
## Cascading deletion
Kubernetes version 1.5 includes support for cascading deletion of federated
resources. With cascading deletion, when you delete a resource from the
federation control plane, the corresponding resources in all underlying clusters
are also deleted.
To enable cascading deletion, set the option
`DeleteOptions.orphanDependents=false` when you delete a resource from the
federation control plane.
The following Federated resources are affected by cascading deletion:
* [Ingress](https://kubernetes.io/docs/user-guide/federation/federated-ingress/)
* [Namespaces](https://kubernetes.io/docs/user-guide/federation/namespaces/)
* [ReplicaSets](https://kubernetes.io/docs/user-guide/federation/replicasets/)
* [Secrets](https://kubernetes.io/docs/user-guide/federation/secrets/)
* [Deployment](https://kubernetes.io/docs/user-guide/federation/deployment/)
* [DaemonSets](https://kubernetes.io/docs/user-guide/federation/daemonsets/)
Note: By default, deleting a resource from federation control plane does not
delete the corresponding resources from underlying clusters.
## For more information
* [Federation
proposal](https://github.com/kubernetes/community/blob/{{page.githubbranch}}/contributors/design-proposals/federation.md)
* [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8)
[Federation](/docs/concepts/cluster-administration/federation.md)
docs/user-guide/federation/namespaces.md
+2 -85
View File
@@ -2,89 +2,6 @@
title: Federated Namespaces
---
This guide explains how to use namespaces in Federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you). Other tutorials, for example
[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation)
by Kelsey Hightower, are also available to help you.
You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [Namespaces](/docs/user-guide/namespaces/) in particular.
## Overview
Namespaces in federation control plane (referred to as "federated namespaces" in
this guide) are very similar to the traditional [Kubernetes
Namespaces](/docs/user-guide/namespaces/) providing the same functionality.
Creating them in the federation control plane ensures that they are synchronized
across all the clusters in federation.
## Creating a Federated Namespace
The API for Federated Namespaces is 100% compatible with the
API for traditional Kubernetes Namespaces. You can create a namespace by sending
a request to the federation apiserver.
You can do that using kubectl by running:
``` shell
kubectl --context=federation-cluster create -f myns.yaml
```
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a federated namespace is created, the federation control plane will create
a matching namespace in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get namespaces myns
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone. The name and
spec of the underlying namespace will match those of
the Federated Namespace that you created above.
## Updating a Federated Namespace
You can update a federated namespace as you would update a Kubernetes
namespace, just send the request to federation apiserver instead of sending it
to a specific Kubernetes cluster.
Federation control plan will ensure that whenever the federated namespace is
updated, it updates the corresponding namespaces in all underlying clusters to
match it.
## Deleting a Federated Namespace
You can delete a federated namespace as you would delete a Kubernetes
namespace, just send the request to federation apiserver instead of sending it
to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete ns myns
```
As in Kubernetes, deleting a federated namespace will delete all resources in that
namespace from the federation control plane.
Note that at this point, deleting a federated namespace will not delete the
corresponding namespaces and resources in those namespaces from underlying clusters.
Users are expected to delete them manually.
We intend to fix this in the future.
[Federated Namespaces](/docs/tasks/administer-federation/namespaces/)
docs/user-guide/federation/replicasets.md
+2 -100
View File
@@ -2,104 +2,6 @@
title: Federated ReplicaSets
---
This guide explains how to use replica sets in the Federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you). Other tutorials, for example
[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation)
by Kelsey Hightower, are also available to help you.
You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [ReplicaSets](/docs/user-guide/replicasets/) in particular.
## Overview
Replica Sets in federation control plane (referred to as "federated replica sets" in
this guide) are very similar to the traditional [Kubernetes
ReplicaSets](/docs/user-guide/replicasets/), and provide the same functionality.
Creating them in the federation control plane ensures that the desired number of
replicas exist across the registered clusters.
## Creating a Federated Replica Set
The API for Federated Replica Set is 100% compatible with the
API for traditional Kubernetes Replica Set. You can create a replica set by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f myrs.yaml
```
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a federated replica set is created, the federation control plane will create
a replica set in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get rs myrs
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone.
These replica sets in underlying clusters will match the federation replica set
except in the number of replicas. Federation control plane will ensure that the
sum of replicas in each cluster match the desired number of replicas in the
federation replica set.
### Spreading Replicas in Underlying Clusters
By default, replicas are spread equally in all the underlying clusters. For ex:
if you have 3 registered clusters and you create a federated replica set with
`spec.replicas = 9`, then each replica set in the 3 clusters will have
`spec.replicas=3`.
To modify the number of replicas in each cluster, you can specify
[FederatedReplicaSetPreference](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/federation/apis/federation/types.go)
as an annotation with key `federation.kubernetes.io/replica-set-preferences`
on federated replica set.
## Updating a Federated Replica Set
You can update a federated replica set as you would update a Kubernetes
replica set; however, for a federated replica set, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
The Federation control plan ensures that whenever the federated replica set is
updated, it updates the corresponding replica sets in all underlying clusters to
match it.
If your update includes a change in number of replicas, the federation
control plane will change the number of replicas in underlying clusters to
ensure that their sum remains equal to the number of desired replicas in
federated replica set.
## Deleting a Federated Replica Set
You can delete a federated replica set as you would delete a Kubernetes
replica set; however, for a federated replica set, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete rs myrs
```
Note that at this point, deleting a federated replica set will not delete the
corresponding replica sets from underlying clusters.
You must delete the underlying Replica Sets manually.
We intend to fix this in the future.
[Federated ReplicaSets](/docs/tasks/administer-federation/replicaset/)
docs/user-guide/federation/secrets.md
+2 -82
View File
@@ -2,86 +2,6 @@
title: Federated Secrets
---
This guide explains how to use secrets in Federation control plane.
{% include user-guide-content-moved.md %}
* TOC
{:toc}
## Prerequisites
This guide assumes that you have a running Kubernetes Cluster
Federation installation. If not, then head over to the
[federation admin guide](/docs/admin/federation/) to learn how to
bring up a cluster federation (or have your cluster administrator do
this for you). Other tutorials, for example
[this one](https://github.com/kelseyhightower/kubernetes-cluster-federation)
by Kelsey Hightower, are also available to help you.
You are also expected to have a basic
[working knowledge of Kubernetes](/docs/getting-started-guides/) in
general and [Secrets](/docs/user-guide/secrets/) in particular.
## Overview
Secrets in federation control plane (referred to as "federated secrets" in
this guide) are very similar to the traditional [Kubernetes
Secrets](/docs/user-guide/secrets/) providing the same functionality.
Creating them in the federation control plane ensures that they are synchronized
across all the clusters in federation.
## Creating a Federated Secret
The API for Federated Secret is 100% compatible with the
API for traditional Kubernetes Secret. You can create a secret by sending
a request to the federation apiserver.
You can do that using [kubectl](/docs/user-guide/kubectl/) by running:
``` shell
kubectl --context=federation-cluster create -f mysecret.yaml
```
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation apiserver instead of sending it to a Kubernetes
cluster.
Once a federated secret is created, the federation control plane will create
a matching secret in all underlying Kubernetes clusters.
You can verify this by checking each of the underlying clusters, for example:
``` shell
kubectl --context=gce-asia-east1a get secret mysecret
```
The above assumes that you have a context named 'gce-asia-east1a'
configured in your client for your cluster in that zone.
These secrets in underlying clusters will match the federated secret.
## Updating a Federated Secret
You can update a federated secret as you would update a Kubernetes
secret; however, for a federated secret, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
The Federation control plan ensures that whenever the federated secret is
updated, it updates the corresponding secrets in all underlying clusters to
match it.
## Deleting a Federated Secret
You can delete a federated secret as you would delete a Kubernetes
secret; however, for a federated secret, you must send the request to
the federation apiserver instead of sending it to a specific Kubernetes cluster.
For example, you can do that using kubectl by running:
```shell
kubectl --context=federation-cluster delete secret mysecret
```
Note that at this point, deleting a federated secret will not delete the
corresponding secrets from underlying clusters.
You must delete the underlying secrets manually.
We intend to fix this in the future.
[Federated Secrets](/docs/tasks/administer-federation/secret/)
docs/user-guide/horizontal-pod-autoscaling/index.md
+1 -1
View File
@@ -69,7 +69,7 @@ The detailed documentation of `kubectl autoscale` can be found [here](/docs/user
## Autoscaling during rolling update
Currently in Kubernetes, it is possible to perform a [rolling update](/docs/user-guide/rolling-updates/) by managing replication controllers directly,
Currently in Kubernetes, it is possible to perform a [rolling update](/docs/tasks/run-application/rolling-update-replication-controller/) by managing replication controllers directly,
or by using the deployment object, which manages the underlying replication controllers for you.
Horizontal Pod Autoscaler only supports the latter approach: the Horizontal Pod Autoscaler is bound to the deployment object,
it sets the size for the deployment object, and the deployment is responsible for setting sizes of underlying replication controllers.
docs/user-guide/images.md
+1 -1
View File
@@ -25,7 +25,7 @@ your image.
If you did not specify tag of your image, it will be assumed as `:latest`, with
pull image policy of `Always` correspondingly.
Note that you should avoid using `:latest` tag, see [Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information.
Note that you should avoid using `:latest` tag, see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information.
## Using a Private Registry
docs/user-guide/index.md
+5 -3
View File
@@ -4,6 +4,8 @@ assignees:
title: User Guide
---
{% include user-guide-migration-notice.md %}
The Kubernetes **Guides** can help you work with various aspects of the Kubernetes system.
* The Kubernetes [User Guide](#user-guide-internal) can help you run programs and services on an existing Kubernetes cluster.
@@ -19,7 +21,7 @@ The following topics in the Kubernetes User Guide can help you run applications
1. [Deploying continuously running applications](/docs/user-guide/deploying-applications/)
1. [Connecting applications: exposing applications to clients and users](/docs/user-guide/connecting-applications/)
1. [Working with containers in production](/docs/user-guide/production-pods/)
1. [Managing deployments](/docs/user-guide/managing-deployments/)
1. [Managing deployments](/docs/concepts/cluster-administration/manage-deployment/)
1. [Application introspection and debugging](/docs/user-guide/introspection-and-debugging/)
1. [Using the Kubernetes web user interface](/docs/user-guide/ui/)
1. [Logging](/docs/user-guide/logging/overview/)
@@ -83,8 +85,8 @@ Pods and containers
* [Downward API: accessing system configuration from a pod](/docs/user-guide/downward-api/)
* [Images and registries](/docs/user-guide/images/)
* [Migrating from docker-cli to kubectl](/docs/user-guide/docker-cli-to-kubectl/)
* [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/)
* [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/)
* [Assign pods to selected nodes](/docs/user-guide/node-selection/)
* [Perform a rolling update on a running group of pods](/docs/user-guide/update-demo/)
* [Perform a rolling update on a running group of pods](/docs/tasks/run-application/rolling-update-replication-controller/)
[Developer Guide]: https://github.com/kubernetes/community/blob/master/contributors/devel/README.md
docs/user-guide/jobs.md
+1 -375
View File
@@ -1,379 +1,5 @@
---
assignees:
- erictune
- soltysh
title: Jobs
---
* TOC
{:toc}
## What is a Job?
A _job_ creates one or more pods and ensures that a specified number of them successfully terminate.
As pods successfully complete, the _job_ tracks the successful completions. When a specified number
of successful completions is reached, the job itself is complete. Deleting a Job will cleanup the
pods it created.
A simple case is to create one Job object in order to reliably run one Pod to completion.
The Job object will start a new Pod if the first pod fails or is deleted (for example
due to a node hardware failure or a node reboot).
A Job can also be used to run multiple pods in parallel.
## Running an example Job
Here is an example Job config. It computes π to 2000 places and prints it out.
It takes around 10s to complete.
{% include code.html language="yaml" file="job.yaml" ghlink="/docs/user-guide/job.yaml" %}
Run the example job by downloading the example file and then running this command:
```shell
$ kubectl create -f ./job.yaml
job "pi" created
```
Check on the status of the job using this command:
```shell
$ kubectl describe jobs/pi
Name: pi
Namespace: default
Image(s): perl
Selector: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495
Parallelism: 1
Completions: 1
Start Time: Tue, 07 Jun 2016 10:56:16 +0200
Labels: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495,job-name=pi
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: pi-dtn4q
```
To view completed pods of a job, use `kubectl get pods --show-all`. The `--show-all` will show completed pods too.
To list all the pods that belong to a job in a machine readable form, you can use a command like this:
```shell
$ pods=$(kubectl get pods --show-all --selector=job-name=pi --output=jsonpath={.items..metadata.name})
echo $pods
pi-aiw0a
```
Here, the selector is the same as the selector for the job. The `--output=jsonpath` option specifies an expression
that just gets the name from each pod in the returned list.
View the standard output of one of the pods:
```shell
$ kubectl logs $pods
3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901
```
## Writing a Job Spec
As with all other Kubernetes config, a Job needs `apiVersion`, `kind`, and `metadata` fields. For
general information about working with config files, see [here](/docs/user-guide/simple-yaml),
[here](/docs/user-guide/configuring-containers), and [here](/docs/user-guide/working-with-resources).
A Job also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status).
### Pod Template
The `.spec.template` is the only required field of the `.spec`.
The `.spec.template` is a [pod template](/docs/user-guide/replication-controller/#pod-template). It has exactly
the same schema as a [pod](/docs/user-guide/pods), except it is nested and does not have an `apiVersion` or
`kind`.
In addition to required fields for a Pod, a pod template in a job must specify appropriate
labels (see [pod selector](#pod-selector)) and an appropriate restart policy.
Only a [`RestartPolicy`](/docs/user-guide/pod-states/#restartpolicy) equal to `Never` or `OnFailure` is allowed.
### Pod Selector
The `.spec.selector` field is optional. In almost all cases you should not specify it.
See section [specifying your own pod selector](#specifying-your-own-pod-selector).
### Parallel Jobs
There are three main types of jobs:
1. Non-parallel Jobs
- normally only one pod is started, unless the pod fails.
- job is complete as soon as Pod terminates successfully.
1. Parallel Jobs with a *fixed completion count*:
- specify a non-zero positive value for `.spec.completions`
- the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`.
- **not implemented yet:** each pod passed a different index in the range 1 to `.spec.completions`.
1. Parallel Jobs with a *work queue*:
- do not specify `.spec.completions`, default to `.spec.Parallelism`
- the pods must coordinate with themselves or an external service to determine what each should work on
- each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done.
- when _any_ pod terminates with success, no new pods are created.
- once at least one pod has terminated with success and all pods are terminated, then the job is completed with success.
- once any pod has exited with success, no other pod should still be doing any work or writing any output. They should all be
in the process of exiting.
For a Non-parallel job, you can leave both `.spec.completions` and `.spec.parallelism` unset. When both are
unset, both are defaulted to 1.
For a Fixed Completion Count job, you should set `.spec.completions` to the number of completions needed.
You can set `.spec.parallelism`, or leave it unset and it will default to 1.
For a Work Queue Job, you must leave `.spec.completions` unset, and set `.spec.parallelism` to
a non-negative integer.
For more information about how to make use of the different types of job, see the [job patterns](#job-patterns) section.
#### Controlling Parallelism
The requested parallelism (`.spec.parallelism`) can be set to any non-negative value.
If it is unspecified, it defaults to 1.
If it is specified as 0, then the Job is effectively paused until it is increased.
A job can be scaled up using the `kubectl scale` command. For example, the following
command sets `.spec.parallelism` of a job called `myjob` to 10:
```shell
$ kubectl scale --replicas=$N jobs/myjob
job "myjob" scaled
```
You can also use the `scale` subresource of the Job resource.
Actual parallelism (number of pods running at any instant) may be more or less than requested
parallelism, for a variety or reasons:
- For Fixed Completion Count jobs, the actual number of pods running in parallel will not exceed the number of
remaining completions. Higher values of `.spec.parallelism` are effectively ignored.
- For work queue jobs, no new pods are started after any pod has succeeded -- remaining pods are allowed to complete, however.
- If the controller has not had time to react.
- If the controller failed to create pods for any reason (lack of ResourceQuota, lack of permission, etc.),
then there may be fewer pods than requested.
- The controller may throttle new pod creation due to excessive previous pod failures in the same Job.
- When a pod is gracefully shutdown, it takes time to stop.
## Handling Pod and Container Failures
A Container in a Pod may fail for a number of reasons, such as because the process in it exited with
a non-zero exit code, or the Container was killed for exceeding a memory limit, etc. If this
happens, and the `.spec.template.spec.restartPolicy = "OnFailure"`, then the Pod stays
on the node, but the Container is re-run. Therefore, your program needs to handle the case when it is
restarted locally, or else specify `.spec.template.spec.restartPolicy = "Never"`.
See [pods-states](/docs/user-guide/pod-states) for more information on `restartPolicy`.
An entire Pod can also fail, for a number of reasons, such as when the pod is kicked off the node
(node is upgraded, rebooted, deleted, etc.), or if a container of the Pod fails and the
`.spec.template.spec.restartPolicy = "Never"`. When a Pod fails, then the Job controller
starts a new Pod. Therefore, your program needs to handle the case when it is restarted in a new
pod. In particular, it needs to handle temporary files, locks, incomplete output and the like
caused by previous runs.
Note that even if you specify `.spec.parallelism = 1` and `.spec.completions = 1` and
`.spec.template.spec.restartPolicy = "Never"`, the same program may
sometimes be started twice.
If you do specify `.spec.parallelism` and `.spec.completions` both greater than 1, then there may be
multiple pods running at once. Therefore, your pods must also be tolerant of concurrency.
## Job Termination and Cleanup
When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated,
they don't show up with `kubectl get pods`, but they will show up with `kubectl get pods -a`. Keeping them around
allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output.
The job object also remains after it is completed so that you can view its status. It is up to the user to delete
old jobs after noting their status. Delete the job with `kubectl` (e.g. `kubectl delete jobs/pi` or `kubectl delete -f ./job.yaml`). When you delete the job using `kubectl`, all the pods it created are deleted too.
If a Job's pods are failing repeatedly, the Job will keep creating new pods forever, by default.
Retrying forever can be a useful pattern. If an external dependency of the Job's
pods is missing (for example an input file on a networked storage volume is not present), then the
Job will keep trying Pods, and when you later resolve the external dependency (for example, creating
the missing file) the Job will then complete without any further action.
However, if you prefer not to retry forever, you can set a deadline on the job. Do this by setting the
`spec.activeDeadlineSeconds` field of the job to a number of seconds. The job will have status with
`reason: DeadlineExceeded`. No more pods will be created, and existing pods will be deleted.
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: pi-with-timeout
spec:
activeDeadlineSeconds: 100
template:
metadata:
name: pi
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
```
Note that both the Job Spec and the Pod Template Spec within the Job have a field with the same name.
Set the one on the Job.
## Job Patterns
The Job object can be used to support reliable parallel execution of Pods. The Job object is not
designed to support closely-communicating parallel processes, as commonly found in scientific
computing. It does support parallel processing of a set of independent but related *work items*.
These might be emails to be sent, frames to be rendered, files to be transcoded, ranges of keys in a
NoSQL database to scan, and so on.
In a complex system, there may be multiple different sets of work items. Here we are just
considering one set of work items that the user wants to manage together &mdash; a *batch job*.
There are several different patterns for parallel computation, each with strengths and weaknesses.
The tradeoffs are:
- One Job object for each work item, vs. a single Job object for all work items. The latter is
better for large numbers of work items. The former creates some overhead for the user and for the
system to manage large numbers of Job objects. Also, with the latter, the resource usage of the job
(number of concurrently running pods) can be easily adjusted using the `kubectl scale` command.
- Number of pods created equals number of work items, vs. each pod can process multiple work items.
The former typically requires less modification to existing code and containers. The latter
is better for large numbers of work items, for similar reasons to the previous bullet.
- Several approaches use a work queue. This requires running a queue service,
and modifications to the existing program or container to make it use the work queue.
Other approaches are easier to adapt to an existing containerised application.
The tradeoffs are summarized here, with columns 2 to 4 corresponding to the above tradeoffs.
The pattern names are also links to examples and more detailed description.
| Pattern | Single Job object | Fewer pods than work items? | Use app unmodified? | Works in Kube 1.1? |
| -------------------------------------------------------------------- |:-----------------:|:---------------------------:|:-------------------:|:-------------------:|
| [Job Template Expansion](/docs/user-guide/jobs/expansions) | | | ✓ | ✓ |
| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | ✓ | | sometimes | ✓ |
| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | ✓ | ✓ | | ✓ |
| Single Job with Static Work Assignment | ✓ | | ✓ | |
When you specify completions with `.spec.completions`, each Pod created by the Job controller
has an identical [`spec`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status). This means that
all pods will have the same command line and the same
image, the same volumes, and (almost) the same environment variables. These patterns
are different ways to arrange for pods to work on different things.
This table shows the required settings for `.spec.parallelism` and `.spec.completions` for each of the patterns.
Here, `W` is the number of work items.
| Pattern | `.spec.completions` | `.spec.parallelism` |
| -------------------------------------------------------------------- |:-------------------:|:--------------------:|
| [Job Template Expansion](/docs/user-guide/jobs/expansions/) | 1 | should be 1 |
| [Queue with Pod Per Work Item](/docs/user-guide/jobs/work-queue-1/) | W | any |
| [Queue with Variable Pod Count](/docs/user-guide/jobs/work-queue-2/) | 1 | any |
| Single Job with Static Work Assignment | W | any |
## Advanced Usage
### Specifying your own pod selector
Normally, when you create a job object, you do not specify `spec.selector`.
The system defaulting logic adds this field when the job is created.
It picks a selector value that will not overlap with any other jobs.
However, in some cases, you might need to override this automatically set selector.
To do this, you can specify the `spec.selector` of the job.
Be very careful when doing this. If you specify a label selector which is not
unique to the pods of that job, and which matches unrelated pods, then pods of the unrelated
job may be deleted, or this job may count other pods as completing it, or one or both
of the jobs may refuse to create pods or run to completion. If a non-unique selector is
chosen, then other controllers (e.g. ReplicationController) and their pods may behave
in unpredicatable ways too. Kubernetes will not stop you from making a mistake when
specifying `spec.selector`.
Here is an example of a case when you might want to use this feature.
Say job `old` is already running. You want existing pods
to keep running, but you want the rest of the pods it creates
to use a different pod template and for the job to have a new name.
You cannot update the job because these fields are not updatable.
Therefore, you delete job `old` but leave its pods
running, using `kubectl delete jobs/old-one --cascade=false`.
Before deleting it, you make a note of what selector it uses:
```
kind: Job
metadata:
name: old
...
spec:
selector:
matchLabels:
job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002
...
```
Then you create a new job with name `new` and you explicitly specify the same selector.
Since the existing pods have label `job-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002`,
they are controlled by job `new` as well.
You need to specify `manualSelector: true` in the new job since you are not using
the selector that the system normally generates for you automatically.
```
kind: Job
metadata:
name: new
...
spec:
manualSelector: true
selector:
matchLabels:
job-uid: a8f3d00d-c6d2-11e5-9f87-42010af00002
...
```
The new Job itself will have a different uid from `a8f3d00d-c6d2-11e5-9f87-42010af00002`. Setting
`manualSelector: true` tells the system to that you know what you are doing and to allow this
mismatch.
## Alternatives
### Bare Pods
When the node that a pod is running on reboots or fails, the pod is terminated
and will not be restarted. However, a Job will create new pods to replace terminated ones.
For this reason, we recommend that you use a job rather than a bare pod, even if your application
requires only a single pod.
### Replication Controller
Jobs are complementary to [Replication Controllers](/docs/user-guide/replication-controller).
A Replication Controller manages pods which are not expected to terminate (e.g. web servers), and a Job
manages pods that are expected to terminate (e.g. batch jobs).
As discussed in [life of a pod](/docs/user-guide/pod-states), `Job` is *only* appropriate for pods with
`RestartPolicy` equal to `OnFailure` or `Never`. (Note: If `RestartPolicy` is not set, the default
value is `Always`.)
### Single Job starts Controller Pod
Another pattern is for a single Job to create a pod which then creates other pods, acting as a sort
of custom controller for those pods. This allows the most flexibility, but may be somewhat
complicated to get started with and offers less integration with Kubernetes.
One example of this pattern would be a Job which starts a Pod which runs a script that in turn
starts a Spark master controller (see [spark example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/spark/README.md)), runs a spark
driver, and then cleans up.
An advantage of this approach is that the overall process gets the completion guarantee of a Job
object, but complete control over what pods are created and how work is assigned to them.
## Cron Jobs
Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [cron job documents](http://kubernetes.io/docs/user-guide/cron-jobs/)
{% include user-guide-content-moved.md %}
docs/user-guide/jobs/expansions/index.md
+2 -189
View File
@@ -2,194 +2,7 @@
title: Parallel Processing using Expansions
---
* TOC
{:toc}
{% include user-guide-content-moved.md %}
# Example: Multiple Job Objects from Template Expansion
[Parallel Processing Expansion](/docs/tasks/job/parallel-processing-expansion/)
In this example, we will run multiple Kubernetes Jobs created from
a common template. You may want to be familiar with the basic,
non-parallel, use of [Jobs](/docs/user-guide/jobs) first.
## Basic Template Expansion
First, download the following template of a job to a file called `job.yaml.txt`
{% include code.html language="yaml" file="job.yaml.txt" ghlink="/docs/user-guide/job/expansions/job.yaml.txt" %}
Unlike a *pod template*, our *job template* is not a Kubernetes API type. It is just
a yaml representation of a Job object that has some placeholders that need to be filled
in before it can be used. The `$ITEM` syntax is not meaningful to Kubernetes.
In this example, the only processing the container does is to `echo` a string and sleep for a bit.
In a real use case, the processing would be some substantial computation, such as rendering a frame
of a movie, or processing a range of rows in a database. The "$ITEM" parameter would specify for
example, the frame number or the row range.
This Job and its Pod template have a label: `jobgroup=jobexample`. There is nothing special
to the system about this label. This label
makes it convenient to operate on all the jobs in this group at once.
We also put the same label on the pod template so that we can check on all Pods of these Jobs
with a single command.
After the job is created, the system will add more labels that distinguish one Job's pods
from another Job's pods.
Note that the label key `jobgroup` is not special to Kubernetes. You can pick your own label scheme.
Next, expand the template into multiple files, one for each item to be processed.
```shell
# Expand files into a temporary directory
mkdir ./jobs
for i in apple banana cherry
do
cat job.yaml.txt | sed "s/\$ITEM/$i/" > ./jobs/job-$i.yaml
done
```
Check if it worked:
```shell
$ ls jobs/
job-apple.yaml
job-banana.yaml
job-cherry.yaml
```
Here, we used `sed` to replace the string `$ITEM` with the loop variable.
You could use any type of template language (jinja2, erb) or write a program
to generate the Job objects.
Next, create all the jobs with one kubectl command:
```shell
$ kubectl create -f ./jobs
job "process-item-apple" created
job "process-item-banana" created
job "process-item-cherry" created
```
Now, check on the jobs:
```shell
$ kubectl get jobs -l jobgroup=jobexample
JOB CONTAINER(S) IMAGE(S) SELECTOR SUCCESSFUL
process-item-apple c busybox app in (jobexample),item in (apple) 1
process-item-banana c busybox app in (jobexample),item in (banana) 1
process-item-cherry c busybox app in (jobexample),item in (cherry) 1
```
Here we use the `-l` option to select all jobs that are part of this
group of jobs. (There might be other unrelated jobs in the system that we
do not care to see.)
We can check on the pods as well using the same label selector:
```shell
$ kubectl get pods -l jobgroup=jobexample --show-all
NAME READY STATUS RESTARTS AGE
process-item-apple-kixwv 0/1 Completed 0 4m
process-item-banana-wrsf7 0/1 Completed 0 4m
process-item-cherry-dnfu9 0/1 Completed 0 4m
```
There is not a single command to check on the output of all jobs at once,
but looping over all the pods is pretty easy:
```shell
$ for p in $(kubectl get pods -l jobgroup=jobexample -o name)
do
kubectl logs $p
done
Processing item apple
Processing item banana
Processing item cherry
```
## Multiple Template Parameters
In the first example, each instance of the template had one parameter, and that parameter was also
used as a label. However label keys are limited in [what characters they can
contain](/docs/user-guide/labels/#syntax-and-character-set).
This slightly more complex example uses the jinja2 template language to generate our objects.
We will use a one-line python script to convert the template to a file.
First, copy and paste the following template of a Job object, into a file called `job.yaml.jinja2`:
```liquid{% raw %}
{%- set params = [{ "name": "apple", "url": "http://www.orangepippin.com/apples", },
{ "name": "banana", "url": "https://en.wikipedia.org/wiki/Banana", },
{ "name": "raspberry", "url": "https://www.raspberrypi.org/" }]
%}
{%- for p in params %}
{%- set name = p["name"] %}
{%- set url = p["url"] %}
apiVersion: batch/v1
kind: Job
metadata:
name: jobexample-{{ name }}
labels:
jobgroup: jobexample
spec:
template:
name: jobexample
labels:
jobgroup: jobexample
spec:
containers:
- name: c
image: busybox
command: ["sh", "-c", "echo Processing URL {{ url }} && sleep 5"]
restartPolicy: Never
---
{%- endfor %}
{% endraw %}
```
The above template defines parameters for each job object using a list of
python dicts (lines 1-4). Then a for loop emits one job yaml object
for each set of parameters (remaining lines).
We take advantage of the fact that multiple yaml documents can be concatenated
with the `---` separator (second to last line).
.) We can pipe the output directly to kubectl to
create the objects.
You will need the jinja2 package if you do not already have it: `pip install --user jinja2`.
Now, use this one-line python program to expand the template:
```shell
alias render_template='python -c "from jinja2 import Template; import sys; print(Template(sys.stdin.read()).render());"'
```
The output can be saved to a file, like this:
```shell
cat job.yaml.jinja2 | render_template > jobs.yaml
```
or sent directly to kubectl, like this:
```shell
cat job.yaml.jinja2 | render_template | kubectl create -f -
```
## Alternatives
If you have a large number of job objects, you may find that:
- even using labels, managing so many Job objects is cumbersome.
- You exceed resource quota when creating all the Jobs at once,
and do not want to wait to create them incrementally.
- You need a way to easily scale the number of pods running
concurrently. One reason would be to avoid using too many
compute resources. Another would be to limit the number of
concurrent requests to a shared resource, such as a database,
used by all the pods in the job.
- very large numbers of jobs created at once overload the
Kubernetes apiserver, controller, or scheduler.
In this case, you can consider one of the
other [job patterns](/docs/user-guide/jobs/#job-patterns).
docs/user-guide/jobs/work-queue-1/index.md
+2 -279
View File
@@ -2,283 +2,6 @@
title: Coarse Parallel Processing using a Work Queue
---
* TOC
{:toc}
{% include user-guide-content-moved.md %}
# Example: Job with Work Queue with Pod Per Work Item
In this example, we will run a Kubernetes Job with multiple parallel
worker processes. You may want to be familiar with the basic,
non-parallel, use of [Job](/docs/user-guide/jobs) first.
In this example, as each pod is created, it picks up one unit of work
from a task queue, completes it, deletes it from the queue, and exits.
Here is an overview of the steps in this example:
1. **Start a message queue service.** In this example, we use RabbitMQ, but you could use another
one. In practice you would set up a message queue service once and reuse it for many jobs.
1. **Create a queue, and fill it with messages.** Each message represents one task to be done. In
this example, a message is just an integer that we will do a lengthy computation on.
1. **Start a Job that works on tasks from the queue**. The Job starts several pods. Each pod takes
one task from the message queue, processes it, and repeats until the end of the queue is reached.
## Starting a message queue service
This example uses RabbitMQ, but it should be easy to adapt to another AMQP-type message service.
In practice you could set up a message queue service once in a
cluster and reuse it for many jobs, as well as for long-running services.
Start RabbitMQ as follows:
```shell
$ kubectl create -f examples/celery-rabbitmq/rabbitmq-service.yaml
service "rabbitmq-service" created
$ kubectl create -f examples/celery-rabbitmq/rabbitmq-controller.yaml
replicationController "rabbitmq-controller" created
```
We will only use the rabbitmq part from the [celery-rabbitmq example](https://github.com/kubernetes/kubernetes/tree/release-1.3/examples/celery-rabbitmq).
## Testing the message queue service
Now, we can experiment with accessing the message queue. We will
create a temporary interactive pod, install some tools on it,
and experiment with queues.
First create a temporary interactive Pod.
```shell
# Create a temporary interactive container
$ kubectl run -i --tty temp --image ubuntu:14.04
Waiting for pod default/temp-loe07 to be running, status is Pending, pod ready: false
... [ previous line repeats several times .. hit return when it stops ] ...
```
Note that your pod name and command prompt will be different.
Next install the `amqp-tools` so we can work with message queues.
```shell
# Install some tools
root@temp-loe07:/# apt-get update
.... [ lots of output ] ....
root@temp-loe07:/# apt-get install -y curl ca-certificates amqp-tools python dnsutils
.... [ lots of output ] ....
```
Later, we will make a docker image that includes these packages.
Next, we will check that we can discover the rabbitmq service:
```
# Note the rabitmq-service has a DNS name, provided by Kubernetes:
root@temp-loe07:/# nslookup rabbitmq-service
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: rabbitmq-service.default.svc.cluster.local
Address: 10.0.147.152
# Your address will vary.
```
If Kube-DNS is not setup correctly, the previous step may not work for you.
You can also find the service IP in an env var:
```
# env | grep RABBIT | grep HOST
RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152
# Your address will vary.
```
Next we will verify we can create a queue, and publish and consume messages.
```shell
# In the next line, rabbitmq-service is the hostname where the rabbitmq-service
# can be reached. 5672 is the standard port for rabbitmq.
root@temp-loe07:/# export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672
# If you could not resolve "rabbitmq-service" in the previous step,
# then use this command instead:
# root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
# Now create a queue:
root@temp-loe07:/# /usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d
foo
# Publish one message to it:
root@temp-loe07:/# /usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b Hello
# And get it back.
root@temp-loe07:/# /usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo
Hello
root@temp-loe07:/#
```
In the last command, the `amqp-consume` tool takes one message (`-c 1`)
from the queue, and passes that message to the standard input of an arbitrary command. In this case, the program `cat` is just printing
out what it gets on the standard input, and the echo is just to add a carriage
return so the example is readable.
## Filling the Queue with tasks
Now lets fill the queue with some "tasks". In our example, our tasks are just strings to be
printed.
In a practice, the content of the messages might be:
- names of files to that need to be processed
- extra flags to the program
- ranges of keys in a database table
- configuration parameters to a simulation
- frame numbers of a scene to be rendered
In practice, if there is large data that is needed in a read-only mode by all pods
of the Job, you will typically put that in a shared file system like NFS and mount
that readonly on all the pods, or the program in the pod will natively read data from
a cluster file system like HDFS.
For our example, we will create the queue and fill it using the amqp command line tools.
In practice, you might write a program to fill the queue using an amqp client library.
```shell
$ /usr/bin/amqp-declare-queue --url=$BROKER_URL -q job1 -d
job1
$ for f in apple banana cherry date fig grape lemon melon
do
/usr/bin/amqp-publish --url=$BROKER_URL -r job1 -p -b $f
done
```
So, we filled the queue with 8 messages.
## Create an Image
Now we are ready to create an image that we will run as a job.
We will use the `amqp-consume` utility to read the message
from the queue and run our actual program. Here is a very simple
example program:
{% include code.html language="python" file="worker.py" ghlink="/docs/user-guide/jobs/work-queue-1/worker.py" %}
Now, build an image. If you are working in the source
tree, then change directory to `examples/job/work-queue-1`.
Otherwise, make a temporary directory, change to it,
download the [Dockerfile](Dockerfile?raw=true),
and [worker.py](worker.py?raw=true). In either case,
build the image with this command: `
```shell
$ docker build -t job-wq-1 .
```
For the [Docker Hub](https://hub.docker.com/), tag your app image with
your username and push to the Hub with the below commands. Replace
`<username>` with your Hub username.
```shell
docker tag job-wq-1 <username>/job-wq-1
docker push <username>/job-wq-1
```
If you are using [Google Container
Registry](https://cloud.google.com/tools/container-registry/), tag
your app image with your project ID, and push to GCR. Replace
`<project>` with your project ID.
```shell
docker tag job-wq-1 gcr.io/<project>/job-wq-1
gcloud docker push gcr.io/<project>/job-wq-1
```
## Defining a Job
Here is a job definition. You'll need to make a copy of the Job and edit the
image to match the name you used, and call it `./job.yaml`.
{% include code.html language="yaml" file="job.yaml" ghlink="/docs/user-guide/jobs/work-queue-1/job.yaml" %}
In this example, each pod works on one item from the queue and then exits.
So, the completion count of the Job corresponds to the number of work items
done. So we set, `.spec.completions: 8` for the example, since we put 8 items in the queue.
## Running the Job
So, now run the Job:
```shell
kubectl create -f ./job.yaml
```
Now wait a bit, then check on the job.
```shell
$ kubectl describe jobs/job-wq-1
Name: job-wq-1
Namespace: default
Image(s): gcr.io/causal-jigsaw-637/job-wq-1
Selector: app in (job-wq-1)
Parallelism: 2
Completions: 8
Labels: app=job-wq-1
Pods Statuses: 0 Running / 8 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
27s 27s 1 {job } SuccessfulCreate Created pod: job-wq-1-hcobb
27s 27s 1 {job } SuccessfulCreate Created pod: job-wq-1-weytj
27s 27s 1 {job } SuccessfulCreate Created pod: job-wq-1-qaam5
27s 27s 1 {job } SuccessfulCreate Created pod: job-wq-1-b67sr
26s 26s 1 {job } SuccessfulCreate Created pod: job-wq-1-xe5hj
15s 15s 1 {job } SuccessfulCreate Created pod: job-wq-1-w2zqe
14s 14s 1 {job } SuccessfulCreate Created pod: job-wq-1-d6ppa
14s 14s 1 {job } SuccessfulCreate Created pod: job-wq-1-p17e0
```
All our pods succeeded. Yay.
## Alternatives
This approach has the advantage that you
do not need to modify your "worker" program to be aware that there is a work queue.
It does require that you run a message queue service.
If running a queue service is inconvenient, you may
want to consider one of the other [job patterns](/docs/user-guide/jobs/#job-patterns).
This approach creates a pod for every work item. If your work items only take a few seconds,
though, creating a Pod for every work item may add a lot of overhead. Consider another
[example](/docs/user-guide/jobs/work-queue-2/), that executes multiple work items per Pod.
In this example, we used use the `amqp-consume` utility to read the message
from the queue and run our actual program. This has the advantage that you
do not need to modify your program to be aware of the queue.
A [different example](/docs/user-guide/jobs/work-queue-2/), shows how to
communicate with the work queue using a client library.
## Caveats
If the number of completions is set to less than the number of items in the queue, then
not all items will be processed.
If the number of completions is set to more than the number of items in the queue,
then the Job will not appear to be completed, even though all items in the queue
have been processed. It will start additional pods which will block waiting
for a message.
There is an unlikely race with this pattern. If the container is killed in between the time
that the message is acknowledged by the amqp-consume command and the time that the container
exits with success, or if the node crashes before the kubelet is able to post the success of the pod
back to the api-server, then the Job will not appear to be complete, even though all items
in the queue have been processed.
[Coarse Parallel Processing Using a Work Queue](/docs/tasks/job/work-queue-1/)
docs/user-guide/jobs/work-queue-2/index.md
+2 -208
View File
@@ -2,212 +2,6 @@
title: Fine Parallel Processing using a Work Queue
---
* TOC
{:toc}
{% include user-guide-content-moved.md %}
# Example: Job with Work Queue with Pod Per Work Item
In this example, we will run a Kubernetes Job with multiple parallel
worker processes. You may want to be familiar with the basic,
non-parallel, use of [Job](/docs/user-guide/jobs) first.
In this example, as each pod is created, it picks up one unit of work
from a task queue, completes it, deletes it from the queue, and exits.
Here is an overview of the steps in this example:
1. **Start a storage service to hold the work queue.** In this example, we use Redis to store
our work items. In the previous example, we used RabbitMQ. In this example, we use Redis and
a custom work-queue client library because AMQP does not provide a good way for clients to
detect when a finite-length work queue is empty. In practice you would set up a store such
as Redis once and reuse it for the work queues of many jobs, and other things.
1. **Create a queue, and fill it with messages.** Each message represents one task to be done. In
this example, a message is just an integer that we will do a lengthy computation on.
1. **Start a Job that works on tasks from the queue**. The Job starts several pods. Each pod takes
one task from the message queue, processes it, and repeats until the end of the queue is reached.
## Starting Redis
For this example, for simplicitly, we will start a single instance of Redis.
See the [Redis Example](https://github.com/kubernetes/kubernetes/tree/master/examples/guestbook) for an example
of deploying Redis scalably and redundantly.
Start a temporary Pod running Redis and a service so we can find it.
```shell
$ kubectl create -f docs/user-guide/jobs/work-queue-2/redis-pod.yaml
pod "redis-master" created
$ kubectl create -f docs/user-guide/jobs/work-queue-2/redis-service.yaml
service "redis" created
```
If you're not working from the source tree, you could also download [`redis-pod.yaml`](redis-pod.yaml?raw=true) and [`redis-service.yaml`](redis-service.yaml?raw=true) directly.
## Filling the Queue with tasks
Now let's fill the queue with some "tasks". In our example, our tasks are just strings to be
printed.
Start a temporary interactive pod for running the Redis CLI
```shell
$ kubectl run -i --tty temp --image redis --command "/bin/sh"
Waiting for pod default/redis2-c7h78 to be running, status is Pending, pod ready: false
Hit enter for command prompt
```
Now hit enter, start the redis CLI, and create a list with some work items in it.
```
# redis-cli -h redis
redis:6379> rpush job2 "apple"
(integer) 1
redis:6379> rpush job2 "banana"
(integer) 2
redis:6379> rpush job2 "cherry"
(integer) 3
redis:6379> rpush job2 "date"
(integer) 4
redis:6379> rpush job2 "fig"
(integer) 5
redis:6379> rpush job2 "grape"
(integer) 6
redis:6379> rpush job2 "lemon"
(integer) 7
redis:6379> rpush job2 "melon"
(integer) 8
redis:6379> rpush job2 "orange"
(integer) 9
redis:6379> lrange job2 0 -1
1) "apple"
2) "banana"
3) "cherry"
4) "date"
5) "fig"
6) "grape"
7) "lemon"
8) "melon"
9) "orange"
```
So, the list with key `job2` will be our work queue.
Note: if you do not have Kube DNS setup correctly, you may need to change
the first step of the above block to `redis-cli -h $REDIS_SERVICE_HOST`.
## Create an Image
Now we are ready to create an image that we will run.
We will use a python worker program with a redis client to read
the messages from the message queue.
A simple Redis work queue client library is provided,
called rediswq.py ([Download](rediswq.py?raw=true)).
The "worker" program in each Pod of the Job uses the work queue
client library to get work. Here it is:
{% include code.html language="python" file="worker.py" ghlink="/docs/user-guide/jobs/work-queue-2/worker.py" %}
If you are working from the source tree,
change directory to the `docs/user-guide/jobs/work-queue-2/` directory.
Otherwise, download [`worker.py`](worker.py?raw=true), [`rediswq.py`](rediswq.py?raw=true), and [`Dockerfile`](Dockerfile?raw=true)
using above links. Then build the image:
```shell
docker build -t job-wq-2 .
```
### Push the image
For the [Docker Hub](https://hub.docker.com/), tag your app image with
your username and push to the Hub with the below commands. Replace
`<username>` with your Hub username.
```shell
docker tag job-wq-2 <username>/job-wq-2
docker push <username>/job-wq-2
```
You need to push to a public repository or [configure your cluster to be able to access
your private repository](/docs/user-guide/images).
If you are using [Google Container
Registry](https://cloud.google.com/tools/container-registry/), tag
your app image with your project ID, and push to GCR. Replace
`<project>` with your project ID.
```shell
docker tag job-wq-2 gcr.io/<project>/job-wq-2
gcloud docker push gcr.io/<project>/job-wq-2
```
## Defining a Job
Here is the job definition:
{% include code.html language="yaml" file="job.yaml" ghlink="/docs/user-guide/jobs/work-queue-2/job.yaml" %}
Be sure to edit the job template to
change `gcr.io/myproject` to your own path.
In this example, each pod works on several items from the queue and then exits when there are no more items.
Since the workers themselves detect when the workqueue is empty, and the Job controller does not
know about the workqueue, it relies on the workers to signal when they are done working.
The workers signal that the queue is empty by exiting with success. So, as soon as any worker
exits with success, the controller knows the work is done, and the Pods will exit soon.
So, we set the completion count of the Job to 1. The job controller will wait for the other pods to complete
too.
## Running the Job
So, now run the Job:
```shell
kubectl create -f ./job.yaml
```
Now wait a bit, then check on the job.
```shell
$ kubectl describe jobs/job-wq-2
Name: job-wq-2
Namespace: default
Image(s): gcr.io/exampleproject/job-wq-2
Selector: app in (job-wq-2)
Parallelism: 2
Completions: Unset
Start Time: Mon, 11 Jan 2016 17:07:59 -0800
Labels: app=job-wq-2
Pods Statuses: 1 Running / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
33s 33s 1 {job-controller } Normal SuccessfulCreate Created pod: job-wq-2-lglf8
$ kubectl logs pods/job-wq-2-7r7b2
Worker with sessionID: bbd72d0a-9e5c-4dd6-abf6-416cc267991f
Initial queue state: empty=False
Working on banana
Working on date
Working on lemon
```
As you can see, one of our pods worked on several work units.
## Alternatives
If running a queue service or modifying your containers to use a work queue is inconvenient, you may
want to consider one of the other [job patterns](/docs/user-guide/jobs/#job-patterns).
If you have a continuous stream of background processing work to run, then
consider running your background workers with a `replicationController` instead,
and consider running a background processing library such as
https://github.com/resque/resque.
[Fine Parallel Processing Using a Work Queue](/docs/tasks/job/fine-parallel-processing-work-queue/)
docs/user-guide/kubectl-conventions.md
+2 -2
View File
@@ -23,7 +23,7 @@ If you need stable output in a script, you should:
In order for `kubectl run` to satisfy infrastructure as code:
* Always tag your image with a version-specific tag and don't move that tag to a new version. For example, use `:v1234`, `v1.2.3`, `r03062016-1-4`, rather than `:latest` (see [Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information.)
* Always tag your image with a version-specific tag and don't move that tag to a new version. For example, use `:v1234`, `v1.2.3`, `r03062016-1-4`, rather than `:latest` (see [Best Practices for Configuration](/docs/concepts/configuration/overview/#container-images) for more information.)
* If the image is lightly parameterized, capture the parameters in a checked-in script, or at least use `--record`, to annotate the created objects with the command line.
* If the image is heavily parameterized, definitely check in the script.
* If features are needed that are not expressible via `kubectl run` flags, switch to configuration files checked into source control.
@@ -68,4 +68,4 @@ flag, which will provide the object to be submitted to the cluster.
### `kubectl apply`
* To use `kubectl apply` to update resources, always create resources initially with `kubectl apply` or with `--save-config`. See [managing resources with kubectl apply](/docs/user-guide/managing-deployments/#kubectl-apply) for the reason behind it.
* To use `kubectl apply` to update resources, always create resources initially with `kubectl apply` or with `--save-config`. See [managing resources with kubectl apply](/docs/concepts/cluster-administration/manage-deployment/#kubectl-apply) for the reason behind it.
docs/user-guide/kubectl-overview.md
+1 -3
View File
@@ -7,8 +7,6 @@ title: kubectl Overview
`kubectl` is a command line interface for running commands against Kubernetes clusters. This overview covers `kubectl` syntax, describes the command operations, and provides common examples. For details about each command, including all the supported flags and subcommands, see the [kubectl](/docs/user-guide/kubectl) reference documentation. For installation instructions see [prerequisites](/docs/user-guide/prereqs).
TODO: Auto-generate this file to ensure it's always in sync with any `kubectl` changes, see [#14177](http://pr.k8s.io/14177).
## Syntax
Use the following syntax to run `kubectl` commands from your terminal window:
@@ -35,7 +33,7 @@ where `command`, `TYPE`, `NAME`, and `flags` are:
* To specify multiple resource types individually: `TYPE1/name1 TYPE1/name2 TYPE2/name3 TYPE<#>/name<#>`<br/>
Example: `$ kubectl get pod/example-pod1 replicationcontroller/example-rc1`
* To specify resources with one or more files: `-f file1 -f file2 -f file<#>`
[Use YAML rather than JSON](/docs/user-guide/config-best-practices/#general-config-tips) since YAML tends to be more user-friendly, especially for configuration files.<br/>
[Use YAML rather than JSON](/docs/concepts/configuration/overview/#general-config-tips) since YAML tends to be more user-friendly, especially for configuration files.<br/>
Example: `$ kubectl get pod -f ./pod.yaml`
* `flags`: Specifies optional flags. For example, you can use the `-s` or `--server` flags to specify the address and port of the Kubernetes API server.<br/>
**Important**: Flags that you specify from the command line override default values and any corresponding environment variables.
docs/user-guide/labels.md
-173
View File
@@ -1,173 +0,0 @@
---
assignees:
- mikedanese
title: Labels and Selectors
---
_Labels_ are key/value pairs that are attached to objects, such as pods.
Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but which do not directly imply semantics to the core system.
Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time.
Each object can have a set of key/value labels defined. Each Key must be unique for a given object.
```json
"labels": {
"key1" : "value1",
"key2" : "value2"
}
```
We'll eventually index and reverse-index labels for efficient queries and watches, use them to sort and group in UIs and CLIs, etc. We don't want to pollute labels with non-identifying, especially large and/or structured, data. Non-identifying information should be recorded using [annotations](/docs/user-guide/annotations).
* TOC
{:toc}
## Motivation
Labels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.
Service deployments and batch processing pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier). Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.
Example labels:
* `"release" : "stable"`, `"release" : "canary"`
* `"environment" : "dev"`, `"environment" : "qa"`, `"environment" : "production"`
* `"tier" : "frontend"`, `"tier" : "backend"`, `"tier" : "cache"`
* `"partition" : "customerA"`, `"partition" : "customerB"`
* `"track" : "daily"`, `"track" : "weekly"`
These are just examples; you are free to develop your own conventions.
## Syntax and character set
_Labels_ are key value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (`/`). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (`.`), not longer than 253 characters in total, followed by a slash (`/`).
If the prefix is omitted, the label key is presumed to be private to the user. Automated system components (e.g. `kube-scheduler`, `kube-controller-manager`, `kube-apiserver`, `kubectl`, or other third-party automation) which add labels to end-user objects must specify a prefix. The `kubernetes.io/` prefix is reserved for Kubernetes core components.
Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between.
## Label selectors
Unlike [names and UIDs](/docs/user-guide/identifiers), labels do not provide uniqueness. In general, we expect many objects to carry the same label(s).
Via a _label selector_, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.
The API currently supports two types of selectors: _equality-based_ and _set-based_.
A label selector can be made of multiple _requirements_ which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as an _AND_ logical operator.
An empty label selector (that is, one with zero requirements) selects every object in the collection.
A null label selector (which is only possible for optional selector fields) selects no objects.
**Note**: the label selectors of two controllers must not overlap within a namespace, otherwise they will fight with each other.
### _Equality-based_ requirement
_Equality-_ or _inequality-based_ requirements allow filtering by label keys and values. Matching objects must satisfy all of the specified label constraints, though they may have additional labels as well.
Three kinds of operators are admitted `=`,`==`,`!=`. The first two represent _equality_ (and are simply synonyms), while the latter represents _inequality_. For example:
```
environment = production
tier != frontend
```
The former selects all resources with key equal to `environment` and value equal to `production`.
The latter selects all resources with key equal to `tier` and value distinct from `frontend`, and all resources with no labels with the `tier` key.
One could filter for resources in `production` excluding `frontend` using the comma operator: `environment=production,tier!=frontend`
### _Set-based_ requirement
_Set-based_ label requirements allow filtering keys according to a set of values. Three kinds of operators are supported: `in`,`notin` and exists (only the key identifier). For example:
```
environment in (production, qa)
tier notin (frontend, backend)
partition
!partition
```
The first example selects all resources with key equal to `environment` and value equal to `production` or `qa`.
The second example selects all resources with key equal to `tier` and values other than `frontend` and `backend`, and all resources with no labels with the `tier` key.
The third example selects all resources including a label with key `partition`; no values are checked.
The fourth example selects all resources without a label with key `partition`; no values are checked.
Similarly the comma separator acts as an _AND_ operator. So filtering resources with a `partition` key (no matter the value) and with `environment` different than  `qa` can be achieved using `partition,environment notin (qa)`.
The _set-based_ label selector is a general form of equality since `environment=production` is equivalent to `environment in (production)`; similarly for `!=` and `notin`.
_Set-based_ requirements can be mixed with _equality-based_ requirements. For example: `partition in (customerA, customerB),environment!=qa`.
## API
### LIST and WATCH filtering
LIST and WATCH operations may specify label selectors to filter the sets of objects returned using a query parameter. Both requirements are permitted (presented here as they would appear in a URL query string):
* _equality-based_ requirements: `?labelSelector=environment%3Dproduction,tier%3Dfrontend`
* _set-based_ requirements: `?labelSelector=environment+in+%28production%2Cqa%29%2Ctier+in+%28frontend%29`
Both label selector styles can be used to list or watch resources via a REST client. For example, targeting `apiserver` with `kubectl` and using _equality-based_ one may write:
```shell
$ kubectl get pods -l environment=production,tier=frontend
```
or using _set-based_ requirements:
```shell
$ kubectl get pods -l 'environment in (production),tier in (frontend)'
```
As already mentioned _set-based_ requirements are more expressive.  For instance, they can implement the _OR_ operator on values:
```shell
$ kubectl get pods -l 'environment in (production, qa)'
```
or restricting negative matching via _exists_ operator:
```shell
$ kubectl get pods -l 'environment,environment notin (frontend)'
```
### Set references in API objects
Some Kubernetes objects, such as [`service`s](/docs/user-guide/services) and [`replicationcontroller`s](/docs/user-guide/replication-controller), also use label selectors to specify sets of other resources, such as [pods](/docs/user-guide/pods).
#### Service and ReplicationController
The set of pods that a `service` targets is defined with a label selector. Similarly, the population of pods that a `replicationcontroller` should manage is also defined with a label selector.
Labels selectors for both objects are defined in `json` or `yaml` files using maps, and only _equality-based_ requirement selectors are supported:
```json
"selector": {
"component" : "redis",
}
```
or
```yaml
selector:
component: redis
```
this selector (respectively in `json` or `yaml` format) is equivalent to `component=redis` or `component in (redis)`.
#### Resources that support set-based requirements
Newer resources, such as [`Job`](/docs/user-guide/jobs), [`Deployment`](/docs/user-guide/deployments/), [`Replica Set`](/docs/user-guide/replicasets/), and [`Daemon Set`](/docs/admin/daemons/), support _set-based_ requirements as well.
```yaml
selector:
matchLabels:
component: redis
matchExpressions:
- {key: tier, operator: In, values: [cache]}
- {key: environment, operator: NotIn, values: [dev]}
```
`matchLabels` is a map of `{key,value}` pairs. A single `{key,value}` in the `matchLabels` map is equivalent to an element of `matchExpressions`, whose `key` field is "key", the `operator` is "In", and the `values` array contains only "value". `matchExpressions` is a list of pod selector requirements. Valid operators include In, NotIn, Exists, and DoesNotExist. The values set must be non-empty in the case of In and NotIn. All of the requirements, from both `matchLabels` and `matchExpressions` are ANDed together -- they must all be satisfied in order to match.
#### Selecting sets of nodes
One use case for selecting over labels is to constrain the set of nodes onto which a pod can schedule.
See the documentation on [node selection](/docs/user-guide/node-selection) for more information.
docs/user-guide/managing-deployments.md
+2 -433
View File
@@ -1,438 +1,7 @@
---
assignees:
- bgrant0607
- janetkuo
- mikedanese
title: Managing Resources
---
You've deployed your application and exposed it via a service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating. Among the features we'll discuss in more depth are [configuration files](/docs/user-guide/configuring-containers/#configuration-in-kubernetes) and [labels](/docs/user-guide/deploying-applications/#labels).
{% include user-guide-content-moved.md %}
[Managing Resources](/docs/concepts/cluster-administration/manage-deployment/)
You can find all the files for this example [in our docs
repo here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/).
* TOC
{:toc}
## Organizing resource configurations
Many applications require multiple resources to be created, such as a Deployment and a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML). For example:
{% include code.html language="yaml" file="nginx-app.yaml" ghlink="/docs/user-guide/nginx-app.yaml" %}
Multiple resources can be created the same way as a single resource:
```shell
$ kubectl create -f docs/user-guide/nginx-app.yaml
service "my-nginx-svc" created
deployment "my-nginx" created
```
The resources will be created in the order they appear in the file. Therefore, it's best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment.
`kubectl create` also accepts multiple `-f` arguments:
```shell
$ kubectl create -f docs/user-guide/nginx/nginx-svc.yaml -f docs/user-guide/nginx/nginx-deployment.yaml
```
And a directory can be specified rather than or in addition to individual files:
```shell
$ kubectl create -f docs/user-guide/nginx/
```
`kubectl` will read any files with suffixes `.yaml`, `.yml`, or `.json`.
It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, then you can then simply deploy all of the components of your stack en masse.
A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into github:
```shell
$ kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/user-guide/nginx-deployment.yaml
deployment "nginx-deployment" created
```
## Bulk operations in kubectl
Resource creation isn't the only operation that `kubectl` can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created:
```shell
$ kubectl delete -f docs/user-guide/nginx/
deployment "my-nginx" deleted
service "my-nginx-svc" deleted
```
In the case of just two resources, it's also easy to specify both on the command line using the resource/name syntax:
```shell
$ kubectl delete deployments/my-nginx services/my-nginx-svc
```
For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using `-l` or `--selector`, to filter resources by their labels:
```shell
$ kubectl delete deployment,services -l app=nginx
deployment "my-nginx" deleted
service "my-nginx-svc" deleted
```
Because `kubectl` outputs resource names in the same syntax it accepts, it's easy to chain operations using `$()` or `xargs`:
```shell
$ kubectl get $(kubectl create -f docs/user-guide/nginx/ -o name | grep service)
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx-svc 10.0.0.208 80/TCP 0s
```
With the above commands, we first create resources under docs/user-guide/nginx/ and print the resources created with `-o name` output format
(print each resource as resource/name). Then we `grep` only the "service", and then print it with `kubectl get`.
If you happen to organize your resources across several subdirectories within a particular directory, you can recursively perform the operations on the subdirectories also, by specifying `--recursive` or `-R` alongside the `--filename,-f` flag.
For instance, assume there is a directory `project/k8s/development` that holds all of the manifests needed for the development environment, organized by resource type:
```
project/k8s/development
├── configmap
│   └── my-configmap.yaml
├── deployment
│   └── my-deployment.yaml
└── pvc
└── my-pvc.yaml
```
By default, performing a bulk operation on `project/k8s/development` will stop at the first level of the directory, not processing any subdirectories. If we tried to create the resources in this directory using the following command, we'd encounter an error:
```shell
$ kubectl create -f project/k8s/development
error: you must provide one or more resources by argument or filename (.json|.yaml|.yml|stdin)
```
Instead, specify the `--recursive` or `-R` flag with the `--filename,-f` flag as such:
```shell
$ kubectl create -f project/k8s/development --recursive
configmap "my-config" created
deployment "my-deployment" created
persistentvolumeclaim "my-pvc" created
```
The `--recursive` flag works with any operation that accepts the `--filename,-f` flag such as: `kubectl {create,get,delete,describe,rollout} etc.`
The `--recursive` flag also works when multiple `-f` arguments are provided:
```shell
$ kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive
namespace "development" created
namespace "staging" created
configmap "my-config" created
deployment "my-deployment" created
persistentvolumeclaim "my-pvc" created
```
If you're interested in learning more about `kubectl`, go ahead and read [kubectl Overview](/docs/user-guide/kubectl-overview).
## Using labels effectively
The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
```yaml
labels:
app: guestbook
tier: frontend
```
while the Redis master and slave would have different `tier` labels, and perhaps even an additional `role` label:
```yaml
labels:
app: guestbook
tier: backend
role: master
```
and
```yaml
labels:
app: guestbook
tier: backend
role: slave
```
The labels allow us to slice and dice our resources along any dimension specified by a label:
```shell
$ kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml
$ kubectl get pods -Lapp -Ltier -Lrole
NAME READY STATUS RESTARTS AGE APP TIER ROLE
guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend <none>
guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend <none>
guestbook-fe-jpy62 1/1 Running 0 1m guestbook frontend <none>
guestbook-redis-master-5pg3b 1/1 Running 0 1m guestbook backend master
guestbook-redis-slave-2q2yf 1/1 Running 0 1m guestbook backend slave
guestbook-redis-slave-qgazl 1/1 Running 0 1m guestbook backend slave
my-nginx-divi2 1/1 Running 0 29m nginx <none> <none>
my-nginx-o0ef1 1/1 Running 0 29m nginx <none> <none>
$ kubectl get pods -lapp=guestbook,role=slave
NAME READY STATUS RESTARTS AGE
guestbook-redis-slave-2q2yf 1/1 Running 0 3m
guestbook-redis-slave-qgazl 1/1 Running 0 3m
```
## Canary deployments
Another scenario where multiple labels are needed is to distinguish deployments of different releases or configurations of the same component. It is common practice to deploy a *canary* of a new application release (specified via image tag in the pod template) side by side with the previous release so that the new release can receive live production traffic before fully rolling it out.
For instance, you can use a `track` label to differentiate different releases.
The primary, stable release would have a `track` label with value as `stable`:
```yaml
name: frontend
replicas: 3
...
labels:
app: guestbook
tier: frontend
track: stable
...
image: gb-frontend:v3
```
and then you can create a new release of the guestbook frontend that carries the `track` label with different value (i.e. `canary`), so that two sets of pods would not overlap:
```yaml
name: frontend-canary
replicas: 1
...
labels:
app: guestbook
tier: frontend
track: canary
...
image: gb-frontend:v4
```
The frontend service would span both sets of replicas by selecting the common subset of their labels (i.e. omitting the `track` label), so that the traffic will be redirected to both applications:
```yaml
selector:
app: guestbook
tier: frontend
```
You can tweak the number of replicas of the stable and canary releases to determine the ratio of each release that will receive live production traffic (in this case, 3:1).
Once you're confident, you can update the stable track to the new application release and remove the canary one.
For a more concrete example, check the [tutorial of deploying Ghost](https://github.com/kelseyhightower/talks/tree/master/kubecon-eu-2016/demo#deploy-a-canary).
## Updating labels
Sometimes existing pods and other resources need to be relabeled before creating new resources. This can be done with `kubectl label`.
For example, if you want to label all your nginx pods as frontend tier, simply run:
```shell
$ kubectl label pods -l app=nginx tier=fe
pod "my-nginx-2035384211-j5fhi" labeled
pod "my-nginx-2035384211-u2c7e" labeled
pod "my-nginx-2035384211-u3t6x" labeled
```
This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe".
To see the pods you just labeled, run:
```shell
$ kubectl get pods -l app=nginx -L tier
NAME READY STATUS RESTARTS AGE TIER
my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe
my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe
my-nginx-2035384211-u3t6x 1/1 Running 0 23m fe
```
This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with `-L` or `--label-columns`).
For more information, please see [labels](/docs/user-guide/labels/) and [kubectl label](/docs/user-guide/kubectl/kubectl_label/) document.
## Updating annotations
Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with `kubectl annotate`. For example:
```shell
$ kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx'
$ kubectl get pods my-nginx-v4-9gw19 -o yaml
apiversion: v1
kind: pod
metadata:
annotations:
description: my frontend running nginx
...
```
For more information, please see [annotations](/docs/user-guide/annotations/) and [kubectl annotate](/docs/user-guide/kubectl/kubectl_annotate/) document.
## Scaling your application
When load on your application grows or shrinks, it's easy to scale with `kubectl`. For instance, to decrease the number of nginx replicas from 3 to 1, do:
```shell
$ kubectl scale deployment/my-nginx --replicas=1
deployment "my-nginx" scaled
```
Now you only have one pod managed by the deployment.
```shell
$ kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
my-nginx-2035384211-j5fhi 1/1 Running 0 30m
```
To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do:
```shell
$ kubectl autoscale deployment/my-nginx --min=1 --max=3
deployment "my-nginx" autoscaled
```
Now your nginx replicas will be scaled up and down as needed, automatically.
For more information, please see [kubectl scale](/docs/user-guide/kubectl/kubectl_scale/), [kubectl autoscale](/docs/user-guide/kubectl/kubectl_autoscale/) and [horizontal pod autoscaler](/docs/user-guide/horizontal-pod-autoscaler/) document.
## In-place updates of resources
Sometimes it's necessary to make narrow, non-disruptive updates to resources you've created.
### kubectl apply
It is suggested to maintain a set of configuration files in source control (see [configuration as code](http://martinfowler.com/bliki/InfrastructureAsCode.html)),
so that they can be maintained and versioned along with the code for the resources they configure.
Then, you can use [`kubectl apply`](/docs/user-guide/kubectl/kubectl_apply/) to push your configuration changes to the cluster.
This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified.
```shell
$ kubectl apply -f docs/user-guide/nginx/nginx-deployment.yaml
deployment "my-nginx" configured
```
Note that `kubectl apply` attaches an annotation to the resource in order to determine the changes to the configuration since the previous invocation. When it's invoked, `kubectl apply` does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource.
Currently, resources are created without this annotation, so the first invocation of `kubectl apply` will fall back to a two-way diff between the provided input and the current configuration of the resource. During this first invocation, it cannot detect the deletion of properties set when the resource was created. For this reason, it will not remove them.
All subsequent calls to `kubectl apply`, and other commands that modify the configuration, such as `kubectl replace` and `kubectl edit`, will update the annotation, allowing subsequent calls to `kubectl apply` to detect and perform deletions using a three-way diff.
**Note:** To use apply, always create resource initially with either `kubectl apply` or `kubectl create --save-config`.
### kubectl edit
Alternatively, you may also update resources with `kubectl edit`:
```shell
$ kubectl edit deployment/my-nginx
```
This is equivalent to first `get` the resource, edit it in text editor, and then `apply` the resource with the updated version:
```shell
$ kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml
$ vi /tmp/nginx.yaml
# do some edit, and then save the file
$ kubectl apply -f /tmp/nginx.yaml
deployment "my-nginx" configured
$ rm /tmp/nginx.yaml
```
This allows you to do more significant changes more easily. Note that you can specify the editor with your `EDITOR` or `KUBE_EDITOR` environment variables.
For more information, please see [kubectl edit](/docs/user-guide/kubectl/kubectl_edit/) document.
### kubectl patch
Suppose you want to fix a typo of the container's image of a Deployment. One way to do that is with `kubectl patch`:
```shell
# Suppose you have a Deployment with a container named "nginx" and its image "nignx" (typo),
# use container name "nginx" as a key to update the image from "nignx" (typo) to "nginx"
$ kubectl get deployment my-nginx -o yaml
```
```yaml
apiVersion: extensions/v1beta1
kind: Deployment
...
spec:
template:
spec:
containers:
- image: nignx
name: nginx
...
```
```shell
$ kubectl patch deployment my-nginx -p'{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx"}]}}}}'
"my-nginx" patched
$ kubectl get pod my-nginx-1jgkf -o yaml
```
```yaml
apiVersion: extensions/v1beta1
kind: Deployment
...
spec:
template:
spec:
containers:
- image: nginx
name: nginx
...
```
The patch is specified using json.
The system ensures that you don't clobber changes made by other users or components by confirming that the `resourceVersion` doesn't differ from the version you edited. If you want to update regardless of other changes, remove the `resourceVersion` field when you edit the resource. However, if you do this, don't use your original configuration file as the source since additional fields most likely were set in the live state.
For more information, please see [kubectl patch](/docs/user-guide/kubectl/kubectl_patch/) document.
## Disruptive updates
In some cases, you may need to update resource fields that cannot be updated once initialized, or you may just want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use `replace --force`, which deletes and re-creates the resource. In this case, you can simply modify your original configuration file:
```shell
$ kubectl replace -f docs/user-guide/nginx/nginx-deployment.yaml --force
deployment "my-nginx" deleted
deployment "my-nginx" replaced
```
## Updating your application without a service outage
At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios.
We'll guide you through how to create and update applications with Deployments. If your deployed application is managed by Replication Controllers,
you should read [how to use `kubectl rolling-update`](/docs/user-guide/rolling-updates/) instead.
Let's say you were running version 1.7.9 of nginx:
```shell
$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3
deployment "my-nginx" created
```
To update to version 1.9.1, simply change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`, with the kubectl commands we learned above.
```shell
$ kubectl edit deployment/my-nginx
```
That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit [Deployment page](/docs/user-guide/deployments/).
## What's next?
- [Learn about how to use `kubectl` for application introspection and debugging.](/docs/user-guide/introspection-and-debugging/)
- [Configuration Best Practices and Tips](/docs/user-guide/config-best-practices/)
docs/user-guide/pod-templates.md
+1 -1
View File
@@ -5,7 +5,7 @@ title: Pod Templates
---
Pod templates are [pod](/docs/user-guide/pods/) specifications which are included in other objects, such as
[Replication Controllers](/docs/user-guide/replication-controller/), [Jobs](/docs/user-guide/jobs/), and
[Replication Controllers](/docs/user-guide/replication-controller/), [Jobs](/docs/concepts/jobs/run-to-completion-finite-workloads/), and
[DaemonSets](/docs/admin/daemons/). Controllers use Pod Templates to make actual pods.
Rather than specifying the current desired state of all replicas, pod templates are like cookie cutters. Once a cookie has been cut, the cookie has no relationship to the cutter. There is no quantum entanglement. Subsequent changes to the template or even switching to a new template has no direct effect on the pods already created. Similarly, pods created by a replication controller may subsequently be updated directly. This is in deliberate contrast to pods, which do specify the current desired state of all containers belonging to the pod. This approach radically simplifies system semantics and increases the flexibility of the primitive.
docs/user-guide/replication-controller/index.md
+3 -3
View File
@@ -194,7 +194,7 @@ Ideally, the rolling update controller would take application readiness into acc
The two ReplicationControllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates.
Rolling update is implemented in the client tool
[`kubectl rolling-update`](/docs/user-guide/kubectl/kubectl_rolling-update). Visit [`kubectl rolling-update` tutorial](/docs/user-guide/rolling-updates/) for more concrete examples.
[`kubectl rolling-update`](/docs/user-guide/kubectl/kubectl_rolling-update). Visit [`kubectl rolling-update` task](/docs/tasks/run-application/rolling-update-replication-controller/) for more concrete examples.
### Multiple release tracks
@@ -249,7 +249,7 @@ Unlike in the case where a user directly created pods, a ReplicationController r
### Job
Use a [`Job`](/docs/user-guide/jobs/) instead of a ReplicationController for pods that are expected to terminate on their own
Use a [`Job`](/docs/concepts/jobs/run-to-completion-finite-workloads/) instead of a ReplicationController for pods that are expected to terminate on their own
(i.e. batch jobs).
### DaemonSet
@@ -261,4 +261,4 @@ safe to terminate when the machine is otherwise ready to be rebooted/shutdown.
## For more information
Read [ReplicationController Operations](/docs/user-guide/replication-controller/operations/).
Read [Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/).
docs/user-guide/replication-controller/operations.md
+2 -226
View File
@@ -1,230 +1,6 @@
---
assignees:
- bprashanth
title: Replication Controller Operations
---
* TOC
{:toc}
A replication controller ensures that a specified number of pod "replicas" are
running at any one time. If there are too many, it will kill some. If there are
too few, it will start more.
## Creating a replication controller
Replication controllers are created with `kubectl create`:
```shell
$ kubectl create -f FILE
```
Where:
* `-f FILE` or `--filename FILE` is a relative path to a
[configuration file](#replication_controller_configuration_file) in
either JSON or YAML format.
You can use the [sample file](#sample_file) below to try a create request.
A successful create request returns the name of the replication controller. To
view more details about the controller, see
[Viewing replication controllers](#viewing_replication_controllers) below.
### Replication controller configuration file
When creating a replication controller, you must point to a configuration file
as the value of the `-f` flag. The configuration
file can be formatted as YAML or as JSON, and supports the following fields:
```json
{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"name": "",
"labels": "",
"namespace": ""
},
"spec": {
"replicas": int,
"selector": {
"":""
},
"template": {
"metadata": {
"labels": {
"":""
}
},
"spec": {
// See 'The spec schema' below
}
}
}
}
```
Required fields are:
* `kind`: Always `ReplicationController`.
* `apiVersion`: Currently `v1`.
* `metadata`: An object containing:
* `name`: Required if `generateName` is not specified. The name of this
replication controller. It must be an
[RFC1035](https://www.ietf.org/rfc/rfc1035.txt) compatible value and be
unique within the namespace.
* `labels`: Optional. Labels are arbitrary key:value pairs that can be used
for grouping and targeting by other resources and services.
* `generateName`: Required if `name` is not set. A prefix to use to generate
a unique name. Has the same validation rules as `name`.
* `namespace`: Optional. The namespace of the replication controller.
* `annotations`: Optional. A map of string keys and values that can be used
by external tooling to store and retrieve arbitrary metadata about
objects.
* `spec`: The configuration for this replication controller. It must
contain:
* `replicas`: The number of pods to create and maintain.
* `selector`: A map of key:value pairs assigned to the set of pods that
this replication controller is responsible for managing. **This must**
**match the key:value pairs in the `template`'s `labels` field**.
* `template` contains:
* A `metadata` object with `labels` for the pod.
* The [`spec` schema](#the_spec_schema) that defines the pod
configuration.
### The `spec` schema
The `spec` schema (that is a child of `template`) is described in the locations
below:
* The [`spec` schema](/docs/user-guide/pods/multi-container/#the_spec_schema)
section of the Creating Multi-Container Pods page covers required and
frequently-used fields.
* The entire `spec` schema is documented in the
[Kubernetes API reference](/docs/api-reference/v1/definitions/#_v1_podspec).
### Sample file
The following sample file creates 2 pods, each containing a single container
using the `redis` image. Port 80 on each container is opened. The replication
controller is tagged with the `serving` label. The pods are given the label
`frontend` and the `selector` is set to `frontend`, to indicate that the
controller should manage pods with the `frontend` label.
```json
{
"kind": "ReplicationController",
"apiVersion": "v1",
"metadata": {
"name": "frontend-controller",
"labels": {
"state": "serving"
}
},
"spec": {
"replicas": 2,
"selector": {
"app": "frontend"
},
"template": {
"metadata": {
"labels": {
"app": "frontend"
}
},
"spec": {
"volumes": null,
"containers": [
{
"name": "php-redis",
"image": "redis",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"dnsPolicy": "ClusterFirst"
}
}
}
}
```
## Updating replication controller pods
See [Rolling Updates](/docs/user-guide/rolling-updates/).
## Resizing a replication controller
See
[Resizing a replication controller](/docs/user-guide/resizing-a-replication-controller/).
## Viewing replication controllers
To list replication controllers on a cluster, use the `kubectl get` command:
```shell
$ kubectl get rc
```
A successful get command returns all replication controllers on the cluster in
the specified or default namespace. For example:
```shell
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
frontend php-redis redis name=frontend 2
```
You can also use `get rc NAME` to return information about a specific
replication controller.
To view detailed information about a specific replication controller, use the
`kubectl describe` command:
```shell
$ kubectl describe rc NAME
```
A successful describe request returns details about the replication controller
including number and status of pods managed, and recent events:
```conf
Name: frontend
Namespace: default
Image(s): gcr.io/google_samples/gb-frontend:v3
Selector: name=frontend
Labels: name=frontend
Replicas: 2 current / 2 desired
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-gyx2h
Fri, 06 Nov 2015 16:52:50 -0800 Fri, 06 Nov 2015 16:52:50 -0800 1 {replication-controller } SuccessfulCreate Created pod: frontend-vc9w4
```
## Deleting replication controllers
To delete a replication controller as well as the pods that it controls, use
`kubectl delete`:
```shell
$ kubectl delete rc NAME
```
By default, `kubectl delete rc` will resize the controller to zero (effectively
deleting all pods) before deleting it.
To delete a replication controller without deleting its pods, use
`kubectl delete` and specify `--cascade=false`:
```shell
$ kubectl delete rc NAME --cascade=false
```
A successful delete request returns the name of the deleted resource.
{% include user-guide-content-moved.md %}
[Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/)
docs/user-guide/resizing-a-replication-controller.md
+2 -30
View File
@@ -4,33 +4,5 @@ assignees:
title: Resizing a Replication Controller
---
To increase or decrease the number of pods under a replication controller's
control, use the `kubectl scale` command:
$ kubectl scale rc NAME --replicas=COUNT \
[--current-replicas=COUNT] \
[--resource-version=VERSION]
Tip: You can use the `rc` alias in your commands in place of
`replicationcontroller`.
Required fields are:
* `NAME`: The name of the replication controller to update.
* `--replicas=COUNT`: The desired number of replicas.
Optional fields are:
* `--current-replicas=COUNT`: A precondition for current size. If specified,
the resize will only take place if the current number of replicas matches
this value.
* `--resource-version=VERSION`: A precondition for resource version. If
specified, the resize will only take place if the current replication
controller version matches this value. Versions are specified in the
`labels` field of the replication controller's configuration file, as a
key:value pair with a key of `version`. For example,
`--resource-version test` matches:
"labels": {
"version": "test"
}
{% include user-guide-content-moved.md %}
[Run Stateless AP Replication Controller](/docs/tutorials/stateless-application/run-stateless-ap-replication-controller/#resizing-a-replication-controller)
docs/user-guide/rolling-updates.md
+2 -252
View File
@@ -1,256 +1,6 @@
---
assignees:
- janetkuo
title: Rolling Updates
---
* TOC
{:toc}
## Overview
To update a service without an outage, `kubectl` supports what is called ['rolling update'](/docs/user-guide/kubectl/kubectl_rolling-update), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) and the [example of rolling update](/docs/user-guide/update-demo/) for more information.
Note that `kubectl rolling-update` only supports Replication Controllers. However, if you deploy applications with Replication Controllers,
consider switching them to [Deployments](/docs/user-guide/deployments/). A Deployment is a higher-level controller that automates rolling updates
of applications declaratively, and therefore is recommended. If you still want to keep your Replication Controllers and use `kubectl rolling-update`, keep reading:
A rolling update applies changes to the configuration of pods being managed by
a replication controller. The changes can be passed as a new replication
controller configuration file; or, if only updating the image, a new container
image can be specified directly.
A rolling update works by:
1. Creating a new replication controller with the updated configuration.
2. Increasing/decreasing the replica count on the new and old controllers until
the correct number of replicas is reached.
3. Deleting the original replication controller.
Rolling updates are initiated with the `kubectl rolling-update` command:
$ kubectl rolling-update NAME \
([NEW_NAME] --image=IMAGE | -f FILE)
## Passing a configuration file
To initiate a rolling update using a configuration file, pass the new file to
`kubectl rolling-update`:
$ kubectl rolling-update NAME -f FILE
The configuration file must:
* Specify a different `metadata.name` value.
* Overwrite at least one common label in its `spec.selector` field.
* Use the same `metadata.namespace`.
Replication controller configuration files are described in
[Creating Replication Controllers](/docs/user-guide/replication-controller/operations/).
### Examples
// Update pods of frontend-v1 using new replication controller data in frontend-v2.json.
$ kubectl rolling-update frontend-v1 -f frontend-v2.json
// Update pods of frontend-v1 using JSON data passed into stdin.
$ cat frontend-v2.json | kubectl rolling-update frontend-v1 -f -
## Updating the container image
To update only the container image, pass a new image name and tag with the
`--image` flag and (optionally) a new controller name:
$ kubectl rolling-update NAME [NEW_NAME] --image=IMAGE:TAG
The `--image` flag is only supported for single-container pods. Specifying
`--image` with multi-container pods returns an error.
If no `NEW_NAME` is specified, a new replication controller is created with
a temporary name. Once the rollout is complete, the old controller is deleted,
and the new controller is updated to use the original name.
The update will fail if `IMAGE:TAG` is identical to the
current value. For this reason, we recommend the use of versioned tags as
opposed to values such as `:latest`. Doing a rolling update from `image:latest`
to a new `image:latest` will fail, even if the image at that tag has changed.
Moreover, the use of `:latest` is not recommended, see
[Best Practices for Configuration](/docs/user-guide/config-best-practices/#container-images) for more information.
### Examples
// Update the pods of frontend-v1 to frontend-v2
$ kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2
// Update the pods of frontend, keeping the replication controller name
$ kubectl rolling-update frontend --image=image:v2
## Required and optional fields
Required fields are:
* `NAME`: The name of the replication controller to update.
as well as either:
* `-f FILE`: A replication controller configuration file, in either JSON or
YAML format. The configuration file must specify a new top-level `id` value
and include at least one of the existing `spec.selector` key:value pairs.
See the
[Replication Controller Operations](/docs/user-guide/replication-controller/operations#replication-controller-configuration-file)
page for details.
<br>
<br>
or:
<br>
<br>
* `--image IMAGE:TAG`: The name and tag of the image to update to. Must be
different than the current image:tag currently specified.
Optional fields are:
* `NEW_NAME`: Only used in conjunction with `--image` (not with `-f FILE`). The
name to assign to the new replication controller.
* `--poll-interval DURATION`: The time between polling the controller status
after update. Valid units are `ns` (nanoseconds), `us` or `µs` (microseconds),
`ms` (milliseconds), `s` (seconds), `m` (minutes), or `h` (hours). Units can
be combined (e.g. `1m30s`). The default is `3s`.
* `--timeout DURATION`: The maximum time to wait for the controller to update a
pod before exiting. Default is `5m0s`. Valid units are as described for
`--poll-interval` above.
* `--update-period DURATION`: The time to wait between updating pods. Default
is `1m0s`. Valid units are as described for `--poll-interval` above.
Additional information about the `kubectl rolling-update` command is available
from the [`kubectl` reference](/docs/user-guide/kubectl/kubectl_rolling-update/).
## Walkthrough
Let's say you were running version 1.7.9 of nginx:
```yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: my-nginx
spec:
replicas: 5
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
```
To update to version 1.9.1, you can use [`kubectl rolling-update --image`](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) to specify the new image:
```shell
$ kubectl rolling-update my-nginx --image=nginx:1.9.1
Created my-nginx-ccba8fbd8cc8160970f63f9a2696fc46
```
In another window, you can see that `kubectl` added a `deployment` label to the pods, whose value is a hash of the configuration, to distinguish the new pods from the old:
```shell
$ kubectl get pods -l app=nginx -L deployment
NAME READY STATUS RESTARTS AGE DEPLOYMENT
my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-k156z 1/1 Running 0 1m ccba8fbd8cc8160970f63f9a2696fc46
my-nginx-ccba8fbd8cc8160970f63f9a2696fc46-v95yh 1/1 Running 0 35s ccba8fbd8cc8160970f63f9a2696fc46
my-nginx-divi2 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e
my-nginx-o0ef1 1/1 Running 0 2h 2d1d7a8f682934a254002b56404b813e
my-nginx-q6all 1/1 Running 0 8m 2d1d7a8f682934a254002b56404b813e
```
`kubectl rolling-update` reports progress as it progresses:
```
Scaling up my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 0 to 3, scaling down my-nginx from 3 to 0 (keep 3 pods available, don't exceed 4 pods)
Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 1
Scaling my-nginx down to 2
Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 2
Scaling my-nginx down to 1
Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 up to 3
Scaling my-nginx down to 0
Update succeeded. Deleting old controller: my-nginx
Renaming my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 to my-nginx
replicationcontroller "my-nginx" rolling updated
```
If you encounter a problem, you can stop the rolling update midway and revert to the previous version using `--rollback`:
```shell
$ kubectl rolling-update my-nginx --rollback
Setting "my-nginx" replicas to 1
Continuing update with existing controller my-nginx.
Scaling up nginx from 1 to 1, scaling down my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 from 1 to 0 (keep 1 pods available, don't exceed 2 pods)
Scaling my-nginx-ccba8fbd8cc8160970f63f9a2696fc46 down to 0
Update succeeded. Deleting my-nginx-ccba8fbd8cc8160970f63f9a2696fc46
replicationcontroller "my-nginx" rolling updated
```
This is one example where the immutability of containers is a huge asset.
If you need to update more than just the image (e.g., command arguments, environment variables), you can create a new replication controller, with a new name and distinguishing label value, such as:
```yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: my-nginx-v4
spec:
replicas: 5
selector:
app: nginx
deployment: v4
template:
metadata:
labels:
app: nginx
deployment: v4
spec:
containers:
- name: nginx
image: nginx:1.9.2
args: ["nginx", "-T"]
ports:
- containerPort: 80
```
and roll it out:
```shell
$ kubectl rolling-update my-nginx -f ./nginx-rc.yaml
Created my-nginx-v4
Scaling up my-nginx-v4 from 0 to 5, scaling down my-nginx from 4 to 0 (keep 4 pods available, don't exceed 5 pods)
Scaling my-nginx-v4 up to 1
Scaling my-nginx down to 3
Scaling my-nginx-v4 up to 2
Scaling my-nginx down to 2
Scaling my-nginx-v4 up to 3
Scaling my-nginx down to 1
Scaling my-nginx-v4 up to 4
Scaling my-nginx down to 0
Scaling my-nginx-v4 up to 5
Update succeeded. Deleting old controller: my-nginx
replicationcontroller "my-nginx-v4" rolling updated
```
You can also run the [update demo](/docs/user-guide/update-demo/) to see a visual representation of the rolling update process.
## Troubleshooting
If the `timeout` duration is reached during a rolling update, the operation will
fail with some pods belonging to the new replication controller, and some to the
original controller.
To continue the update from where it failed, retry using the same command.
To roll back to the original state before the attempted update, append the
`--rollback=true` flag to the original command. This will revert all changes.
{% include user-guide-content-moved.md %}
[Rolling Update Replication Controller](/docs/tasks/run-application/rolling-update-replication-controller/)
docs/user-guide/update-demo/index.md
+2 -103
View File
@@ -1,107 +1,6 @@
---
assignees:
- mikedanese
title: Rolling Update Demo
---
This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/user-guide/managing-deployments/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information.
The files for this example are viewable in [our docs repo
here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/update-demo).
### Step Zero: Prerequisites
This example assumes that you have forked the docs repository and [turned up a Kubernetes cluster](/docs/getting-started-guides/):
```shell
$ git clone -b {{page.docsbranch}} https://github.com/kubernetes/kubernetes.github.io
$ cd kubernetes.github.io
```
### Step One: Turn up the UX for the demo
You can use bash job control to run this in the background (note that you must use the default port -- 8001 -- for the following demonstration to work properly).
This can sometimes spew to the output so you could also run it in a different terminal. You have to run `kubectl proxy` in the root of the
Kubernetes repository. Otherwise you will get "404 page not found" errors as the paths will not match. You can find more information about `kubectl proxy`
[here](/docs/user-guide/kubectl/kubectl_proxy).
```shell
$ kubectl proxy --www=docs/user-guide/update-demo/local/ &
I0218 15:18:31.623279 67480 proxy.go:36] Starting to serve on localhost:8001
```
Now visit the [demo website](http://localhost:8001/static). You won't see anything much quite yet.
### Step Two: Run the replication controller
Now we will turn up two replicas of an [image](/docs/user-guide/images/). They all serve on internal port 80.
```shell
$ kubectl create -f docs/user-guide/update-demo/nautilus-rc.yaml
```
After pulling the image from the Docker Hub to your worker nodes (which may take a minute or so) you'll see a couple of squares in the UI detailing the pods that are running along with the image that they are serving up. A cute little nautilus.
### Step Three: Try scaling the replication controller
Now we will increase the number of replicas from two to four:
```shell
$ kubectl scale rc update-demo-nautilus --replicas=4
```
If you go back to the [demo website](http://localhost:8001/static/index.html) you should eventually see four boxes, one for each pod.
### Step Four: Update the docker image
We will now update the docker image to serve a different image by doing a rolling update to a new Docker image.
```shell
$ kubectl rolling-update update-demo-nautilus --update-period=10s -f docs/user-guide/update-demo/kitten-rc.yaml
```
The rolling-update command in kubectl will do 2 things:
1. Create a new [replication controller](/docs/user-guide/replication-controller/) with a pod template that uses the new image (`gcr.io/google_containers/update-demo:kitten`)
2. Scale the old and new replication controllers until the new controller replaces the old. This will kill the current pods one at a time, spinning up new ones to replace them.
Watch the [demo website](http://localhost:8001/static/index.html), it will update one pod every 10 seconds until all of the pods have the new image.
Note that the new replication controller definition does not include the replica count, so the current replica count of the old replication controller is preserved.
But if the replica count had been specified, the final replica count of the new replication controller will be equal to this number.
### Step Five: Bring down the pods
```shell
$ kubectl delete rc update-demo-kitten
```
This first stops the replication controller by turning the target number of replicas to 0 and then deletes the controller.
### Step Six: Cleanup
After you are done running this demo make sure to kill the proxy running in the background:
```shell
$ jobs
[1]+ Running ./kubectl proxy --www=local/ &
$ kill %1
[1]+ Terminated: 15 ./kubectl proxy --www=local/
```
### Updating the Docker images
If you want to build your own docker images, you can set `$DOCKER_HUB_USER` to your Docker user id and run the included shell script. It can take a few minutes to download/upload stuff.
```shell
$ export DOCKER_HUB_USER=my-docker-id
$ ./docs/user-guide/update-demo/build-images.sh
```
To use your custom docker image in the above examples, you will need to change the image name in `docs/user-guide/update-demo/nautilus-rc.yaml` and `docs/user-guide/update-demo/kitten-rc.yaml`.
### Image Copyright
Note that the images included here are public domain.
* [kitten](http://commons.wikimedia.org/wiki/File:Kitten-stare.jpg)
* [nautilus](http://commons.wikimedia.org/wiki/File:Nautilus_pompilius.jpg)
{% include user-guide-content-moved.md %}
[Rolling Update Replication Controller](/docs/tasks/run-application/rolling-update-replication-controller/)
docs/user-guide/update-demo/index.md.orig
+107
View File
@@ -0,0 +1,107 @@
---
assignees:
- mikedanese
title: Rolling Update Demo
---
This example demonstrates the usage of Kubernetes to perform a [rolling update](/docs/user-guide/kubectl/kubectl_rolling-update/) on a running group of [pods](/docs/user-guide/pods/). See [here](/docs/concepts/cluster-administration/manage-deployment/#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/simple-rolling-update.md) for more information.
The files for this example are viewable in [our docs repo
here](https://github.com/kubernetes/kubernetes.github.io/tree/{{page.docsbranch}}/docs/user-guide/update-demo).
### Step Zero: Prerequisites
This example assumes that you have forked the docs repository and [turned up a Kubernetes cluster](/docs/getting-started-guides/):
```shell
$ git clone -b {{page.docsbranch}} https://github.com/kubernetes/kubernetes.github.io
$ cd kubernetes.github.io
```
### Step One: Turn up the UX for the demo
You can use bash job control to run this in the background (note that you must use the default port -- 8001 -- for the following demonstration to work properly).
This can sometimes spew to the output so you could also run it in a different terminal. You have to run `kubectl proxy` in the root of the
Kubernetes repository. Otherwise you will get "404 page not found" errors as the paths will not match. You can find more information about `kubectl proxy`
[here](/docs/user-guide/kubectl/kubectl_proxy).
```shell
$ kubectl proxy --www=docs/user-guide/update-demo/local/ &
I0218 15:18:31.623279 67480 proxy.go:36] Starting to serve on localhost:8001
```
Now visit the [demo website](http://localhost:8001/static). You won't see anything much quite yet.
### Step Two: Run the replication controller
Now we will turn up two replicas of an [image](/docs/user-guide/images/). They all serve on internal port 80.
```shell
$ kubectl create -f docs/user-guide/update-demo/nautilus-rc.yaml
```
After pulling the image from the Docker Hub to your worker nodes (which may take a minute or so) you'll see a couple of squares in the UI detailing the pods that are running along with the image that they are serving up. A cute little nautilus.
### Step Three: Try scaling the replication controller
Now we will increase the number of replicas from two to four:
```shell
$ kubectl scale rc update-demo-nautilus --replicas=4
```
If you go back to the [demo website](http://localhost:8001/static/index.html) you should eventually see four boxes, one for each pod.
### Step Four: Update the docker image
We will now update the docker image to serve a different image by doing a rolling update to a new Docker image.
```shell
$ kubectl rolling-update update-demo-nautilus --update-period=10s -f docs/user-guide/update-demo/kitten-rc.yaml
```
The rolling-update command in kubectl will do 2 things:
1. Create a new [replication controller](/docs/user-guide/replication-controller/) with a pod template that uses the new image (`gcr.io/google_containers/update-demo:kitten`)
2. Scale the old and new replication controllers until the new controller replaces the old. This will kill the current pods one at a time, spinning up new ones to replace them.
Watch the [demo website](http://localhost:8001/static/index.html), it will update one pod every 10 seconds until all of the pods have the new image.
Note that the new replication controller definition does not include the replica count, so the current replica count of the old replication controller is preserved.
But if the replica count had been specified, the final replica count of the new replication controller will be equal to this number.
### Step Five: Bring down the pods
```shell
$ kubectl delete rc update-demo-kitten
```
This first stops the replication controller by turning the target number of replicas to 0 and then deletes the controller.
### Step Six: Cleanup
After you are done running this demo make sure to kill the proxy running in the background:
```shell
$ jobs
[1]+ Running ./kubectl proxy --www=local/ &
$ kill %1
[1]+ Terminated: 15 ./kubectl proxy --www=local/
```
### Updating the Docker images
If you want to build your own docker images, you can set `$DOCKER_HUB_USER` to your Docker user id and run the included shell script. It can take a few minutes to download/upload stuff.
```shell
$ export DOCKER_HUB_USER=my-docker-id
$ ./docs/user-guide/update-demo/build-images.sh
```
To use your custom docker image in the above examples, you will need to change the image name in `docs/user-guide/update-demo/nautilus-rc.yaml` and `docs/user-guide/update-demo/kitten-rc.yaml`.
### Image Copyright
Note that the images included here are public domain.
* [kitten](http://commons.wikimedia.org/wiki/File:Kitten-stare.jpg)
* [nautilus](http://commons.wikimedia.org/wiki/File:Nautilus_pompilius.jpg)