whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'release-1.5' of github.com:kubernetes/kubernetes.github.io into ss-basics

Browse Source
This commit is contained in:
Kenneth Owens
2016-12-01 12:35:14 -08:00
parent 5e15fc667f f9b5c0649f
commit 76e314e32a
100 changed files with 2357 additions and 2092 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/admin/addons.md
+6 -4
View File
@@ -7,18 +7,20 @@ Add-ons extend the functionality of Kubernetes.
This page lists some of the available add-ons and links to their respective installation instructions.
Add-ons in each section are sorted alphabetically - the ordering does not imply any preferential status.
## Networking and Network Policy
* [Weave Net](https://github.com/weaveworks/weave-kube) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.
* [Calico](http://docs.projectcalico.org/v1.5/getting-started/kubernetes/installation/hosted/) is a secure L3 networking and network policy provider.
* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml) is a overlay network provider that can be used with Kubernetes.
* [Calico](http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/hosted/) is a secure L3 networking and network policy provider.
* [Canal](https://github.com/tigera/canal/tree/master/k8s-install/kubeadm) unites Flannel and Calico, providing networking and network policy.
* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml) is a overlay network provider that can be used with Kubernetes.
* [Romana](http://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/user-guide/networkpolicies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize).
* [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.
## Visualization & Control
* [Weave Scope](https://www.weave.works/documentation/scope-latest-installing/#k8s) is a tool for graphically visualizing your containers, pods, services etc. Use it in conjunction with a [Weave Cloud account](https://cloud.weave.works/) or host the UI yourself.
* [Dashboard](https://github.com/kubernetes/dashboard#kubernetes-dashboard) is a dashboard web interface for Kubernetes.
* [Weave Scope](https://www.weave.works/documentation/scope-latest-installing/#k8s) is a tool for graphically visualizing your containers, pods, services etc. Use it in conjunction with a [Weave Cloud account](https://cloud.weave.works/) or host the UI yourself.
## Legacy Add-ons
docs/admin/authentication.md
+24 -2
View File
@@ -6,7 +6,8 @@ assignees:
- deads2k
---
* TOC
{:toc}
## Users in Kubernetes
@@ -33,7 +34,7 @@ or be treated as an anonymous user.
## Authentication strategies
Kubernetes uses client certificates, bearer tokens, or HTTP basic auth to
Kubernetes uses client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth to
authenticate API requests through authentication plugins. As HTTP request are
made to the API server plugins attempts to associate the following attributes
with the request:
@@ -360,6 +361,20 @@ An unsuccessful request would return:
HTTP status codes can be used to supply additional error context.
### Authenticating Proxy
The API server can be configured to identify users from request header values, such as `X-Remote-User`.
It is designed for use in combination with an authenticating proxy, which sets the request header value.
In order to prevent header spoofing, the authenticating proxy is required to present a valid client
certificate to the API server for validation against the specified CA before the request headers are
checked.
* `--requestheader-username-headers` Required, case-insensitive. Header names to check, in order, for the user identity. The first header containing a value is used as the identity.
* `--requestheader-client-ca-file` Required. PEM-encoded certificate bundle. A valid client certificate must be presented and validated against the certificate authorities in the specified file before the request headers are checked for user names.
* `--requestheader-allowed-names` Optional. List of common names (cn). If set, a valid client certificate with a Common Name (cn) in the specified list must be presented before the request headers are checked for user names. If empty, any Common Name is allowed.
### Keystone Password
Keystone authentication is enabled by passing the `--experimental-keystone-url=<AuthURL>`
@@ -367,6 +382,13 @@ option to the API server during startup. The plugin is implemented in
`plugin/pkg/auth/authenticator/password/keystone/keystone.go` and currently uses
basic auth to verify used by username and password.
If you have configured self-signed certificates for the Keystone server,
you may need to set the `--experimental-keystone-ca-file=SOMEFILE` option when
starting the Kubernetes API server. If you set the option, the Keystone
server's certificate is verified by one of the authorities in the
`experimental-keystone-ca-file`. Otherwise, the certificate is verified by
the host's root Certificate Authority.
For details on how to use keystone to manage projects and users, refer to the
[Keystone documentation](http://docs.openstack.org/developer/keystone/). Please
note that this plugin is still experimental, under active development, and likely
docs/admin/authorization.md
+23 -23
View File
@@ -207,6 +207,29 @@ and [enable the API version](
/docs/admin/cluster-management/#turn-on-or-off-an-api-version-for-your-cluster),
with a `--runtime-config=` that includes `rbac.authorization.k8s.io/v1alpha1`.
### Privilege Escalation Prevention and Bootstrapping
The `rbac.authorization.k8s.io` API group inherently attempts to prevent users
from escalating privileges. Simply put, __a user can't grant permissions they
don't already have even when the RBAC authorizer it disabled__. If "user-1"
does not have the ability to read secrets in "namespace-a", they cannot create
a binding that would grant that permission to themselves or any other user.
For bootstrapping the first roles, it becomes necessary for someone to get
around these limitations. For the alpha release of RBAC, an API Server flag was
added to allow one user to step around all RBAC authorization and privilege
escalation checks. NOTE: _This is subject to change with future releases._
```
--authorization-rbac-super-user=admin
```
Once set the specified super user, in this case "admin", can be used to create
the roles and role bindings to initialize the system.
This flag is optional and once the initial bootstrapping is performed can be
unset.
### Roles, RolesBindings, ClusterRoles, and ClusterRoleBindings
The RBAC API Group declares four top level types which will be covered in this
@@ -417,29 +440,6 @@ subjects:
name: system:serviceaccounts
```
### Privilege Escalation Prevention and Bootstrapping
The `rbac.authorization.k8s.io` API group inherently attempts to prevent users
from escalating privileges. Simply put, __a user can't grant permissions they
don't already have even when the RBAC authorizer it disabled__. If "user-1"
does not have the ability to read secrets in "namespace-a", they cannot create
a binding that would grant that permission to themselves or any other user.
For bootstrapping the first roles, it becomes necessary for someone to get
around these limitations. For the alpha release of RBAC, an API Server flag was
added to allow one user to step around all RBAC authorization and privilege
escalation checks. NOTE: _This is subject to change with future releases._
```
--authorization-rbac-super-user=admin
```
Once set the specified super user, in this case "admin", can be used to create
the roles and role bindings to initialize the system.
This flag is optional and once the initial bootstrapping is performed can be
unset.
## Webhook Mode
When specified, mode `Webhook` causes Kubernetes to query an outside REST
docs/admin/cluster-management.md
+3 -3
View File
@@ -95,13 +95,13 @@ If you are using GCE then you can either enable it while creating a cluster with
To configure cluser autoscaler you have to set 3 environment variables:
* `KUBE_ENABLE_CLUSTER_AUTOSCALER` - it enables cluster autoscaler if set to true.
* `KUBE_AUTOSCALING_MIN_NODES` - minimum number of nodes in the cluster.
* `KUBE_AUTOSCALING_MAX_NODES` - maximum number of nodes in the cluster.
* `KUBE_AUTOSCALER_MIN_NODES` - minimum number of nodes in the cluster.
* `KUBE_AUTOSCALER_MAX_NODES` - maximum number of nodes in the cluster.
Example:
```shell
KUBE_ENABLE_CLUSTER_AUTOSCALER=true KUBE_AUTOSCALING_MIN_NODES=3 KUBE_AUTOSCALING_MAX_NODES=10 NUM_NODES=5 ./cluster/kube-up.sh
KUBE_ENABLE_CLUSTER_AUTOSCALER=true KUBE_AUTOSCALER_MIN_NODES=3 KUBE_AUTOSCALER_MAX_NODES=10 NUM_NODES=5 ./cluster/kube-up.sh
```
On GKE you configure cluster autoscaler either on cluster creation or update or when creating a particular node pool
docs/admin/dns.md
+142 -4
View File
@@ -124,7 +124,7 @@ With v1.3, the following annotations are deprecated: `pod.beta.kubernetes.io/hos
## How do I test if it is working?
### Create a simple Pod to use as a test environment.
### Create a simple Pod to use as a test environment
Create a file named busybox.yaml with the
following contents:
@@ -152,7 +152,7 @@ Then create a pod using this file:
kubectl create -f busybox.yaml
```
### Wait for this pod to go into the running state.
### Wait for this pod to go into the running state
You can get its status with:
```
@@ -160,12 +160,13 @@ kubectl get pods busybox
```
You should see:
```
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 <some-time>
```
### Validate DNS works
### Validate that DNS is working
Once that pod is running, you can exec nslookup in that environment:
@@ -185,6 +186,115 @@ Address 1: 10.0.0.1
If you see that, DNS is working correctly.
### Troubleshooting Tips
If the nslookup command fails, check the following:
#### Check the local DNS configuration first
Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" below for more information)
```
cat /etc/resolv.conf
```
Verify that the search path and name server are set up like the following (note that seach path may vary for different cloud providers):
```
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5
```
#### Quick diagnosis
Errors such as the following indicate a problem with the kube-dns add-on or associated Services:
```
$ kubectl exec busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10
nslookup: can't resolve 'kubernetes.default'
```
or
```
$ kubectl exec busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes.default'
```
#### Check if the DNS pod is running
Use the kubectl get pods command to verify that the DNS pod is running.
```
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
```
You should see something like:
```
NAME READY STATUS RESTARTS AGE
...
kube-dns-v19-ezo1y 3/3 Running 0 1h
...
```
If you see that no pod is running or that the pod has failed/completed, the dns add-on may not be deployed by default in your current environment and you will have to deploy it manually.
#### Check for Errors in the DNS pod
Use `kubectl logs` command to see logs for the DNS daemons.
```
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz
```
See if there is any suspicious log. W, E, F letter at the beginning represent Warning, Error and Failure. Please search for entries that have these as the logging level and use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to report unexpected errors.
#### Is dns service up?
Verify that the DNS service is up by using the `kubectl get service` command.
```
kubectl get svc --namespace=kube-system
```
You should see:
```
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
kube-dns 10.0.0.10 <none> 53/UDP,53/TCP 1h
...
```
If you have created the service or in the case it should be created by default but it does not appear, see this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information.
#### Are dns endpoints exposed?
You can verify that dns endpoints are exposed by using the `kubectl get endpoints` command.
```
kubectl get ep kube-dns --namespace=kube-system
```
You should see something like:
```
NAME ENDPOINTS AGE
kube-dns 10.180.3.17:53,10.180.3.17:53 1h
```
If you do not see the endpoints, see endpoints section in the [debugging services documentation](http://kubernetes.io/docs/user-guide/debugging-services/).
For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns) in the Kubernetes GitHub repository.
## Kubernetes Federation (Multiple Zone support)
Release 1.3 introduced Cluster Federation support for multi-site
@@ -213,8 +323,36 @@ the flag `--cluster-domain=<default local domain>`
The Kubernetes cluster DNS server (based off the [SkyDNS](https://github.com/skynetservices/skydns) library)
supports forward lookups (A records), service lookups (SRV records) and reverse IP address lookups (PTR records).
## Inheriting DNS from the node
When running a pod, kubelet will prepend the cluster DNS server and search
paths to the node's own DNS settings. If the node is able to resolve DNS names
specific to the larger environment, pods should be able to, also. See "Known
issues" below for a caveat.
If you don't want this, or if you want a different DNS config for pods, you can
use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will
not inherit DNS. Setting it to a valid file path means that kubelet will use
this file instead of `/etc/resolv.conf` for DNS inheritance.
## Known issues
Kubernetes installs do not configure the nodes' resolv.conf files to use the
cluster DNS by default, because that process is inherently distro-specific.
This should probably be implemented eventually.
Linux's libc is impossibly stuck ([see this bug from
2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just
3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to
consume 1 `nameserver` record and 3 `search` records. This means that if a
local installation already uses 3 `nameserver`s or uses more than 3 `search`es,
some of those settings will be lost. As a partial workaround, the node can run
`dnsmasq` which will provide more `nameserver` entries, but not more `search`
entries. You can also use kubelet's `--resolv-conf` flag.
If you are using Alpine version 3.3 or earlier as your base image, dns may not
work properly owing to a known issue with Alpine. Check [here](https://github.com/kubernetes/kubernetes/issues/30215)
for more information.
## References
- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/build/kube-dns/README.md)
- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/build-tools/kube-dns/README.md)
docs/admin/high-availability/index.md
+3 -4
View File
@@ -100,16 +100,15 @@ for `${NODE_IP}` on each machine.
#### Validating your cluster
Once you copy this into all three nodes, you should have a clustered etcd set up. You can validate with
Once you copy this into all three nodes, you should have a clustered etcd set up. You can validate on master with
```shell
etcdctl member list
kubectl exec < pod_name > etcdctl member list
```
and
```shell
etcdctl cluster-health
kubectl exec < pod_name > etcdctl cluster-health
```
You can also validate that this is working with `etcdctl set foo bar` on one node, and `etcdctl get foo`
docs/admin/network-plugins.md
+10 -4
View File
@@ -36,15 +36,21 @@ Place plugins in `network-plugin-dir/plugin-name/plugin-name`, i.e if you have a
### CNI
The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`).
The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`).
If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used.
In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI `lo` plugin, at minimum version 0.2.0
In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/cni/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0
Limitation: Due to [#31307](https://github.com/kubernetes/kubernetes/issues/31307), `HostPort` won't work with CNI networking plugin at the moment. That means all `hostPort` attribute in pod would be simply ignored.
### kubenet
The Linux-only kubenet plugin provides functionality similar to the `--configure-cbr0` kubelet command-line option. It creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host. The kubenet plugin is currently mutually exclusive with, and will eventually replace, the --configure-cbr0 option. It is also currently incompatible with the flannel experimental overlay.
Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.
Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host.
The kubenet plugin is mutually exclusive with the --configure-cbr0 option.
The plugin requires a few things:
@@ -72,4 +78,4 @@ This option is provided to the network-plugin; currently **only kubenet supports
* `--network-plugin=exec` specifies that we use the `exec` plugin, with executables located in `--network-plugin-dir`.
* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`).
* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge` and `host-local` plugins placed in `/opt/cni/bin` or `network-plugin-dir`.
* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin.
* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin.
docs/admin/networking.md
+22 -24
View File
@@ -1,4 +1,4 @@
---
---
assignees:
- lavalamp
- thockin
@@ -83,7 +83,7 @@ talk to other VMs in your project. This is the same basic model.
Until now this document has talked about containers. In reality, Kubernetes
applies IP addresses at the `Pod` scope - containers within a `Pod` share their
network namespaces - including their IP address. This means that containers
within a `Pod` can all reach each other’s ports on `localhost`. This does imply
within a `Pod` can all reach each other's ports on `localhost`. This does imply
that containers within a `Pod` must coordinate port usage, but this is no
different than processes in a VM. We call this the "IP-per-pod" model. This
is implemented in Docker as a "pod container" which holds the network namespace
@@ -100,8 +100,19 @@ existence or non-existence of host ports.
There are a number of ways that this network model can be implemented. This
document is not an exhaustive study of the various methods, but hopefully serves
as an introduction to various technologies and serves as a jumping-off point.
If some techniques become vastly preferable to others, we might detail them more
here.
The following networking options are sorted alphabetically - the order does not
imply any preferential status.
### Contiv
[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced.
### Flannel
[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay
network that satisfies the Kubernetes requirements. Many
people have reported success with Flannel and Kubernetes.
### Google Compute Engine (GCE)
@@ -158,32 +169,15 @@ Follow the "With Linux Bridge devices" section of [this very nice
tutorial](http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/) from
Lars Kellogg-Stedman.
### Weave Net from Weaveworks
[Weave Net](https://www.weave.works/products/weave-net/) is a
resilient and simple to use network for Kubernetes and its hosted applications.
Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/)
or stand-alone. In either version, it doesn’t require any configuration or extra code
to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.
### Flannel
[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay
network that satisfies the Kubernetes requirements. It installs in minutes and
should get you up and running if the above techniques are not working. Many
people have reported success with Flannel and Kubernetes.
### OpenVSwitch
[OpenVSwitch](/docs/admin/ovs-networking) is a somewhat more mature but also
complicated way to build an overlay network. This is endorsed by several of the
"Big Shops" for networking.
### Project Calico
[Project Calico](https://github.com/projectcalico/calico-containers/blob/master/docs/cni/kubernetes/README.md) is an open source container networking provider and network policy engine.
[Project Calico](http://docs.projectcalico.org/) is an open source container networking provider and network policy engine.
Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall.
@@ -193,9 +187,13 @@ Calico can also be run in policy enforcement mode in conjunction with other netw
[Romana](http://romana.io) is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network. Romana supports Kubernetes [Network Policy](/docs/user-guide/networkpolicies/) to provide isolation across network namespaces.
### Contiv
### Weave Net from Weaveworks
[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced.
[Weave Net](https://www.weave.works/products/weave-net/) is a
resilient and simple to use network for Kubernetes and its hosted applications.
Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/)
or stand-alone. In either version, it doesn't require any configuration or extra code
to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.
## Other reading
docs/admin/node.md
+1 -1
View File
@@ -27,7 +27,7 @@ pieces of information:
The usage of these fields varies depending on your cloud provider or bare metal configuration.
* HostName: Generally not used
* HostName: The hostname as reported by the node's kernel. Can be overridden via the kubelet `--hostname-override` parameter.
* ExternalIP: Generally the IP address of the node that is externally routable (available from outside the cluster)
docs/api.md
-43
View File
@@ -95,46 +95,3 @@ DaemonSets, Deployments, HorizontalPodAutoscalers, Ingress, Jobs and ReplicaSets
Other extensions resources can be enabled by setting runtime-config on
apiserver. runtime-config accepts comma separated values. For ex: to disable deployments and jobs, set
`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/jobs=false`
## v1beta1, v1beta2, and v1beta3 are deprecated; please move to v1 ASAP
As of June 4, 2015, the Kubernetes v1 API has been enabled by default. The v1beta1 and v1beta2 APIs were deleted on June 1, 2015. v1beta3 is planned to be deleted on July 6, 2015.
### v1 conversion tips (from v1beta3)
We're working to convert all documentation and examples to v1. Use `kubectl create --validate` in order to validate your json or yaml against our Swagger spec.
Changes to services are the most significant difference between v1beta3 and v1.
* The `service.spec.portalIP` property is renamed to `service.spec.clusterIP`.
* The `service.spec.createExternalLoadBalancer` property is removed. Specify `service.spec.type: "LoadBalancer"` to create an external load balancer instead.
* The `service.spec.publicIPs` property is deprecated and now called `service.spec.deprecatedPublicIPs`. This property will be removed entirely when v1beta3 is removed. The vast majority of users of this field were using it to expose services on ports on the node. Those users should specify `service.spec.type: "NodePort"` instead. Read [External Services](/docs/user-guide/services/#external-services) for more info. If this is not sufficient for your use case, please file an issue or contact @thockin.
Some other difference between v1beta3 and v1:
* The `pod.spec.containers[*].privileged` and `pod.spec.containers[*].capabilities` properties are now nested under the `pod.spec.containers[*].securityContext` property. See [Security Contexts](/docs/user-guide/security-context).
* The `pod.spec.host` property is renamed to `pod.spec.nodeName`.
* The `endpoints.subsets[*].addresses.IP` property is renamed to `endpoints.subsets[*].addresses.ip`.
* The `pod.status.containerStatuses[*].state.termination` and `pod.status.containerStatuses[*].lastState.termination` properties are renamed to `pod.status.containerStatuses[*].state.terminated` and `pod.status.containerStatuses[*].lastState.terminated` respectively.
* The `pod.status.Condition` property is renamed to `pod.status.conditions`.
* The `status.details.id` property is renamed to `status.details.name`.
### v1beta3 conversion tips (from v1beta1/2)
Some important differences between v1beta1/2 and v1beta3:
* The resource `id` is now called `name`.
* `name`, `labels`, `annotations`, and other metadata are now nested in a map called `metadata`
* `desiredState` is now called `spec`, and `currentState` is now called `status`
* `/minions` has been moved to `/nodes`, and the resource has kind `Node`
* The namespace is required (for all namespaced resources) and has moved from a URL parameter to the path: `/api/v1beta3/namespaces/{namespace}/{resource_collection}/{resource_name}`. If you were not using a namespace before, use `default` here.
* The names of all resource collections are now lower cased - instead of `replicationControllers`, use `replicationcontrollers`.
* To watch for changes to a resource, open an HTTP or Websocket connection to the collection query and provide the `?watch=true` query parameter along with the desired `resourceVersion` parameter to watch from.
* The `labels` query parameter has been renamed to `labelSelector`.
* The `fields` query parameter has been renamed to `fieldSelector`.
* The container `entrypoint` has been renamed to `command`, and `command` has been renamed to `args`.
* Container, volume, and node resources are expressed as nested maps (e.g., `resources{cpu:1}`) rather than as individual fields, and resource values support [scaling suffixes](/docs/user-guide/compute-resources/#specifying-resource-quantities) rather than fixed scales (e.g., milli-cores).
* Restart policy is represented simply as a string (e.g., `"Always"`) rather than as a nested map (`always{}`).
* Pull policies changed from `PullAlways`, `PullNever`, and `PullIfNotPresent` to `Always`, `Never`, and `IfNotPresent`.
* The volume `source` is inlined into `volume` rather than nested.
* Host volumes have been changed from `hostDir` to `hostPath` to better reflect that they can be files or directories.
docs/contribute/create-pull-request.md
+2 -3
View File
@@ -14,10 +14,9 @@ repository. This page shows how to create a pull request.
1. Create a [GitHub account](https://github.com){: target="_blank"}.
1. Sign the
[Google Contributor License Agreement](https://cla.developers.google.com/about/google-individual){: target="_blank"}.
[Linux Foundation Contributor License Agreement](https://identity.linuxfoundation.org/projects/cncf){: target="_blank"}.
1. Sign the
[Linux Contributor License Agreement](https://identity.linuxfoundation.org/projects/cncf){: target="_blank"}.
Documentation will be published under the [CC BY SA 4.0](https://github.com/kubernetes/kubernetes.github.io/blob/master/LICENSE) license.
{% endcapture %}
docs/contribute/page-templates.md
+3
View File
@@ -1,4 +1,7 @@
---
redirect_from:
- /docs/templatedemos/
- /docs/templatedemos.html
---
<!--<html>
docs/contribute/style-guide.md
+203
View File
@@ -0,0 +1,203 @@
---
---
{% capture overview %}
This page gives writing style guidelines for the Kubernetes documentation.
These are guidelines, not rules. Use your best judgment, and feel free to
propose changes to this document in a pull request.
For additional information on creating new content for the Kubernetes
docs, follow the instructions on
[using page templates](/docs/contribute/page-templates/) and
[creating a documentation pull request](/docs/contribute/create-pull-request/).
{% endcapture %}
{% capture body %}
## Documentation formatting standards
### Capitalize API objects
Capitalize the names of API objects. Refer to API objects without saying
"object."
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>The Pod has two Containers.</td><td>The pod has two containers.</td></tr>
<tr><td>The Deployment is responsible for ...</td><td>The Deployment object is responsible for ...</td></tr>
</table>
### Use angle brackets for placeholders
Use angle brackets for placeholders. Tell the reader what a placeholder
represents.
1. Display information about a pod:
kubectl describe pod <pod-name>
where `<pod-name>` is the name of one of your pods.
### Use bold for user interface elements
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>Click <b>Fork</b>.</td><td>Click "Fork".</td></tr>
<tr><td>Select <b>Other</b>.</td><td>Select 'Other'.</td></tr>
</table>
### Use italics to define or introduce new terms
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>A <i>cluster</i> is a set of nodes ...</td><td>A "cluster" is a set of nodes ...</td></tr>
<tr><td>These components form the <i>control plane.</i></td><td>These components form the <b>control plane.</b></td></tr>
</table>
### Use code style for filenames, directories, and paths
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>Open the <code>envars.yaml</code> file.</td><td>Open the envars.yaml file.</td></tr>
<tr><td>Go to the <code>/docs/tutorials</code> directory.</td><td>Go to the /docs/tutorials directory.</td></tr>
<tr><td>Open the <code>/_data/concepts.yaml</code> file.</td><td>Open the /_data/concepts.yaml file.</td></tr>
</table>
## Code snippet formatting
### Use code style for inline code and commands
For inline code in an HTML document, use the `<code>` tag. In a Markdown
document, use the backtick (`).
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>Set the value of the <code>replicas</code> field in the configuration file.</td><td>Set the value of the "replicas" field in the configuration file.</td></tr>
<tr><td>The <code>kubectl run</code> command creates a Deployment.</td><td>The "kubectl run" command creates a Deployment.</td></tr>
</table>
### Don't include the command prompt
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>kubectl get pods</td><td>$ kubectl get pods</td></tr>
</table>
### Separate commands from output
Verify that the pod is running on your chosen node:
kubectl get pods --output=wide
The output is similar to this:
NAME READY STATUS RESTARTS AGE IP NODE
nginx 1/1 Running 0 13s 10.200.0.4 worker0
{% comment %}## Kubernetes.io word list
A list of Kubernetes-specific terms and words to be used consistently across the site.
<table>
<tr><th>Term</th><th>Useage</th></tr>
<tr><td>TBD</td><td>TBD</td></tr>
</table>{% endcomment %}
## Content best practices
This section contains suggested best practices for clear, concise, and consistent content.
### Use present tense
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>This command starts a proxy.</td><td>This command will start a proxy.</td></tr>
</table>
Exception: Use future or past tense if it is required to convey the correct
meaning.
### Use active voice
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>You can explore the API using a browser.</td><td>The API can be explored using a browser.</td></tr>
<tr><td>The YAML file specifies the replica count.</td><td>The replica count is specified in the YAML file.</td></tr>
</table>
Exception: Use passive voice if active voice leads to an awkward construction.
### Use simple and direct language
Use simple and direct language. Avoid using unnecessary phrases, such as saying "please."
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>To create a ReplicaSet, ...</td><td>In order to create a ReplicaSet, ...</td></tr>
<tr><td>See the configuration file.</td><td>Please see the configuration file.</td></tr>
<tr><td>View the Pods.</td><td>With this next command, we'll view the Pods.</td></tr>
</table>
### Address the reader as "you"
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>You can create a Deployment by ...</td><td>We'll create a Deployment by ...</td></tr>
<tr><td>In the preceding output, you can see...</td><td>In the preceding output, we can see ...</td></tr>
</table>
## Patterns to avoid
### Avoid using "we"
Using "we" in a sentence can be confusing, because the reader might not know
whether they're part of the "we" you're describing.
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>Version 1.4 includes ...</td><td>In version 1.4, we have added ...</td></tr>
<tr><td>Kubernetes provides a new feature for ...</td><td>We provide a new feature ...</td></tr>
<tr><td>This page teaches you how to use pods.</td><td>In this page, we are going to learn about pods.</td></tr>
</table>
### Avoid jargon and idioms
Some readers speak English as a second language. Avoid jargon and idioms to help make their understanding easier.
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>Internally, ...</td><td>Under the hood, ...</td></tr>
<tr><td>Create a new cluster.</td><td>Turn up a new cluster.</td></tr>
</table>
### Avoid statements about the future
Avoid making promises or giving hints about the future. If you need to talk about
an alpha feature, put the text under a heading that identifies it as alpha
information.
### Avoid statements that will soon be out of date
Avoid words like "currently" and "new." A feature that is new today might not be
considered new in a few months.
<table>
<tr><th>Do</th><th>Don't</th></tr>
<tr><td>In version 1.4, ...</td><td>In the current version, ...</td></tr>
<tr><td>The Federation feature provides ...</td><td>The new Federation feature provides ...</td></tr>
</table>
{% endcapture %}
{% capture whatsnext %}
* Learn about [writing a new topic](/docs/contribute/write-new-topic/).
* Learn about [using page templates](/docs/contribute/page-templates/).
* Learn about [staging your changes](/docs/contribute/stage-documentation-changes/)
* Learn about [creating a pull request](/docs/contribute/create-pull-request/).
{% endcapture %}
{% include templates/concept.md %}
docs/getting-started-guides/binary_release.md
+2 -2
View File
@@ -33,7 +33,7 @@ cd kubernetes
make release
```
For more details on the release process see the [`build/` directory](http://releases.k8s.io/{{page.githubbranch}}/build/)
For more details on the release process see the [`build/`](http://releases.k8s.io/{{page.githubbranch}}/build/) directory
### Download Kubernetes and automatically set up a default cluster
@@ -57,4 +57,4 @@ Possible values for `YOUR_PROVIDER` include:
* `vsphere` - VMWare VSphere
* `rackspace` - Rackspace
For the complete, up-to-date list of providers supported by this script, see [the `/cluster` folder in the main Kubernetes repo](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/cluster), where each folder represents a possible value for `YOUR_PROVIDER`. If you don't see your desired provider, try looking at our [getting started guides](/docs/getting-started-guides); there's a good chance we have docs for them.
For the complete, up-to-date list of providers supported by this script, see the [`/cluster`](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/cluster) folder in the main Kubernetes repo, where each folder represents a possible value for `YOUR_PROVIDER`. If you don't see your desired provider, try looking at our [getting started guides](/docs/getting-started-guides); there's a good chance we have docs for them.
docs/getting-started-guides/cloudstack.md
+1 -1
View File
@@ -10,7 +10,7 @@ CloudStack is a software to build public and private clouds based on hardware vi
[CoreOS](http://coreos.com) templates for CloudStack are built [nightly](http://stable.release.core-os.net/amd64-usr/current/). CloudStack operators need to [register](http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/templates.html) this template in their cloud before proceeding with these Kubernetes deployment instructions.
This guide uses an [Ansible playbook](https://github.com/runseb/ansible-kubernetes).
This is a completely automated, a single playbook deploys Kubernetes based on the coreOS [instructions](/docs/getting-started-guides/coreos/coreos_multinode_cluster).
This is completely automated, a single playbook deploys Kubernetes.
This [Ansible](http://ansibleworks.com) playbook deploys Kubernetes on a CloudStack based Cloud using CoreOS images. The playbook, creates an ssh key pair, creates a security group and associated rules and finally starts coreOS instances configured via cloud-init.
docs/getting-started-guides/coreos/bare_metal_calico.md
-209
View File
@@ -1,209 +0,0 @@
---
---
This document describes how to deploy Kubernetes with Calico networking on _bare metal_ CoreOS. For more information on Project Calico, visit [projectcalico.org](http://projectcalico.org) and the [calico-containers repository](https://github.com/projectcalico/calico-containers).
To install Calico on an existing Kubernetes cluster, or for more information on deploying Calico with Kubernetes in a number of other environments take a look at our supported [deployment guides](https://github.com/projectcalico/calico-containers/tree/master/docs/cni/kubernetes).
Specifically, this guide will have you do the following:
- Deploy a Kubernetes master node on CoreOS using cloud-config.
- Deploy two Kubernetes compute nodes with Calico Networking using cloud-config.
- Configure `kubectl` to access your cluster.
The resulting cluster will use SSL between Kubernetes components. It will run the SkyDNS service and kube-ui, and be fully conformant with the Kubernetes v1.1 conformance tests.
## Prerequisites and Assumptions
- At least three bare-metal machines (or VMs) to work with. This guide will configure them as follows:
- 1 Kubernetes Master
- 2 Kubernetes Nodes
- Your nodes should have IP connectivity to each other and the internet.
- This guide assumes a DHCP server on your network to assign server IPs.
- This guide uses `192.168.0.0/16` as the subnet from which pod IP addresses are assigned. If this overlaps with your host subnet, you will need to configure Calico to use a different [IP pool](https://github.com/projectcalico/calico-containers/blob/master/docs/calicoctl/pool.md#calicoctl-pool-commands).
## Cloud-config
This guide will use [cloud-config](https://coreos.com/docs/cluster-management/setup/cloudinit-cloud-config/) to configure each of the nodes in our Kubernetes cluster.
We'll use two cloud-config files:
- `master-config.yaml`: cloud-config for the Kubernetes master
- `node-config.yaml`: cloud-config for each Kubernetes node
## Download CoreOS
Download the stable CoreOS bootable ISO from the [CoreOS website](https://coreos.com/docs/running-coreos/platforms/iso/).
## Configure the Kubernetes Master
1. Once you've downloaded the ISO image, burn the ISO to a CD/DVD/USB key and boot from it (if using a virtual machine you can boot directly from the ISO). Once booted, you should be automatically logged in as the `core` user at the terminal. At this point CoreOS is running from the ISO and it hasn't been installed yet.
2. *On another machine*, download the [master cloud-config template](https://raw.githubusercontent.com/projectcalico/calico-cni/k8s-1.1-docs/samples/kubernetes/cloud-config/master-config-template.yaml) and save it as `master-config.yaml`.
3. Replace the following variables in the `master-config.yaml` file.
- `<SSH_PUBLIC_KEY>`: The public key you will use for SSH access to this server. See [generating ssh keys](https://help.github.com/articles/generating-ssh-keys/)
4. Copy the edited `master-config.yaml` to your Kubernetes master machine (using a USB stick, for example).
5. The CoreOS bootable ISO comes with a tool called `coreos-install` which will allow us to install CoreOS and configure the machine using a cloud-config file. The following command will download and install stable CoreOS using the `master-config.yaml` file we just created for configuration. Run this on the Kubernetes master.
> **Warning:** this is a destructive operation that erases disk `sda` on your server.
```shell
sudo coreos-install -d /dev/sda -C stable -c master-config.yaml
```
6. Once complete, restart the server and boot from `/dev/sda` (you may need to remove the ISO image). When it comes back up, you should have SSH access as the `core` user using the public key provided in the `master-config.yaml` file.
### Configure TLS
The master requires the CA certificate, `ca.pem`; its own certificate, `apiserver.pem` and its private key, `apiserver-key.pem`. This [CoreOS guide](https://coreos.com/kubernetes/docs/latest/openssl.html) explains how to generate these.
1. Generate the necessary certificates for the master. This [guide for generating Kubernetes TLS Assets](https://coreos.com/kubernetes/docs/latest/openssl.html) explains how to use OpenSSL to generate the required assets.
2. Send the three files to your master host (using `scp` for example).
3. Move them to the `/etc/kubernetes/ssl` folder and ensure that only the root user can read the key:
```shell
# Move keys
sudo mkdir -p /etc/kubernetes/ssl/
sudo mv -t /etc/kubernetes/ssl/ ca.pem apiserver.pem apiserver-key.pem
# Set Permissions
sudo chmod 600 /etc/kubernetes/ssl/apiserver-key.pem
sudo chown root:root /etc/kubernetes/ssl/apiserver-key.pem
```
4. Restart the kubelet to pick up the changes:
```shell
sudo systemctl restart kubelet
```
## Configure the compute nodes
The following steps will set up a single Kubernetes node for use as a compute host. Run these steps to deploy each Kubernetes node in your cluster.
1. Boot up the node machine using the bootable ISO we downloaded earlier. You should be automatically logged in as the `core` user.
2. Make a copy of the [node cloud-config template](https://raw.githubusercontent.com/projectcalico/calico-cni/k8s-1.1-docs/samples/kubernetes/cloud-config/node-config-template.yaml) for this machine.
3. Replace the following placeholders in the `node-config.yaml` file to match your deployment.
- `<HOSTNAME>`: Hostname for this node (e.g. kube-node1, kube-node2)
- `<SSH_PUBLIC_KEY>`: The public key you will use for SSH access to this server.
- `<KUBERNETES_MASTER>`: The IPv4 address of the Kubernetes master.
4. Replace the following placeholders with the contents of their respective files.
- `<CA_CERT>`: Complete contents of `ca.pem`
- `<CA_KEY_CERT>`: Complete contents of `ca-key.pem`
> **Important:** in a production deployment, embedding the secret key in cloud-config is a bad idea! In production you should use an appropriate secret manager.
> **Important:** Make sure you indent the entire file to match the indentation of the placeholder. For example:
>
> ```shell
> - path: /etc/kubernetes/ssl/ca.pem
> owner: core
> permissions: 0644
> content: |
> <CA_CERT>
> ```
>
> should look like this once the certificate is in place:
>
> ```shell
> - path: /etc/kubernetes/ssl/ca.pem
> owner: core
> permissions: 0644
> content: |
> -----BEGIN CERTIFICATE-----
> MIIC9zCCAd+gAwIBAgIJAJMnVnhVhy5pMA0GCSqGSIb3DQEBCwUAMBIxEDAOBgNV
> ...<snip>...
> QHwi1rNc8eBLNrd4BM/A1ZeDVh/Q9KxN+ZG/hHIXhmWKgN5wQx6/81FIFg==
> -----END CERTIFICATE-----
> ```
5. Move the modified `node-config.yaml` to your Kubernetes node machine and install and configure CoreOS on the node using the following command.
> **Warning:** this is a destructive operation that erases disk `sda` on your server.
```shell
sudo coreos-install -d /dev/sda -C stable -c node-config.yaml
```
6. Once complete, restart the server and boot into `/dev/sda`. When it comes back up, you should have SSH access as the `core` user using the public key provided in the `node-config.yaml` file. It will take some time for the node to be fully configured.
## Configure Kubeconfig
To administer your cluster from a separate host, you will need the client and admin certificates generated earlier (`ca.pem`, `admin.pem`, `admin-key.pem`). With certificates in place, run the following commands with the appropriate filepaths.
```shell
kubectl config set-cluster calico-cluster --server=https://<KUBERNETES_MASTER> --certificate-authority=<CA_CERT_PATH>
kubectl config set-credentials calico-admin --certificate-authority=<CA_CERT_PATH> --client-key=<ADMIN_KEY_PATH> --client-certificate=<ADMIN_CERT_PATH>
kubectl config set-context calico --cluster=calico-cluster --user=calico-admin
kubectl config use-context calico
```
Check your work with `kubectl get nodes`.
## Install the DNS Addon
Most Kubernetes deployments will require the DNS addon for service discovery. To install DNS, create the skydns service and replication controller provided.
```shell
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico-cni/k8s-1.1-docs/samples/kubernetes/master/dns/skydns.yaml
```
## Install the Kubernetes UI Addon (Optional)
The Kubernetes UI can be installed using `kubectl` to run the following manifest file.
```shell
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico-cni/k8s-1.1-docs/samples/kubernetes/master/kube-ui/kube-ui.yaml
```
## Launch other Services With Calico-Kubernetes
At this point, you have a fully functioning cluster running on Kubernetes with a master and two nodes networked with Calico. You can now follow any of the [standard documentation](https://github.com/kubernetes/kubernetes/tree/{{page.version}}.0/examples/) to set up other services on your cluster.
## Connectivity to outside the cluster
Because containers in this guide have private `192.168.0.0/16` IPs, you will need NAT to allow connectivity between containers and the internet. However, in a production data center deployment, NAT is not always necessary, since Calico can peer with the data center's border routers over BGP.
### NAT on the nodes
The simplest method for enabling connectivity from containers to the internet is to use outgoing NAT on your Kubernetes nodes.
Calico can provide outgoing NAT for containers. To enable it, use the following `calicoctl` command:
```shell
ETCD_AUTHORITY=<master_ip:6666> calicoctl pool add <CONTAINER_SUBNET> --nat-outgoing
```
By default, `<CONTAINER_SUBNET>` will be `192.168.0.0/16`. You can find out which pools have been configured with the following command:
```shell
ETCD_AUTHORITY=<master_ip:6666> calicoctl pool show
```
### NAT at the border router
In a data center environment, it is recommended to configure Calico to peer with the border routers over BGP. This means that the container IPs will be routable anywhere in the data center, and so NAT is not needed on the nodes (though it may be enabled at the data center edge to allow outbound-only internet connectivity).
The Calico documentation contains more information on how to configure Calico to [peer with existing infrastructure](https://github.com/projectcalico/calico-containers/blob/master/docs/ExternalConnectivity.md).
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Bare-metal | CoreOS | CoreOS | Calico | [docs](/docs/getting-started-guides/coreos/bare_metal_calico) | | Community ([@caseydavenport](https://github.com/caseydavenport))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/coreos/coreos_multinode_cluster.md
-197
View File
@@ -1,197 +0,0 @@
---
assignees:
- dchen1107
---
Use the [master.yaml](/docs/getting-started-guides/coreos/cloud-configs/master.yaml) and [node.yaml](/docs/getting-started-guides/coreos/cloud-configs/node.yaml) cloud-configs to provision a multi-node Kubernetes cluster.
> **Attention**: This requires at least CoreOS version **[695.0.0][coreos695]**, which includes `etcd2`.
[coreos695]: https://coreos.com/releases/#695.0.0
* TOC
{:toc}
### AWS
*Attention:* Replace `<ami_image_id>` below for a [suitable version of CoreOS image for AWS](https://coreos.com/docs/running-coreos/cloud-providers/ec2/).
#### Provision the Master
```shell
aws ec2 create-security-group --group-name kubernetes --description "Kubernetes Security Group"
aws ec2 authorize-security-group-ingress --group-name kubernetes --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-name kubernetes --protocol tcp --port 80 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-name kubernetes --source-security-group-name kubernetes
```
```shell
aws ec2 run-instances \
--image-id <ami_image_id> \
--key-name <keypair> \
--region us-west-2 \
--security-groups kubernetes \
--instance-type m3.medium \
--user-data file://master.yaml
```
#### Capture the private IP address
```shell
aws ec2 describe-instances --instance-id <master-instance-id>
```
#### Edit node.yaml
Edit `node.yaml` and replace all instances of `<master-private-ip>` with the private IP address of the master node.
#### Provision worker nodes
```shell
aws ec2 run-instances \
--count 1 \
--image-id <ami_image_id> \
--key-name <keypair> \
--region us-west-2 \
--security-groups kubernetes \
--instance-type m3.medium \
--user-data file://node.yaml
```
### Google Compute Engine (GCE)
*Attention:* Replace `<gce_image_id>` below for a [suitable version of CoreOS image for Google Compute Engine](https://coreos.com/docs/running-coreos/cloud-providers/google-compute-engine/).
#### Provision the Master
```shell
gcloud compute instances create master \
--image-project coreos-cloud \
--image <gce_image_id> \
--boot-disk-size 200GB \
--machine-type n1-standard-1 \
--zone us-central1-a \
--metadata-from-file user-data=master.yaml
```
#### Capture the private IP address
```shell
gcloud compute instances list
```
#### Edit node.yaml
Edit `node.yaml` and replace all instances of `<master-private-ip>` with the private IP address of the master node.
#### Provision worker nodes
```shell
gcloud compute instances create node1 \
--image-project coreos-cloud \
--image <gce_image_id> \
--boot-disk-size 200GB \
--machine-type n1-standard-1 \
--zone us-central1-a \
--metadata-from-file user-data=node.yaml
```
#### Establish network connectivity
Next, setup an ssh tunnel to the master so you can run kubectl from your local host.
In one terminal, run `gcloud compute ssh master --ssh-flag="-L 8080:127.0.0.1:8080"` and in a second
run `gcloud compute ssh master --ssh-flag="-R 8080:127.0.0.1:8080"`.
### OpenStack
These instructions are for running on the command line. Most of this you can also do through the Horizon dashboard.
These instructions were tested on the Ice House release on a Metacloud distribution of OpenStack but should be similar if not the same across other versions/distributions of OpenStack.
#### Make sure you can connect with OpenStack
Make sure the environment variables are set for OpenStack such as:
```shell
OS_TENANT_ID
OS_PASSWORD
OS_AUTH_URL
OS_USERNAME
OS_TENANT_NAME
```
Test this works with something like:
```shell
nova list
```
#### Get a Suitable CoreOS Image
You'll need a [suitable version of CoreOS image for OpenStack](https://coreos.com/os/docs/latest/booting-on-openstack.html)
Once you download that, upload it to glance. An example is shown below:
```shell
glance image-create --name CoreOS723 \
--container-format bare --disk-format qcow2 \
--file coreos_production_openstack_image.img \
--is-public True
```
#### Create security group
```shell
nova secgroup-create kubernetes "Kubernetes Security Group"
nova secgroup-add-rule kubernetes tcp 22 22 0.0.0.0/0
nova secgroup-add-rule kubernetes tcp 80 80 0.0.0.0/0
```
#### Provision the Master
```shell
nova boot \
--image <image_name> \
--key-name <my_key> \
--flavor <flavor id> \
--security-group kubernetes \
--user-data files/master.yaml \
kube-master
```
`<image_name>` is the CoreOS image name. In our example we can use the image we created in the previous step and put in 'CoreOS723'
`<my_key>` is the keypair name that you already generated to access the instance.
`<flavor_id>` is the flavor ID you use to size the instance. Run `nova flavor-list` to get the IDs. 3 on the system this was tested with gives the m1.large size.
The important part is to ensure you have the files/master.yml as this is what will do all the post boot configuration. This path is relevant so we are assuming in this example that you are running the nova command in a directory where there is a subdirectory called files that has the master.yml file in it. Absolute paths also work.
Next, assign it a public IP address:
```shell
nova floating-ip-list
```
Get an IP address that's free and run:
```shell
nova floating-ip-associate kube-master <ip address>
```
where `<ip address>` is the IP address that was available from the `nova floating-ip-list` command.
#### Provision Worker Nodes
Edit `node.yaml` and replace all instances of `<master-private-ip>` with the private IP address of the master node. You can get this by running `nova show kube-master` assuming you named your instance kube master. This is not the floating IP address you just assigned it.
```shell
nova boot \
--image <image_name> \
--key-name <my_key> \
--flavor <flavor id> \
--security-group kubernetes \
--user-data files/node.yaml \
minion01
```
This is basically the same as the master nodes but with the node.yaml post-boot script instead of the master.
docs/getting-started-guides/coreos/index.md
-6
View File
@@ -41,12 +41,6 @@ A generic guide to setting up an HA cluster on any cloud or bare metal, with ful
These guides are maintained by community members, cover specific platforms and use cases, and experiment with different ways of configuring Kubernetes on CoreOS.
[**Multi-node Cluster**](/docs/getting-started-guides/coreos/coreos_multinode_cluster)
Set up a single master, multi-worker cluster on your choice of platform: AWS, GCE, or VMware Fusion.
<hr/>
[**Easy Multi-node Cluster on Google Compute Engine**](https://github.com/rimusz/coreos-multi-node-k8s-gce/blob/master/README.md)
Scripted installation of a single master, multi-worker cluster on GCE. Kubernetes components are managed by [fleet](https://github.com/coreos/fleet).
docs/getting-started-guides/index.md
+13 -10
View File
@@ -43,6 +43,8 @@ clusters.
[KCluster.io](https://kcluster.io) provides highly available and scalable managed Kubernetes clusters for AWS.
[Platform9](https://platform9.com/products/kubernetes/) offers managed Kubernetes on-premises or any public cloud, and provides 24/7 health monitoring and alerting.
### Turn-key Cloud Solutions
These solutions allow you to create Kubernetes clusters on a range of Cloud IaaS providers with only a
@@ -74,8 +76,7 @@ These solutions are combinations of cloud provider and OS not covered by the abo
- [AWS + CoreOS](/docs/getting-started-guides/coreos)
- [GCE + CoreOS](/docs/getting-started-guides/coreos)
- [AWS + Ubuntu](/docs/getting-started-guides/juju)
- [Joyent + Ubuntu](/docs/getting-started-guides/juju)
- [AWS/GCE/Rackspace/Joyent + Ubuntu](/docs/getting-started-guides/ubuntu/automated)
- [Rackspace + CoreOS](/docs/getting-started-guides/rackspace)
#### On-Premises VMs
@@ -84,7 +85,7 @@ These solutions are combinations of cloud provider and OS not covered by the abo
- [CloudStack](/docs/getting-started-guides/cloudstack) (uses Ansible, CoreOS and flannel)
- [Vmware vSphere](/docs/getting-started-guides/vsphere) (uses Debian)
- [Vmware Photon Controller](/docs/getting-started-guides/photon-controller) (uses Debian)
- [juju.md](/docs/getting-started-guides/juju) (uses Juju, Ubuntu and flannel)
- [Vmware vSphere, OpenStack, or Bare Metal](/docs/getting-started-guides/ubuntu/automated) (uses Juju, Ubuntu and flannel)
- [Vmware](/docs/getting-started-guides/coreos) (uses CoreOS and flannel)
- [libvirt-coreos.md](/docs/getting-started-guides/libvirt-coreos) (uses CoreOS)
- [oVirt](/docs/getting-started-guides/ovirt)
@@ -99,7 +100,8 @@ These solutions are combinations of cloud provider and OS not covered by the abo
- [Fedora single node](/docs/getting-started-guides/fedora/fedora_manual_config)
- [Fedora multi node](/docs/getting-started-guides/fedora/flannel_multi_node_cluster)
- [Centos](/docs/getting-started-guides/centos/centos_manual_config)
- [Ubuntu](/docs/getting-started-guides/ubuntu)
- [Bare Metal with Ubuntu](/docs/getting-started-guides/ubuntu/automated)
- [Ubuntu Manual](/docs/getting-started-guides/ubuntu/manual)
- [Docker Multi Node](/docs/getting-started-guides/docker-multinode)
- [CoreOS](/docs/getting-started-guides/coreos)
@@ -123,6 +125,7 @@ GKE | | | GCE | [docs](https://clou
Stackpoint.io | | multi-support | multi-support | [docs](http://www.stackpointcloud.com) | | Commercial
AppsCode.com | Saltstack | Debian | multi-support | [docs](https://appscode.com/products/cloud-deployment/) | | Commercial
KCluster.io | | multi-support | multi-support | [docs](https://kcluster.io) | | Commercial
Platform9 | | multi-support | multi-support | [docs](https://platform9.com/products/kubernetes/) | | Commercial
GCE | Saltstack | Debian | GCE | [docs](/docs/getting-started-guides/gce) | ['œ“][1] | Project
Azure | CoreOS | CoreOS | Weave | [docs](/docs/getting-started-guides/coreos/azure/) | | Community ([@errordeveloper](https://github.com/errordeveloper), [@squillace](https://github.com/squillace), [@chanezon](https://github.com/chanezon), [@crossorigin](https://github.com/crossorigin))
Azure | Ignition | Ubuntu | Azure | [docs](/docs/getting-started-guides/azure) | | Community (Microsoft: [@brendandburns](https://github.com/brendandburns), [@colemickens](https://github.com/colemickens))
@@ -140,17 +143,17 @@ AWS | CoreOS | CoreOS | flannel | [docs](/docs/gettin
GCE | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/coreos) | | Community ([@pires](https://github.com/pires))
Vagrant | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/coreos) | | Community ([@pires](https://github.com/pires), [@AntonioMeireles](https://github.com/AntonioMeireles))
Bare-metal (Offline) | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/coreos/bare_metal_offline) | | Community ([@jeffbean](https://github.com/jeffbean))
Bare-metal | CoreOS | CoreOS | Calico | [docs](/docs/getting-started-guides/coreos/bare_metal_calico) | | Community ([@caseydavenport](https://github.com/caseydavenport))
CloudStack | Ansible | CoreOS | flannel | [docs](/docs/getting-started-guides/cloudstack) | | Community ([@runseb](https://github.com/runseb))
Vmware vSphere | Saltstack | Debian | OVS | [docs](/docs/getting-started-guides/vsphere) | | Community ([@imkin](https://github.com/imkin))
Vmware Photon | Saltstack | Debian | OVS | [docs](/docs/getting-started-guides/photon-controller) | | Community ([@alainroy](https://github.com/alainroy))
Bare-metal | custom | CentOS | _none_ | [docs](/docs/getting-started-guides/centos/centos_manual_config) | | Community ([@coolsvap](https://github.com/coolsvap))
AWS | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/whitmo/bundle-kubernetes) ( [@whit](https://github.com/whitmo), [@matt](https://github.com/mbruzek), [@chuck](https://github.com/chuckbutler) )
OpenStack/HPCloud | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/whitmo/bundle-kubernetes) ( [@whit](https://github.com/whitmo), [@matt](https://github.com/mbruzek), [@chuck](https://github.com/chuckbutler) )
Joyent | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/whitmo/bundle-kubernetes) ( [@whit](https://github.com/whitmo), [@matt](https://github.com/mbruzek), [@chuck](https://github.com/chuckbutler) )
AWS | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu/automated) | | [Community](https://github.com/juju-solutions/bundle-canonical-kubernetes) ( [@matt](https://github.com/mbruzek), [@chuck]*(https://github.com/chuckbutler) )
GCE | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu/automated) | | [Community](https://github.com/juju-solutions/bundle-canonical-kubernetes) ( [@matt](https://github.com/mbruzek), [@chuck]*(https://github.com/chuckbutler) )
Bare Metal | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu/automated) | | [Community](https://github.com/juju-solutions/bundle-canonical-kubernetes) ( [@matt](https://github.com/mbruzek), [@chuck]*(https://github.com/chuckbutler) )
Rackspace | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu/automated) | | [Community](https://github.com/juju-solutions/bundle-canonical-kubernetes) ( [@matt](https://github.com/mbruzek), [@chuck]*(https://github.com/chuckbutler) )
Vmware vSphere | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu/automated) | | [Community](https://github.com/juju-solutions/bundle-canonical-kubernetes) ( [@matt](https://github.com/mbruzek), [@chuck]*(https://github.com/chuckbutler) )
AWS | Saltstack | Debian | AWS | [docs](/docs/getting-started-guides/aws) | | Community ([@justinsb](https://github.com/justinsb))
AWS | kops | Debian | AWS | [docs](https://github.com/kubernetes/kops) | | Community ([@justinsb](https://github.com/justinsb))
Bare-metal | custom | Ubuntu | Calico | [docs](/docs/getting-started-guides/ubuntu-calico) | | Community ([@djosborne](https://github.com/djosborne))
Bare-metal | custom | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | Community ([@resouer](https://github.com/resouer), [@WIZARD-CXY](https://github.com/WIZARD-CXY))
libvirt/KVM | CoreOS | CoreOS | libvirt/KVM | [docs](/docs/getting-started-guides/libvirt-coreos) | | Community ([@lhuard1A](https://github.com/lhuard1A))
oVirt | | | | [docs](/docs/getting-started-guides/ovirt) | | Community ([@simon3z](https://github.com/simon3z))
@@ -161,7 +164,7 @@ any | any | any | any | [docs](/docs/gettin
*Note*: The above table is ordered by version test/used in notes followed by support level.
Definition of columns:
Definition of columns
- **IaaS Provider** is who/what provides the virtual or physical machines (nodes) that Kubernetes runs on.
- **OS** is the base operating system of the nodes.
docs/getting-started-guides/juju.md
-325
View File
@@ -1,325 +0,0 @@
---
assignees:
- caesarxuchao
- erictune
---
[Juju](https://jujucharms.com/docs/2.0/about-juju) encapsulates the
operational knowledge of provisioning, installing, and securing a Kubernetes
cluster into one step. Juju allows you to deploy a Kubernetes cluster on
different cloud providers with a consistent, repeatable user experience.
Once deployed the cluster can easily scale up with one command.
The Juju Kubernetes work is curated by a dedicated team of community members,
let us know how we are doing. If you find any problems please open an
[issue on the kubernetes project](https://github.com/kubernetes/kubernetes/issues)
and tag the issue with "juju" so we can find them.
* TOC
{:toc}
## Prerequisites
> Note: If you're running kube-up, on Ubuntu - all of the dependencies
> will be handled for you. You may safely skip to the section:
> [Launch a Kubernetes Cluster](#launch-a-kubernetes-cluster)
### On Ubuntu
[Install the Juju client](https://jujucharms.com/docs/2.0/getting-started-general)
> This documentation focuses on the Juju 2.0 release which will be
> promoted to stable during the April 2016 release cycle.
To paraphrase, on your local Ubuntu system:
```shell
sudo add-apt-repository ppa:juju/devel
sudo apt-get update
sudo apt-get install juju
```
If you are using another distro/platform - please consult the
[getting started guide](https://jujucharms.com/docs/2.0/getting-started-general)
to install the Juju dependencies for your platform.
### With Docker
If you prefer the isolation of Docker, you can run the Juju client in a
container. Create a local directory to store the Juju configuration, then
volume mount the container:
```shell
mkdir -p $HOME/.local/share/juju
docker run --rm -ti \
-v $HOME/.local/share/juju:/home/ubuntu/.local/share/juju \
jujusolutions/charmbox:devel
```
> While this is a common target, the charmbox flavors of images are
> unofficial, and should be treated as experimental. If you encounter any issues
> turning up the Kubernetes cluster with charmbox, please file a bug on the
> [charmbox issue tracker](https://github.com/juju-solutions/charmbox/issues).
### Configure Juju to your favorite cloud provider
At this point you have access to the Juju client. Before you can deploy a
cluster you have to configure Juju with the
[cloud credentials](https://jujucharms.com/docs/2.0/credentials) for each
cloud provider you would like to use.
Juju [supports a wide variety of public clouds](#cloud-compatibility) to set
up the credentials for your chosen cloud see the
[cloud setup page](https://jujucharms.com/docs/devel/getting-started-general#2.-choose-a-cloud).
After configuration is complete test your setup with a `juju bootstrap`
command: `juju bootstrap $controllername $cloudtype` you are ready to launch
the Kubernetes cluster.
## Launch a Kubernetes cluster
You can deploy a Kubernetes cluster with Juju from the `kubernetes` directory of
the [kubernetes github project](https://github.com/kubernetes/kubernetes.git).
Clone the repository on your local system. Export the `KUBERNETES_PROVIDER`
environment variable before bringing up the cluster.
```shell
cd kubernetes
export KUBERNETES_PROVIDER=juju
cluster/kube-up.sh
```
If this is your first time running the `kube-up.sh` script, it will attempt to
install the required dependencies to get started with Juju.
The script will deploy two nodes of Kubernetes, 1 unit of etcd, and network
the units so containers on different hosts can communicate with each other.
## Exploring the cluster
The `juju status` command provides information about each unit in the cluster:
```shell
$ juju status
MODEL CONTROLLER CLOUD/REGION VERSION
default windows azure/centralus 2.0-beta13
APP VERSION STATUS EXPOSED ORIGIN CHARM REV OS
etcd active false jujucharms etcd 3 ubuntu
kubernetes active true jujucharms kubernetes 5 ubuntu
RELATION PROVIDES CONSUMES TYPE
cluster etcd etcd peer
etcd etcd kubernetes regular
certificates kubernetes kubernetes peer
UNIT WORKLOAD AGENT MACHINE PORTS PUBLIC-ADDRESS MESSAGE
etcd/0 active idle 0 2379/tcp 13.67.217.11 (leader) cluster is healthy
kubernetes/0 active idle 1 8088/tcp 13.67.219.76 Kubernetes running.
kubernetes/1 active idle 2 6443/tcp 13.67.219.182 (master) Kubernetes running.
MACHINE STATE DNS INS-ID SERIES AZ
0 started 13.67.217.11 machine-0 trusty
1 started 13.67.219.76 machine-1 trusty
2 started 13.67.219.182 machine-2 trusty
```
## Run some containers!
The `kubectl` file, and the TLS certificates along with the configuration are
all available on the Kubernetes master unit. Fetch the kubectl package so you
can run commands on the new Kuberntetes cluster.
Use the `juju status` command to figure out which unit is the master. In the
example above the "kubernetes/1" unit is the master. Use the `juju scp`
command to copy the file from the unit:
```shell
juju scp kubernetes/1:kubectl_package.tar.gz .
tar xvfz kubectl_package.tar.gz
./kubectl --kubeconfig kubeconfig get pods
```
If you are not on a Linux amd64 host system, you will need to find or build a
kubectl binary package for your architecture.
Copy the `kubeconfig` file to the home directory so you don't have to specify
it on the command line each time. The default location is
`${HOME}/.kube/config`.
No pods will be available before starting a container:
```shell
kubectl get pods
NAME READY STATUSRESTARTS AGE
kubectl get replicationcontrollers
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
```
We'll follow the aws-coreos example. Create a pod manifest: `pod.json`
```json
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "hello",
"labels": {
"name": "hello",
"environment": "testing"
}
},
"spec": {
"containers": [{
"name": "hello",
"image": "quay.io/kelseyhightower/hello",
"ports": [{
"containerPort": 80,
"hostPort": 80
}]
}]
}
}
```
Create the pod with kubectl:
```shell
kubectl create -f pod.json
```
Get info on the pod:
```shell
kubectl get pods
```
To test the hello app, we need to locate which node is hosting
the container. We can use `juju run` and `juju status` commands to find
our hello app.
Exit out of our ssh session and run:
```shell
juju run --unit kubernetes/0 "docker ps -n=1"
...
juju run --unit kubernetes/1 "docker ps -n=1"
CONTAINER IDIMAGE COMMAND CREATED STATUS PORTS NAMES
02beb61339d8quay.io/kelseyhightower/hello:latest /hello About an hour ago Up About an hourk8s_hello....
```
We see "kubernetes/1" has our container, expose the kubernetes charm and open
port 80:
```shell
juju run --unit kubernetes/1 "open-port 80"
juju expose kubernetes
sudo apt-get install curl
curl $(juju status --format=oneline kubernetes/1 | cut -d' ' -f3)
```
Finally delete the pod:
```shell
juju ssh kubernetes/0
kubectl delete pods hello
```
## Scale up cluster
Want larger Kubernetes nodes? It is easy to request different sizes of cloud
resources from Juju by using **constraints**. You can increase the amount of
CPU or memory (RAM) in any of the systems requested by Juju. This allows you
to fine tune th Kubernetes cluster to fit your workload. Use flags on the
bootstrap command or as a separate `juju constraints` command. Look to the
[Juju documentation for machine](https://jujucharms.com/docs/2.0/charms-constraints)
details.
## Scale out cluster
Need more workers? Juju makes it easy to add units of a charm:
```shell
juju add-unit kubernetes
```
Or multiple units at one time:
```shell
juju add-unit -n3 kubernetes
```
You can also scale the etcd charm for more fault tolerant key/value storage:
```shell
juju add-unit -n2 etcd
```
## Tear down cluster
We recommend that you use the `kube-down.sh` script when you are done using
the cluster, as it properly brings down the cloud and removes some of the
build directories.
```shell
./cluster/kube-down.sh
```
Alternately if you want stop the servers you can destroy the Juju model or the
controller. Use the `juju switch` command to get the current controller name:
```shell
juju switch
juju destroy-controller $controllername --destroy-all-models
```
## More Info
Juju works with charms and bundles to deploy solutions. The code that stands up
a Kubernetes cluster is done in the charm code. The charm is built from using
a layered approach to keep the code smaller and more focused on the operations
of Kubernetes.
The Kubernetes layer and bundles can be found in the `kubernetes`
project on github.com:
- [Bundle location](https://github.com/kubernetes/kubernetes/tree/master/cluster/juju/bundles)
- [Kubernetes charm layer location](https://github.com/kubernetes/kubernetes/tree/master/cluster/juju/layers/kubernetes)
- [More about Juju](https://jujucharms.com)
### Cloud compatibility
Juju is cloud agnostic and gives you a consistent experience across different
cloud providers. Juju supports a variety of public cloud providers: [Amazon Web Service](https://jujucharms.com/docs/2.0/help-aws),
[Microsoft Azure](https://jujucharms.com/docs/2.0/help-azure),
[Google Compute Engine](https://jujucharms.com/docs/2.0/help-google),
[Joyent](https://jujucharms.com/docs/2.0/help-joyent),
[Rackspace](https://jujucharms.com/docs/2.0/help-rackspace), any
[OpenStack cloud](https://jujucharms.com/docs/2.0/clouds#specifying-additional-clouds),
and
[Vmware vSphere](https://jujucharms.com/docs/2.0/config-vmware).
If you do not see your favorite cloud provider listed many clouds with ssh
access can be configured for
[manual provisioning](https://jujucharms.com/docs/2.0/clouds-manual).
To change to a different cloud you can use the `juju switch` command and set
up the credentials for that cloud provider and continue to use the `kubeup.sh`
script.
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Amazon Web Services (AWS) | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
OpenStack | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
Microsoft Azure | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
Google Compute Engine (GCE) | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/kops.md
+6
View File
@@ -26,6 +26,12 @@ a building block. kops builds on the kubeadm work.
### (1/5) Install kops
#### Requirements
You must have [kubectl](http://kubernetes.io/docs/getting-started-guides/kubectl/) installed in order for kops to work.
#### Installation
Download kops from the [releases page](https://github.com/kubernetes/kops/releases) (it is also easy to build from source):
On MacOS:
docs/getting-started-guides/kubeadm.md
+2 -2
View File
@@ -115,7 +115,7 @@ To initialize the master, pick one of the machines you previously installed `kub
**Note:** this will autodetect the network interface to advertise the master on as the interface with the default gateway.
If you want to use a different interface, specify `--api-advertise-addresses=<ip-address>` argument to `kubeadm init`.
If you want to use [flannel](https://github.com/coreos/flannel) as the pod network; specify `--pod-network-cidr=10.244.0.0/16` if you're using the daemonset manifest below. _However, please note that this is not required for any other networks, including Weave, which is the recommended pod network._
If you want to use [flannel](https://github.com/coreos/flannel) as the pod network, specify `--pod-network-cidr=10.244.0.0/16` if you're using the daemonset manifest below. _However, please note that this is not required for any other networks besides Flannel._
Please refer to the [kubeadm reference doc](/docs/admin/kubeadm/) if you want to read more about the flags `kubeadm init` provides.
@@ -260,7 +260,7 @@ If there is a firewall, make sure it exposes this port to the internet before yo
* To undo what `kubeadm` did, simply run:
# kubeadm reset
# kubeadm reset
If you wish to start over, run `systemctl start kubelet` followed by `kubeadm init` or `kubeadm join`.
docs/getting-started-guides/kubectl.md
+110
View File
@@ -0,0 +1,110 @@
---
---
<style>
li>.highlighter-rouge {position:relative; top:3px;}
</style>
## Overview
kubectl is the command line tool you use to interact with Kubernetes clusters.
You should use a version of kubectl that is at least as new as your server.
`kubectl version` will print the server and client versions. Using the same version of kubectl
as your server naturally works; using a newer kubectl than your server also works; but if you use
an older kubectl with a newer server you may see odd validation errors .
## Download a release
Download kubectl from the [official Kubernetes releases](https://console.cloud.google.com/storage/browser/kubernetes-release/release/):
On MacOS:
```shell
wget https://storage.googleapis.com/kubernetes-release/release/v1.4.4/bin/darwin/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/local/bin/kubectl
```
On Linux:
```shell
wget https://storage.googleapis.com/kubernetes-release/release/v1.4.4/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/local/bin/kubectl
```
You may need to `sudo` the `mv`; you can put it anywhere in your `PATH` - some people prefer to install to `~/bin`.
## Alternatives
### Download as part of the Google Cloud SDK
kubectl can be installed as part of the Google Cloud SDK:
First install the [Google Cloud SDK](https://cloud.google.com/sdk/).
After Google Cloud SDK installs, run the following command to install `kubectl`:
```shell
gcloud components install kubectl
```
Do check that the version is sufficiently up-to-date using `kubectl version`.
### Install with brew
If you are on MacOS and using brew, you can install with:
```shell
brew install kubectl
```
The homebrew project is independent from kubernetes, so do check that the version is
sufficiently up-to-date using `kubectl version`.
# Enabling shell autocompletion
kubectl includes autocompletion support, which can save a lot of typing!
The completion script itself is generated by kubectl, so you typically just need to invoke it from your profile.
Common examples are provided here, but for more details please consult `kubectl completion -h`
## On Linux, using bash
To add it to your current shell: `source <(kubectl completion bash)`
To add kubectl autocompletion to your profile (so it is automatically loaded in future shells):
```shell
echo "source <(kubectl completion bash)" >> ~/.bashrc
```
## On MacOS, using bash
On MacOS, you will need to install the bash-completion support first:
```shell
brew install bash-completion
```
To add it to your current shell:
```shell
source $(brew --prefix)/etc/bash_completion
source <(kubectl completion bash)
```
To add kubectl autocompletion to your profile (so it is automatically loaded in future shells):
```shell
echo "source $(brew --prefix)/etc/bash_completion" >> ~/.bash_profile
echo "source <(kubectl completion bash)" >> ~/.bash_profile
```
Please note that this only appears to work currently if you install using `brew install kubectl`,
and not if you downloaded kubectl directly.
docs/getting-started-guides/libvirt-coreos.md
+1 -1
View File
@@ -121,7 +121,7 @@ setfacl -m g:kvm:--x ~
By default, the libvirt-coreos setup will create a single Kubernetes master and 3 Kubernetes nodes. Because the VM drives use Copy-on-Write and because of memory ballooning and KSM, there is a lot of resource over-allocation.
There is both an automated way and a manual, customizable way of setting up libvert Kubernetes clusters on CoreOS.
There is both an automated way and a manual, customizable way of setting up libvirt Kubernetes clusters on CoreOS.
#### Automated setup
docs/getting-started-guides/minikube.md
+9 -7
View File
@@ -67,21 +67,23 @@ to run commands against the cluster.
```shell
# linux/amd64
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# linux/386
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/386/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/386/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# linux/arm
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# linux/arm64
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
#linux/ppc64le
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/ppc64le/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/ppc64le/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# OS X/amd64
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# OS X/386
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/386/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/386/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
```
For Windows, download [kubectl.exe](http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/windows/amd64/kubectl.exe) and save it to a location on your PATH.
The generic download path is:
```
https://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/${GOOS}/${GOARCH}/${K8S_BINARY}
docs/getting-started-guides/network-policy/calico.md
+1 -1
View File
@@ -12,7 +12,7 @@ export KUBE_NODE_OS_DISTRIBUTION=debian
curl -sS https://get.k8s.io | bash
```
See the [Calico documentation](https://github.com/projectcalico/calico-containers/tree/master/docs/cni/kubernetes#getting-started) for more options to deploy Calico with Kubernetes.
See the [Calico documentation](http://docs.projectcalico.org/) for more options to deploy Calico with Kubernetes.
Once your cluster using Calico is running, you should see a collection of pods running in the `kube-system` Namespace that support Kubernetes NetworkPolicy.
docs/getting-started-guides/network-policy/walkthrough.md
+2 -1
View File
@@ -10,8 +10,9 @@ In this article, we assume a Kubernetes cluster has been created with network po
* [Calico](/docs/getting-started-guides/network-policy/calico/)
* [Romana](/docs/getting-started-guides/network-policy/romana/)
* [Weave Net](/docs/getting-started-guides/network-policy/weave/)
The reference implementation is [Calico](/docs/getting-started-guides/network-policy/calico) running on GCE.
Add-ons are sorted alphabetically - the ordering does not imply any preferential status.
The following example walkthrough will work on a Kubernetes cluster using any of the listed providers.
docs/getting-started-guides/network-policy/weave.md
+11
View File
@@ -0,0 +1,11 @@
---
assignees:
- bboreham
---
The [Weave Net Addon](https://www.weave.works/docs/net/latest/kube-addon/) for Kubernetes comes with a Network Policy Controller.
This component automatically monitors Kubernetes for any NetworkPolicy annotations on all namespaces, and configures `iptables` rules to allow or block traffic as directed by the policies.
Once you have installed the Weave Net Addon you can follow the [NetworkPolicy gettting started guide](/docs/getting-started-guides/network-policy/walkthrough) to try out Kubernetes NetworkPolicy.
docs/getting-started-guides/rkt/notes.md
+1 -1
View File
@@ -59,7 +59,7 @@ Under rktnetes, `kubectl get logs` currently cannot get logs from applications t
## Init containers
The alpha [init container](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/container-init.md) feature is currently not supported.
The beta [init container](/docs/user-guide/pods/init-containers.md) feature is currently not supported.
## Container restart back-off
docs/getting-started-guides/scratch.md
+4 -4
View File
@@ -81,12 +81,12 @@ to implement one of the above options:
- **Use a network plugin which is called by Kubernetes**
- Kubernetes supports the [CNI](https://github.com/containernetworking/cni) network plugin interface.
- There are a number of solutions which provide plugins for Kubernetes:
- There are a number of solutions which provide plugins for Kubernetes (listed alphabetically):
- [Calico](http://docs.projectcalico.org/)
- [Flannel](https://github.com/coreos/flannel)
- [Calico](https://github.com/projectcalico/calico-containers)
- [Weave](https://weave.works/)
- [Romana](http://romana.io/)
- [Open vSwitch (OVS)](http://openvswitch.org/)
- [Romana](http://romana.io/)
- [Weave](http://weave.works/)
- [More found here](/docs/admin/networking#how-to-achieve-this)
- You can also write your own.
- **Compile support directly into Kubernetes**
docs/getting-started-guides/ubuntu/automated.md
+288
View File
@@ -0,0 +1,288 @@
---
assignees:
- caesarxuchao
- erictune
---
Ubuntu 16.04 introduced the [Canonical Distribution of Kubernetes](https://jujucharms.com/canonical-kubernetes/), a pure upstream distribution of Kubernetes designed for production usage. Out of the box it comes with the following components on 12 machines:
- Kubernetes (automated deployment, operations, and scaling)
- Three node Kubernetes cluster with one master and two worker nodes.
- TLS used for communication between units for security.
- Flannel Software Defined Network (SDN) plugin
- A load balancer for HA kubernetes-master (Experimental)
- Optional Ingress Controller (on worker)
- Optional Dashboard addon (on master) including Heapster for cluster monitoring
- EasyRSA
- Performs the role of a certificate authority serving self signed certificates
to the requesting units of the cluster.
- Etcd (distributed key value store)
- Three unit cluster for reliability.
- Elastic stack
- Two units for ElasticSearch
- One units for a Kibana dashboard
- Beats on every Kubernetes and Etcd units:
- Filebeat for forwarding logs to ElasticSearch
- Topbeat for inserting server monitoring data to ElasticSearch
The Juju Kubernetes work is curated by a dedicated team of community members,
let us know how we are doing. If you find any problems please open an
[issue on our tracker](https://github.com/juju-solutions/bundle-canonical-kubernetes)
so we can find them.
* TOC
{:toc}
## Prerequisites
- A working [Juju client](https://jujucharms.com/docs/2.0/getting-started-general); this does not have to be a Linux machine, it can also be Windows or OSX.
- A [supported cloud](#cloud-compatibility).
### On Ubuntu
On your local Ubuntu system:
```shell
sudo add-apt-repository ppa:juju/stable
sudo apt-get update
sudo apt-get install juju
```
If you are using another distro/platform - please consult the
[getting started guide](https://jujucharms.com/docs/2.0/getting-started-general)
to install the Juju dependencies for your platform.
### Configure Juju to your favorite cloud provider
Deployment of the cluster is [supported on a wide variety of public clouds](#cloud-compatibility), private OpenStack clouds, or raw bare metal clusters.
After deciding which cloud to deploy to, follow the [cloud setup page](https://jujucharms.com/docs/devel/getting-started-general#2.-choose-a-cloud) to configure deploying to that cloud.
Load your [cloud credentials](https://jujucharms.com/docs/2.0/credentials) for each
cloud provider you would like to use.
In this example
```shell
juju add-credential aws
credential name: my_credentials
select auth-type [userpass, oauth, etc]: userpass
enter username: jorge
enter password: *******
```
You can also just auto load credentials for popular clouds with the `juju autoload-credentials` command, which will auto import your credentials from the default files and environment variables for each cloud.
Next we need to bootstrap a controller to manage the cluster. You need to define the cloud you want to bootstrap on, the region, and then any name for your controller node:
```shell
juju update-clouds # This command ensures all the latest regions are up to date on your client
juju bootstrap aws/us-east-2
```
or, another example, this time on Azure:
```shell
juju bootstrap azure/centralus
```
You will need a controller node for each cloud or region you are deploying to. See the [controller documentation](https://jujucharms.com/docs/2.0/controllers) for more information.
Note that each controller can host multiple Kubernetes clusters in a given cloud or region.
## Launch a Kubernetes cluster
The following command will deploy the intial 12-node starter cluster. The speed of execution is very dependent of the performance of the cloud you're deploying to, but
```shell
juju deploy canonical-kubernetes
```
After this command executes we need to wait for the cloud to return back instances and for all the automated deployment tasks to execute.
## Monitor deployment
The `juju status` command provides information about each unit in the cluster. We recommend using the `watch -c juju status --color` command to get a real-time view of the cluster as it deploys. When all the states are green and "Idle", the cluster is ready to go.
```shell
$ juju status
Model Controller Cloud/Region Version
default aws-us-east-2 aws/us-east-2 2.0.1
App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 3 ubuntu
elasticsearch active 2 elasticsearch jujucharms 19 ubuntu
etcd 2.2.5 active 3 etcd jujucharms 14 ubuntu
filebeat active 4 filebeat jujucharms 5 ubuntu
flannel 0.6.1 maintenance 4 flannel jujucharms 5 ubuntu
kibana active 1 kibana jujucharms 15 ubuntu
kubeapi-load-balancer 1.10.0 active 1 kubeapi-load-balancer jujucharms 3 ubuntu exposed
kubernetes-master 1.4.5 active 1 kubernetes-master jujucharms 6 ubuntu
kubernetes-worker 1.4.5 active 3 kubernetes-worker jujucharms 8 ubuntu exposed
topbeat active 3 topbeat jujucharms 5 ubuntu
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 52.15.95.92 Certificate Authority connected.
elasticsearch/0* active idle 1 52.15.67.111 9200/tcp Ready
elasticsearch/1 active idle 2 52.15.109.132 9200/tcp Ready
etcd/0 active idle 3 52.15.79.127 2379/tcp Healthy with 3 known peers.
etcd/1* active idle 4 52.15.111.66 2379/tcp Healthy with 3 known peers. (leader)
etcd/2 active idle 5 52.15.144.25 2379/tcp Healthy with 3 known peers.
kibana/0* active idle 6 52.15.57.157 80/tcp,9200/tcp ready
kubeapi-load-balancer/0* active idle 7 52.15.84.179 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 8 52.15.106.225 6443/tcp Kubernetes master services ready.
filebeat/3 active idle 52.15.106.225 Filebeat ready.
flannel/3 maintenance idle 52.15.106.225 Installing flannel.
kubernetes-worker/0* active idle 9 52.15.153.246 Kubernetes worker running.
filebeat/2 active idle 52.15.153.246 Filebeat ready.
flannel/2 active idle 52.15.153.246 Flannel subnet 10.1.53.1/24
topbeat/2 active idle 52.15.153.246 Topbeat ready.
kubernetes-worker/1 active idle 10 52.15.52.103 Kubernetes worker running.
filebeat/0* active idle 52.15.52.103 Filebeat ready.
flannel/0* active idle 52.15.52.103 Flannel subnet 10.1.31.1/24
topbeat/0* active idle 52.15.52.103 Topbeat ready.
kubernetes-worker/2 active idle 11 52.15.104.181 Kubernetes worker running.
filebeat/1 active idle 52.15.104.181 Filebeat ready.
flannel/1 active idle 52.15.104.181 Flannel subnet 10.1.83.1/24
topbeat/1 active idle 52.15.104.181 Topbeat ready.
Machine State DNS Inst id Series AZ
0 started 52.15.95.92 i-06e66414008eca61c xenial us-east-2c
1 started 52.15.67.111 i-050cbd7eb35fa0fe6 trusty us-east-2a
2 started 52.15.109.132 i-069196660db07c2f6 trusty us-east-2b
3 started 52.15.79.127 i-0038186d2c5103739 xenial us-east-2b
4 started 52.15.111.66 i-0ac66c86a8ec93b18 xenial us-east-2a
5 started 52.15.144.25 i-078cfe79313d598c9 xenial us-east-2c
6 started 52.15.57.157 i-09fd16d9328105ec0 trusty us-east-2a
7 started 52.15.84.179 i-00fd70321a51b658b xenial us-east-2c
8 started 52.15.106.225 i-0109a5fc942c53ed7 xenial us-east-2b
9 started 52.15.153.246 i-0ab63e34959cace8d xenial us-east-2b
10 started 52.15.52.103 i-0108a8cc0978954b5 xenial us-east-2a
11 started 52.15.104.181 i-0f5562571c649f0f2 xenial us-east-2c
```
## Interacting with the cluster
After the cluster is deployed you may assume control over the cluster from any kubernetes-master, or kubernetes-worker node.
First we need to download the credentials and client application to your local workstation:
Create the kubectl config directory.
```shell
mkdir -p ~/.kube
```
Copy the kubeconfig file to the default location.
```shell
juju scp kubernetes-master/0:config ~/.kube/config
```
Fetch a binary for the architecture you have deployed. If your client is a
different architecture you will need to get the appropriate `kubectl` binary
through other means.
```shell
juju scp kubernetes-master/0:kubectl ./kubectl
```
Query the cluster.
```shell
./kubectl cluster-info
Kubernetes master is running at https://52.15.104.227:443
Heapster is running at https://52.15.104.227:443/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://52.15.104.227:443/api/v1/proxy/namespaces/kube-system/services/kube-dns
Grafana is running at https://52.15.104.227:443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at https://52.15.104.227:443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
```
Congratulations, you've now set up a Kubernetes cluster!
## Scale up cluster
Want larger Kubernetes nodes? It is easy to request different sizes of cloud
resources from Juju by using **constraints**. You can increase the amount of
CPU or memory (RAM) in any of the systems requested by Juju. This allows you
to fine tune th Kubernetes cluster to fit your workload. Use flags on the
bootstrap command or as a separate `juju constraints` command. Look to the
[Juju documentation for machine](https://jujucharms.com/docs/2.0/charms-constraints)
details.
## Scale out cluster
Need more workers? We just add more units:
```shell
juju add-unit kubernetes-worker
```
Or multiple units at one time:
```shell
juju add-unit -n3 kubernetes-worker
```
You can also ask for specific instance types or other machine-specific constraints. See the [constraints documentation](https://jujucharms.com/docs/stable/reference-constraints) for more information. Here are some examples, note that generic constraints such as `cores` and `mem` are more portable between clouds. In this case we'll ask for a specific instance type from AWS:
```shell
juju set-constraints kubernetes-worker instance-type=c4.large
juju add-unit kubernetes-worker
```
You can also scale the etcd charm for more fault tolerant key/value storage:
```shell
juju add-unit -n3 etcd
```
It is strongly recommended to run an odd number of units for quorum.
## Tear down cluster
If you want stop the servers you can destroy the Juju model or the
controller. Use the `juju switch` command to get the current controller name:
```shell
juju switch
juju destroy-controller $controllername --destroy-all-models
```
This will shutdown and terminate all running instances on that cloud.
## More Info
We stand up Kubernetes with open-source operations, or operations as code, known as charms. These charms are assembled from layers which keeps the code smaller and more focused on the operations of just Kubernetes and its components.
The Kubernetes layer and bundles can be found in the `kubernetes`
project on github.com:
- [Bundle location](https://github.com/kubernetes/kubernetes/tree/master/cluster/juju/bundles)
- [Kubernetes charm layer location](https://github.com/kubernetes/kubernetes/tree/master/cluster/juju/layers/kubernetes)
- [Canonical Kubernetes home](https://jujucharms.com/canonical-kubernetes/)
Feature requests, bug reports, pull requests or any feedback would be much appreciated.
### Cloud compatibility
This deployment methodology is continually tested on the following clouds:
[Amazon Web Service](https://jujucharms.com/docs/2.0/help-aws),
[Microsoft Azure](https://jujucharms.com/docs/2.0/help-azure),
[Google Compute Engine](https://jujucharms.com/docs/2.0/help-google),
[Joyent](https://jujucharms.com/docs/2.0/help-joyent),
[Rackspace](https://jujucharms.com/docs/2.0/help-rackspace), any
[OpenStack cloud](https://jujucharms.com/docs/2.0/clouds#specifying-additional-clouds),
and
[Vmware vSphere](https://jujucharms.com/docs/2.0/config-vmware).
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Amazon Web Services (AWS) | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
OpenStack | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
Microsoft Azure | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
Google Compute Engine (GCE) | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/juju) | | [Community](https://github.com/juju-solutions/bundle-kubernetes-core) ( [@mbruzek](https://github.com/mbruzek), [@chuckbutler](https://github.com/chuckbutler) )
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/ubuntu-calico.md → docs/getting-started-guides/ubuntu/calico.md
View File
docs/getting-started-guides/ubuntu.md → docs/getting-started-guides/ubuntu/manual.md
View File
docs/getting-started-guides/vsphere.md
+2
View File
@@ -65,6 +65,7 @@ export GOVC_DATACENTER='ha-datacenter' # The datacenter to be used by vSphere cl
```
Sample environment
```shell
export GOVC_URL='10.161.236.217'
export GOVC_USERNAME='administrator'
@@ -79,6 +80,7 @@ export GOVC_DATACENTER='Datacenter'
```
Import this VMDK into your vSphere datastore:
```shell
govc import.vmdk kube.vmdk ./kube/
```
docs/hellonode.md
+1 -1
View File
@@ -145,7 +145,7 @@ docker stop hello_tutorial
Now that the image works as intended and is all tagged with your `$PROJECT_ID`, we can push it to the [Google Container Registry](https://cloud.google.com/tools/container-registry/), a private repository for your Docker images accessible from every Google Cloud project (but also from outside Google Cloud Platform) :
```shell
gcloud docker push gcr.io/$PROJECT_ID/hello-node:v1
gcloud docker -- push gcr.io/$PROJECT_ID/hello-node:v1
```
If all goes well, you should be able to see the container image listed in the console: *Compute > Container Engine > Container Registry*. We now have a project-wide Docker image available which Kubernetes can access and orchestrate.
docs/index.md
+2 -4
View File
@@ -5,9 +5,7 @@ assignees:
---
<p>The Kubernetes documentation can help you set up Kubernetes, learn about the system, or get your applications and workloads running on Kubernetes.</p>
<p><a href="/docs/whatisk8s/" class="button">Read the Kubernetes Overview</a></p>
<p>Kubernetes documentation can help you set up Kubernetes, learn about the system, or get your applications and workloads running on Kubernetes. To learn the basics of what Kubernetes is and how it works, read "<a href="/docs/whatisk8s/">What is Kubernetes</a>". </p>
<h2>Interactive Tutorial</h2>
@@ -40,4 +38,4 @@ assignees:
<h2>Tools</h2>
<p>The <a href="/docs/tools/">tools</a> page contains a list of native and third-party tools for Kubernetes.</p>
<p>The <a href="/docs/tools/">tools</a> page contains a list of native and third-party tools for Kubernetes.</p>
docs/tasks/debug-application-cluster/determine-reason-pod-failure.md
+110
View File
@@ -0,0 +1,110 @@
---
---
{% capture overview %}
This page shows how to write and read a Container
termination message.
Termination messages provide a way for containers to write
information about fatal events to a location where it can
be easily retrieved and surfaced by tools like dashboards
and monitoring software. In most cases, information that you
put in a termination message should also be written to
the general
[Kubernetes logs](/docs/user-guide/logging/).
{% endcapture %}
{% capture prerequisites %}
{% include task-tutorial-prereqs.md %}
{% endcapture %}
{% capture steps %}
### Writing and reading a termination message
In this exercise, you create a Pod that runs one container.
The configuration file specifies a command that runs when
the container starts.
{% include code.html language="yaml" file="termination.yaml" ghlink="/docs/tasks/debug-pod-container/termination.yaml" %}
1. Create a Pod based on the YAML configuration file:
export REPO=https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master
kubectl create -f $REPO/docs/tasks/debug-pod-container/termination.yaml
In the YAML file, in the `cmd` and `args` fields, you can see that the
container sleeps for 10 seconds and then writes "Sleep expired" to
the `/dev/termination-log` file. After the container writes
the "Sleep expired" message, it terminates.
1. Display information about the Pod:
kubectl get pod termination-demo
Repeat the preceding command until the Pod is no longer running.
1. Display detailed information about the Pod:
kubectl get pod --output=yaml
The output includes the "Sleep expired" message:
apiVersion: v1
kind: Pod
...
lastState:
terminated:
containerID: ...
exitCode: 0
finishedAt: ...
message: |
Sleep expired
...
1. Use a Go template to filter the output so that it includes
only the termination message:
```
{% raw %} kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"{% endraw %}
```
### Setting the termination log file
By default Kubernetes retrieves termination messages from
`/dev/termination-log`. To change this to a different file,
specify a `terminationMessagePath` field for your Container.
For example, suppose your Container writes termination messages to
`/tmp/my-log`, and you want Kubernetes to retrieve those messages.
Set `terminationMessagePath` as shown here:
apiVersion: v1
kind: Pod
metadata:
name: msg-path-demo
spec:
containers:
- name: msg-path-demo-container
image: debian
terminationMessagePath: "/tmp/my-log"
{% endcapture %}
{% capture whatsnext %}
* See the `terminationMessagePath` field in
[Container](/docs/api-reference/v1/definitions#_v1_container).
* Learn about [retrieving logs](/docs/user-guide/logging/).
* Learn about [Go templates](https://golang.org/pkg/text/template/).
{% endcapture %}
{% include templates/task.md %}
docs/tasks/debug-application-cluster/termination.yaml
+10
View File
@@ -0,0 +1,10 @@
apiVersion: v1
kind: Pod
metadata:
name: termination-demo
spec:
containers:
- name: termination-demo-container
image: debian
command: ["/bin/sh"]
args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"]
docs/tasks/index.md
+7 -1
View File
@@ -1,7 +1,9 @@
---
---
The Tasks section of the Kubernetes documentation is a work in progress
This section of the Kubernetes documentation contains pages that
show how to do individual tasks. A task page shows how to do a
single thing, typically by giving a short sequence of steps.
#### Configuring Pods and Containers
@@ -13,6 +15,10 @@ The Tasks section of the Kubernetes documentation is a work in progress
* [Using Port Forwarding to Access Applications in a Cluster](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
#### Debugging Applications in a Cluster
* [Determining the Reason for Pod Failure](/docs/tasks/debug-application-cluster/determine-reason-pod-failure/)
#### Accessing the Kubernetes API
* [Using an HTTP Proxy to Access the Kubernetes API](/docs/tasks/access-kubernetes-api/http-proxy-access-api)
docs/templatedemos/OWNERS
-3
View File
@@ -1,3 +0,0 @@
assignees:
- pwittrock
docs/templatedemos/blank.md
-3
View File
@@ -1,3 +0,0 @@
---
---
{% include templates/concept-overview.md %}
docs/templatedemos/blankkubectl.md
-6
View File
@@ -1,6 +0,0 @@
---
assignees:
- bgrant0607
---
{% include templates/kubectl.md %}
docs/templatedemos/blanklanding.md
-3
View File
@@ -1,3 +0,0 @@
---
---
{% include templates/landing-page.md %}
docs/templatedemos/blanktask.md
-3
View File
@@ -1,3 +0,0 @@
---
---
{% include templates/task.md %}
docs/templatedemos/filledout.md
-80
View File
@@ -1,80 +0,0 @@
---
---
{% assign concept="Pod" %}
{% capture what_is %}
A pod is the vehicle for running containers in Kubernetes. A pod consists of:
- One or more containers
- An IP address that is unique within the cluster
- Optionally: Environment variables, storage volumes, and enterprise features (such as health checking)
Resources are shared amongst containers in the pod. Containers within a pod also share an IP address and port space, and can find each other via localhost, or interprocess communications (such as semaphores).
![Pod diagram](/images/docs/pod-overview.svg){: style="max-width: 25%" }
{% comment %}https://drive.google.com/open?id=1pQe4-s76fqyrzB8f3xoJo4MPLNVoBlsE1tT9MyLNINg{% endcomment %}
{% endcapture %}
{% capture when_to_use %}
Pods are used any time you need a container to be run. However, they are rarely created by a user, and are instead automatically created by controllers such as jobs, replication controllers, deployments, daemon set. The following table describes the strategy each controller uses to create pods.
| Controller | Usage Strategy |
|------------|----------------|
| Deployment | For running pods as a continuous and healthy application |
| Replication Controller | Used for the same purpose as Deployments (superseded Replication Controllers) |
| Jobs | For running pods "to completion" (which are then shut down) |
| Daemon Set | Mainly for performing operations on any nodes that match given parameters |
{% endcapture %}
{% capture when_not_to_use %}
Do not use pods directly. Pods should always be managed by a controller.
{% endcapture %}
{% capture status %}
To retrieve the status of a pod, run the following command:
```shell
kubectl get pod <name>
```
| Return Value | Description |
|--------------|-------------|
| `READY` | Describes the number of containers that are ready to receive traffic. |
| `STATUS` | A value from the `PodPhase` enum describing the current status of the pod. Can be `Running`, `Pending`, `Succeeded`, `Failed`, and `Unknown`. |
TODO: Link to refpage for `kubectl get pod`
To get a full description of a pod, including past events, run the following command:
```shell
kubectl describe pod <name>
```
TODO: Link to refpage for `kubectl describe pod`
#### Possible status results
| Value | Description |
|------------|----------------|
| Deployment | For running pods as a continuous and healthy application |
| Replication Controller | Used for the same purpose as Deployments (superseded Replication Controllers) |
| Jobs | For running pods "to completion" (which are then shut down) |
| Daemon Set | Mainly for performing operations on any nodes that match given parameters |
{% endcapture %}
{% capture usage %}
Pods are defined when configuring the controller of your choice. In controller specifications,
the parts that define the contents of the pod are inside the `template:` section.
```yaml
YAML EXAMPLE HERE
```
{% endcapture %}
{% include templates/concept-overview.md %}
docs/templatedemos/index.md
-249
View File
@@ -1,249 +0,0 @@
---
assignees:
- pwittrock
---
* TOC
{:toc}
## Before you Begin: Get the docs code checked out locally
Check out the kubernetes/kubernetes.github.io repo and the docsv2 branch.
### Step 1: Fork and Clone the repo
- Fork [kubernetes/kubernetes.github.io](https://github.com/kubernetes/kubernetes.github.io)
- [Setup your GitHub authentication using ssh](https://help.github.com/articles/generating-an-ssh-key/)
- Clone the repo under ~/go/src/k8s.io
```shell
cd ~/go/src/k8s.io
git clone git@github.com:<your-github-username>/kubernetes.github.io
cd kubernetes.github.io
git remote add upstream https://github.com/kubernetes/kubernetes.github.io.git
```
### Step 2: Switch to the docsv2 branch
Docs v2 development is being performed in the `docsv2` branch. This is the branch
you want to be working from.
From ~/go/src/k8s.io/kubernetes.github.io:
```shell
git checkout -b docsv2
git fetch upstream
git reset --hard upstream/docsv2
```
### Step 3: Make sure you can serve rendered docs
One option is to simply rename your fork's repo on GitHub.com to `yourusername.github.io`, which will auto-stage your commits at that URL.
Or, just use Docker! Run this from within your local `kubernetes.github.io` directory and you should be good:
```shell
docker run -ti --rm -v "$PWD":/k8sdocs -p 4000:4000 gcr.io/google-samples/k8sdocs:1.0
```
The site will then be viewable at [http://localhost:4000/](http://localhost:4000/).
Or, you can [follow the instructions](/editdocs/) for running a from-scratch staging server, which is both the most performant option and the biggest pain to get set up.
## Writing Docs Using Templates
### Types of Templates
- Concept Template
- Introduce K8s Api Objects e.g. Pod
- Task Template
- Step-by-step guide for "Doing X".
- Useful for breaking down various ways of configuring Concepts into sub-topics
- Landing Pages Template
- Collection of click-able cards on a grid
- Useful for directing users to actual content from a visual Table of Contents
## Concept Overview Template Details
A concept overview covers the most essential, important information about core
Kubernetes concepts and features. Examples of Concepts include `Pod`,
`Deployment`, `Service`, etc.
### Reference Examples
- [Link to Example Template: Source](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/docsv2/docs/pod/index.md)
- [Link to Example Template: Rendered](http://k8sdocs.github.io/docs/pod/)
### Usage
### Creating the file
To create a new concept overview page, create a new directory with the concept
name under the docs directory and an index.md file.
e.g. `docs/your-concept-name/index.md`.
### Adding the page to navigation
Once your page is saved, somewhere in the `/docs/` directory, add a reference to the `concepts.yml` file under `/_data/` so that it will appear in the left-hand navigation of the site. This is also where you add a title to the page.
### Adding the Template sections
- concept: the concept name e.g. Pod
- what_is: one sentence description the function / role of the concept. Diagrams are helpful.
- when_to_use: disambiguate when to use this vs alternatives
- when_not_to_use: highlight common anti-patterns
- status: how to get the status for this object using kubectl
- usage: example yaml
- template: include the template at the end
### Tags structure
- `glossary:` a brief (~140 character) definition of what this concept is.
- `object_rankings:` associates the page with API objects/functions.
- `concept_rankings:` associates the page with Kubernetes concepts.
- `command_rankings:` associates the page with CLI commands
In each case, the association is ranked. If ranked "1," the topic will surface as a "Core Topic" (of high importance) on various associated pages. If ranked "2," the topic will be grouped under "Advanced Topics," which are deemed less essential.
Only ranks 1 and 2 are supported.
Tags are mandatory and should be thorough; they are the connective tissue of the site. To see them in action, [visit our sitemap](http://k8sdocs.github.io/docs/sitemap/).
```liquid{% raw %}
---
glossary: A pod is the vehicle for running containers in Kubernetes.
object_rankings:
- object: pod
rank: 1
concept_rankings:
- concept: pod
rank: 1
command_rankings:
- command: kubect describe
rank: 1
- command: kubectl get
rank: 1
---
{% capture concept %} concept-name-here {% endcapture %}
{% capture what_is %} description-of-concept-here {% endcapture %}
{% capture when_to_use %} when-to-use-here {% endcapture %}
{% capture when_not_to_use %} anti-patterns-here {% endcapture %}
{% capture status %} how-to-get-with-kubectl-here {% endcapture %}
{% capture usage %} yaml-config-usage-here {% endcapture %}
{% include templates/concept-overview.md %}
{% endraw %}```
## Task Template Details
A task page offers step-by-step instructions for completing a task with Kubernetes. **A task page should be narrowly focused on task completion and not delve into concepts or reference information.**
### Example
- [Link to Example Template: Source](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/docsv2/docs/tasks/explicitly-scheduling-pod-node.md)
- [Link to Example Template: Rendered](http://k8sdocs.github.io/docs/tasks/explicitly-scheduling-pod-node/)
### Usage
### Creating the file
To create a new task page, create a file under docs/tasks/task-name.
e.g. `docs/tasks/your-task-name`.
Task filenames should match the title, chaining words with dashes in all lowercase, omitting articles and prepositions. For example, the topic "Explictly Scheduling a Pod on a Node" is stored in file `/docs/tasks/explicitly-scheduling-pod-node.md`.
### Adding the page to navigation
Add a reference to the `tasks.yml` file under `/_data/` so that it will appear in the left-hand navigation of the site. This is also where you add a title to the page.
### Adding the Template sections
- metadata: structured description of the doc content
- purpose: one sentence description of the task and motivation
- recommended_background: List of Concepts referenced or other Tasks, Tutorials that provide needed context
- set_by_step: Add multiple sections. 1 per step in the task.
- template: include the template at the end
### Tags structure
- `object_rankings:` associates the page with API objects/functions.
- `concept_rankings:` associates the page with Kubernetes concepts.
- `command_rankings:` associates the page with CLI commands
In each case, the association is ranked. If ranked "1," the topic will surface as a "Core Topic" (of high importance) on various associated pages. If ranked "2," the topic will be grouped under "Advanced Topics," which are deemed less essential.
Only ranks 1 and 2 are supported.
Tags are mandatory and should be thorough; they are the connective tissue of the site. To see them in action, [visit our sitemap](http://k8sdocs.github.io/docs/sitemap/).
```liquid{% raw %}
---
object_rankings:
- object: nodeAffinity
rank: 1
- object: nodeSelector
rank: 2
concept_rankings:
- concept: node
rank: 1
- concept: pod
rank: 1
command_rankings:
- command: kubectl label
rank: 1
- command: kubectl get
rank: 2
---
{% capture purpose %} task-description-here {% endcapture %}
{% capture recommended_background %} prereq-reading-here {% endcapture %}
{% capture step_by_step %} single-step-here {% endcapture %}
{% include templates/task.md %}
{% endraw %}```
## Landing Pages
Landing pages are a set of clickable "cards" arranged in a grid. Each card has a heading and description, and optioninall, a thumbnail image. They are meant to be index pages that quickly forward users on to deeper content.
### Demos
- [Link to Example Landing Page](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/templatedemos/landingpage.md)
- [Link to Rendered Landing Page](landingpage/)
### Usage
To use this template, create a new file with these contents. Essentially, you declare the cards you want by inserting the following YAML structure in the front-matter YAML section at the top of the page, and the body of the page just has the include statement.
```yaml
---
cards:
- progression: no #"yes" = display cards as linearly progressing
- card:
title: Mean Stack
image: /images/docs/meanstack/image_0.png
description: Lorem ipsum dolor it verberum.
# repeat -card: items as necessary
---
{% raw %}{% include templates/landing-page.md %}{% endraw %}
```
### Adding page to navigation
Once your page is saved, somewhere in the `/docs/` directory, add a reference to the appropriate .yml file under `/_data/` so that it will appear in the left-hand navigation of the site. This is also where you add a title to the page.
## kubectl yaml
You probably shouldn't be using this, but we also have templates which consume YAML files that are generated by the Kubernetes authors. These are turned into pages which display the reference information for the various CLI tools.
### Demos
- [Link to Example Template](https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/docsv2/docs/kubectl/kubectl_api-versions.md)
- [Link to Rendered Template](http://k8sdocs.github.io/docs/kubectl/kubectl_api-versions/)
### Adding page to navigation
Once your page is saved, somewhere in the `/docs/` directory, add a reference to the `reference.yml` file under `/_data/` so that it will appear in the left-hand navigation of the site. This is also where you add a title to the page.
docs/templatedemos/kubectl.md
-7
View File
@@ -1,7 +0,0 @@
---
assignees:
- bgrant0607
---
{% capture command %}kubectl_annotate{% endcapture %}
{% include templates/kubectl.md %}
docs/templatedemos/landingpage.md
-53
View File
@@ -1,53 +0,0 @@
---
cards:
- progression: no
- card:
title: Mean Stack
image: /images/docs/meanstack/image_0.png
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Guestbook + Redis
image: /images/docs/redis.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Cassandra
image: /images/docs/cassandra.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: WordPress + MySQL
image: /images/docs/wordpress.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Mean Stack
image: /images/docs/meanstack/image_0.png
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Guestbook + Redis
image: /images/docs/redis.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Cassandra
image: /images/docs/cassandra.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: WordPress + MySQL
image: /images/docs/wordpress.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Mean Stack
image: /images/docs/meanstack/image_0.png
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Guestbook + Redis
image: /images/docs/redis.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: Cassandra
image: /images/docs/cassandra.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
- card:
title: WordPress + MySQL
image: /images/docs/wordpress.svg
description: Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum. Lorem ipsum dolor it verberum.
---
{% include templates/landing-page.md %}
docs/templatedemos/partial.md
-6
View File
@@ -1,6 +0,0 @@
---
---
{% assign concept="Replication Controller" %}
{% include templates/concept-overview.md %}
docs/templatedemos/task.md
-62
View File
@@ -1,62 +0,0 @@
---
---
# Doing a thing with a thing
{% capture purpose %}
This document teaches you how to do a thing.
{% endcapture %}
{% capture recommended_background %}
In order to do a thing, you must be familiar with the following:
- [Thing 1](/foo/)
- [Thing 2](/bar/)
{% endcapture %}
{% capture step_by_step %}
Here's how to do a thing with a thing.
#### 1. Prepare the thing
Lorem ipsum dolor it verberum.
#### 2. Run the thing command
Lorem ipsum dolor it verberum.
#### 3. Create the thing.yaml file
Lorem ipsum dolor it verberum.
```yaml
# Creates three nginx replicas
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
```
#### 4. ???
Lorem ipsum dolor it verberum.
#### 5. Profit!
Lorem ipsum dolor it verberum.
{% endcapture %}
{% include templates/task.md %}
docs/tutorials/index.md
+6 -1
View File
@@ -1,7 +1,10 @@
---
---
The Tutorials section of the Kubernetes documentation is a work in progress.
This section of the Kubernetes documentation contains tutorials.
A tutorial shows how to accomplish a goal that is larger than a single
[task](/docs/tasks/). Typically a tutorial has several sections,
each of which has a sequence of steps.
#### Kubernetes Basics
@@ -19,6 +22,8 @@ The Tutorials section of the Kubernetes documentation is a work in progress.
* [StatefulSet Basics](/docs/tutorials/stateful-application/basic-stateful-set/)
* [Running a Single-Instance Stateful Application](/docs/tutorials/stateful-application/run-stateful-application/)
### What's next
If you would like to write a tutorial, see
docs/tutorials/kubernetes-basics/public/images/module_01_cluster.svg
+177 -405
View File
@@ -1,9 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 19.2.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 476.1 385.3" style="enable-background:new 0 0 476.1 385.3;" xml:space="preserve">
<style type="text/css">
.st0{fill:#FFFFFF;stroke:#006DE9;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
<?xml version="1.0"?>
<svg width="476.1" height="385.3" xmlns="http://www.w3.org/2000/svg" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<style type="text/css">.st0{fill:#FFFFFF;stroke:#006DE9;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
.st1{fill:#FFFFFF;stroke:#006DE9;stroke-width:6;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
.st2{fill:#FFFFFF;stroke:#326DE6;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
.st3{opacity:0.71;fill:#326CE6;}
@@ -66,402 +63,177 @@
.st60{fill:none;stroke:#06F7C9;stroke-width:2;stroke-miterlimit:10;stroke-dasharray:2.724,1.816;}
.st61{fill:#011F38;stroke:#414042;stroke-width:0.3;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
.st62{fill:none;stroke:#011F38;stroke-width:0.3;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
.st63{fill:none;stroke:#011F38;stroke-width:0.2813;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
</style>
<symbol id="master_x5F_level1_1_" viewBox="-68.6 -66.9 137.2 133.9">
<g>
<g>
<line class="st0" x1="0" y1="-11.1" x2="0" y2="0.7"/>
<line class="st0" x1="5.9" y1="-5.2" x2="-5.9" y2="-5.2"/>
</g>
<polygon class="st1" points="-29.2,-63.9 -65.6,-18.3 -52.6,38.6 0,63.9 52.6,38.6 65.6,-18.3 29.2,-63.9 "/>
</g>
</symbol>
<symbol id="node_high_level" viewBox="-81 -93 162 186.1">
<polygon class="st2" points="-80,-46 -80,46 0,92 80,46 80,-46 0,-92 "/>
<g id="Isolation_Mode_3_">
</g>
</symbol>
<symbol id="node_x5F_empty" viewBox="-87.5 -100.6 175.1 201.1">
<use xlink:href="#node_high_level" width="162" height="186.1" id="XMLID_201_" x="-81" y="-93" transform="matrix(1.0808 0 0 1.0808 -3.292006e-05 -3.749943e-05)" style="overflow:visible;"/>
<g>
<polygon class="st3" points="76.8,-28.1 -14,-80.3 0,-88.3 76.7,-44.4 "/>
<polygon class="st4" points="76.8,-28.1 32.1,-53.8 38.8,-66.1 76.7,-44.4 "/>
</g>
</symbol>
<symbol id="node_x5F_new" viewBox="-87.6 -101 175.2 202">
<polygon class="st5" points="0,-100 -86.6,-50 -86.6,50 0,100 86.6,50 86.6,-50 "/>
<polygon class="st6" points="-86.6,-20.2 -86.6,-50 0,-100 25.8,-85.1 "/>
<polygon class="st7" points="-40.8,-70.7 -32.9,-57 15.7,-85.1 0,-94.3 "/>
<text transform="matrix(0.866 -0.5 -0.5 -0.866 -33.9256 -70.7388)" class="st8" style="font-family:'RobotoSlab-Regular'; font-size:11.3632px;">Docker</text>
<text transform="matrix(0.866 -0.5 -0.5 -0.866 -76.0668 -46.4087)" class="st8" style="font-family:'RobotoSlab-Regular'; font-size:11.3632px;">Kubelt</text>
</symbol>
<g id="CLUSTER">
<g id="XMLID_296_" class="st9">
<g>
<linearGradient id="SVGID_1_" gradientUnits="userSpaceOnUse" x1="28.6348" y1="185.2931" x2="343.0902" y2="185.2931">
<stop offset="0" style="stop-color:#326DE6"/>
<stop offset="1" style="stop-color:#10FFC6"/>
</linearGradient>
<polygon class="st10" points="311.9,92.7 343.1,229.2 255.8,338.6 115.9,338.6 28.6,229.2 59.8,92.7 185.9,32 "/>
</g>
</g>
</g>
<g id="master">
<use xlink:href="#master_x5F_level1_1_" width="137.2" height="133.9" x="-68.6" y="-66.9" transform="matrix(0.4 0 0 -0.4 185.8606 187.2514)" style="overflow:visible;"/>
<g id="master_x5F_level1">
</g>
</g>
<g id="Node">
<g id="Node_x5F_level3_x5F_1">
<g id="Isolation_Mode">
</g>
</g>
<polygon class="st16" points="182.7,139.9 147.1,160.4 111.4,139.9 111.4,98.7 147.1,78.2 182.7,98.7 "/>
<polygon class="st13" points="129.3,150.2 147.1,160.4 182.7,139.9 182.7,119.3 "/>
<polygon class="st7" points="163.5,147.7 157.4,137.2 179.9,124.2 179.9,138.3 "/>
<g>
<path class="st8" d="M162.5,139.3c0.4-0.3,0.9-0.3,1.3-0.2c0.4,0.1,0.8,0.4,1,0.8l0.3,0.5c0.3,0.4,0.3,0.9,0.2,1.3
c-0.1,0.4-0.4,0.8-0.8,1l-1.4,0.8l-0.2-0.3l0.3-0.3l-1.6-2.7l-0.4,0.2l-0.2-0.3l0.4-0.2L162.5,139.3z M162.1,140l1.6,2.7l0.6-0.3
c0.3-0.2,0.5-0.4,0.6-0.8c0.1-0.3,0-0.6-0.2-1l-0.3-0.5c-0.2-0.3-0.4-0.5-0.7-0.6c-0.3-0.1-0.6-0.1-0.9,0.1L162.1,140z"/>
<path class="st8" d="M165.7,140.3c-0.2-0.4-0.3-0.8-0.2-1.1c0.1-0.4,0.3-0.6,0.6-0.9c0.4-0.2,0.7-0.2,1.1-0.1
c0.3,0.1,0.6,0.4,0.9,0.8l0,0.1c0.2,0.4,0.3,0.8,0.2,1.1c-0.1,0.4-0.3,0.6-0.6,0.8c-0.4,0.2-0.7,0.2-1.1,0.1
C166.2,141,165.9,140.8,165.7,140.3L165.7,140.3z M166.2,140.1c0.2,0.3,0.3,0.5,0.6,0.6c0.2,0.1,0.4,0.1,0.7,0
c0.2-0.1,0.3-0.3,0.4-0.6c0-0.2-0.1-0.5-0.2-0.8l0-0.1c-0.2-0.3-0.3-0.5-0.6-0.6c-0.2-0.1-0.4-0.1-0.7,0c-0.2,0.1-0.3,0.3-0.4,0.6
C165.9,139.5,166,139.8,166.2,140.1L166.2,140.1z"/>
<path class="st8" d="M170.1,139.1c0.2-0.1,0.3-0.2,0.3-0.4c0.1-0.2,0.1-0.3,0-0.5l0.4-0.2l0,0c0.1,0.2,0.2,0.5,0,0.8
c-0.1,0.3-0.3,0.5-0.6,0.7c-0.4,0.2-0.7,0.3-1.1,0.1c-0.3-0.1-0.6-0.4-0.8-0.7l-0.1-0.1c-0.2-0.4-0.3-0.7-0.2-1.1s0.3-0.6,0.6-0.9
c0.2-0.1,0.4-0.2,0.6-0.2c0.2,0,0.4,0,0.6,0l0.3,0.6l-0.4,0.2l-0.3-0.3c-0.1,0-0.2,0-0.3,0c-0.1,0-0.2,0.1-0.3,0.1
c-0.2,0.1-0.4,0.3-0.4,0.6c0,0.2,0.1,0.5,0.2,0.7l0.1,0.1c0.2,0.3,0.3,0.5,0.5,0.6C169.7,139.2,169.9,139.2,170.1,139.1z"/>
<path class="st8" d="M169.8,135.2l-0.2-0.3l0.9-0.5l1.2,2.1l0.3-0.2l0.2-1l-0.3,0.1l-0.2-0.3l1.1-0.6l0.2,0.3l-0.3,0.2l-0.2,1.2
l1.4,0.6l0.3-0.1l0.2,0.3l-1.1,0.6l-0.2-0.3l0.2-0.2l-1.2-0.5l-0.3,0.2l0.5,0.8l0.4-0.2l0.2,0.3l-1.2,0.7l-0.2-0.3l0.3-0.3l-1.7-3
L169.8,135.2z"/>
<path class="st8" d="M175.9,136.3c-0.4,0.2-0.7,0.3-1.1,0.1s-0.6-0.4-0.9-0.7l-0.1-0.1c-0.2-0.4-0.3-0.7-0.2-1.1
c0.1-0.4,0.3-0.6,0.6-0.8c0.3-0.2,0.7-0.2,1-0.1s0.5,0.3,0.7,0.7l0.2,0.3l-1.7,1l0,0c0.1,0.2,0.3,0.4,0.5,0.5
c0.2,0.1,0.4,0.1,0.6-0.1c0.2-0.1,0.3-0.2,0.4-0.3c0.1-0.1,0.2-0.2,0.2-0.4l0.4,0.2c0,0.1-0.1,0.3-0.2,0.4
C176.3,136,176.1,136.1,175.9,136.3z M174.5,134c-0.2,0.1-0.3,0.2-0.3,0.4s0,0.4,0.1,0.6l0,0l1.2-0.7l0-0.1
c-0.1-0.2-0.2-0.3-0.4-0.4C174.9,133.9,174.7,133.9,174.5,134z"/>
<path class="st8" d="M176,133.1l-0.2-0.3l0.8-0.5l0.3,0.3c0-0.2,0-0.3,0.1-0.5c0.1-0.1,0.2-0.2,0.3-0.3c0,0,0.1,0,0.1-0.1
c0,0,0.1,0,0.1,0l0.2,0.5l-0.3,0.1c-0.1,0.1-0.2,0.2-0.3,0.3c-0.1,0.1-0.1,0.2-0.1,0.4l0.8,1.5l0.4-0.2l0.2,0.3l-1.2,0.7l-0.2-0.3
l0.3-0.3l-1.1-1.8L176,133.1z"/>
</g>
<g>
<path class="st8" d="M141.1,151.8l-0.2-0.3l0.9-0.5l1.2,2.1l0.3-0.2l0.2-1l-0.3,0.1l-0.2-0.3l1.1-0.6l0.2,0.3l-0.3,0.2l-0.2,1.2
l1.4,0.6l0.3-0.1l0.2,0.3l-1.1,0.6l-0.2-0.3l0.2-0.2l-1.2-0.5l-0.3,0.2l0.5,0.8l0.4-0.2l0.2,0.3l-1.2,0.7l-0.2-0.3l0.3-0.3l-1.7-3
L141.1,151.8z"/>
<path class="st8" d="M147.6,152c0,0.2,0,0.3-0.1,0.5c-0.1,0.1-0.2,0.3-0.4,0.4c-0.3,0.2-0.5,0.2-0.8,0.1c-0.3-0.1-0.5-0.3-0.7-0.7
l-0.7-1.1l-0.3,0.1l-0.2-0.3l0.3-0.2l0.5-0.3l0.9,1.5c0.2,0.3,0.3,0.4,0.4,0.5s0.3,0,0.5-0.1c0.2-0.1,0.3-0.2,0.4-0.3
c0.1-0.1,0.1-0.3,0.1-0.4l-0.9-1.5l-0.4,0.1L146,150l0.3-0.2l0.5-0.3l1.3,2.2l0.3-0.1l0.2,0.3l-0.7,0.4L147.6,152z"/>
<path class="st8" d="M150.6,149.2c0.2,0.4,0.3,0.7,0.2,1c0,0.3-0.2,0.6-0.5,0.8c-0.2,0.1-0.3,0.1-0.5,0.2c-0.2,0-0.3,0-0.5-0.1
l0.1,0.4l-0.4,0.2l-1.9-3.3l-0.4,0.2l-0.2-0.3l0.9-0.5l0.8,1.4c0-0.2,0.1-0.3,0.1-0.4c0.1-0.1,0.2-0.2,0.4-0.3
c0.3-0.2,0.6-0.2,1,0S150.4,148.7,150.6,149.2L150.6,149.2z M150.1,149.4c-0.2-0.3-0.4-0.5-0.6-0.6c-0.2-0.1-0.4-0.1-0.7,0
c-0.1,0.1-0.2,0.2-0.3,0.3c-0.1,0.1-0.1,0.3-0.1,0.4l0.7,1.1c0.1,0.1,0.3,0.1,0.4,0.1c0.1,0,0.3,0,0.4-0.1
c0.2-0.1,0.3-0.3,0.4-0.5C150.4,149.9,150.3,149.7,150.1,149.4L150.1,149.4z"/>
<path class="st8" d="M153,149.5c-0.4,0.2-0.7,0.3-1.1,0.1s-0.6-0.4-0.9-0.7l-0.1-0.1c-0.2-0.4-0.3-0.7-0.2-1.1
c0.1-0.4,0.3-0.6,0.6-0.8c0.3-0.2,0.7-0.2,1-0.1s0.5,0.3,0.7,0.7l0.2,0.3l-1.7,1v0c0.1,0.2,0.3,0.4,0.5,0.5
c0.2,0.1,0.4,0.1,0.6-0.1c0.2-0.1,0.3-0.2,0.4-0.3c0.1-0.1,0.2-0.2,0.2-0.4l0.4,0.2c0,0.1-0.1,0.3-0.2,0.4S153.2,149.3,153,149.5z
M151.6,147.2c-0.2,0.1-0.3,0.2-0.3,0.4c0,0.2,0,0.4,0.1,0.6l0,0l1.2-0.7l0-0.1c-0.1-0.2-0.2-0.3-0.4-0.4S151.8,147.1,151.6,147.2
z"/>
<path class="st8" d="M152.4,145.2l-0.2-0.3l0.9-0.5l1.9,3.3l0.4-0.2l0.2,0.3l-1.2,0.7l-0.2-0.3l0.3-0.3l-1.7-3L152.4,145.2z"/>
<path class="st8" d="M157.1,147.1c-0.4,0.2-0.7,0.3-1.1,0.1s-0.6-0.4-0.9-0.7l-0.1-0.1c-0.2-0.4-0.3-0.7-0.2-1.1
c0.1-0.4,0.3-0.6,0.6-0.8c0.3-0.2,0.7-0.2,1-0.1s0.5,0.3,0.7,0.7l0.2,0.3l-1.7,1v0c0.1,0.2,0.3,0.4,0.5,0.5
c0.2,0.1,0.4,0.1,0.6-0.1c0.2-0.1,0.3-0.2,0.4-0.3c0.1-0.1,0.2-0.2,0.2-0.4l0.4,0.2c0,0.1-0.1,0.3-0.2,0.4S157.3,147,157.1,147.1z
M155.7,144.8c-0.2,0.1-0.3,0.2-0.3,0.4s0,0.4,0.1,0.6l0,0l1.2-0.7l0-0.1c-0.1-0.2-0.2-0.3-0.4-0.4
C156.1,144.7,155.9,144.7,155.7,144.8z"/>
<path class="st8" d="M157.5,142.6l0.4,0.6l0.5-0.3l0.2,0.3l-0.5,0.3l0.9,1.6c0.1,0.1,0.1,0.2,0.2,0.2c0.1,0,0.2,0,0.2,0
c0,0,0.1-0.1,0.1-0.1c0,0,0.1-0.1,0.1-0.1l0.2,0.3c0,0.1-0.1,0.1-0.2,0.2c-0.1,0.1-0.2,0.1-0.2,0.2c-0.2,0.1-0.4,0.1-0.6,0.1
c-0.2,0-0.3-0.2-0.5-0.4l-0.9-1.6l-0.4,0.2l-0.2-0.3l0.4-0.2l-0.4-0.6L157.5,142.6z"/>
</g>
<polygon class="st16" points="225.1,160.4 189.4,139.9 189.4,98.7 225.1,78.2 260.7,98.7 260.7,139.9 "/>
<polygon class="st13" points="189.4,119.3 189.4,139.9 225.1,160.4 242.9,150.2 "/>
<polygon class="st7" points="208.7,147.7 214.8,137.2 237.3,150.2 225.1,157.2 "/>
<g>
<path class="st8" d="M215.5,142.6c0.5,0.3,0.7,0.6,0.8,1c0.1,0.4,0,0.9-0.2,1.3l-0.3,0.5c-0.3,0.4-0.6,0.7-1,0.8
c-0.4,0.1-0.9,0-1.3-0.2l-1.4-0.8l0.2-0.3l0.4,0.2l1.6-2.7l-0.3-0.3l0.2-0.3l0.4,0.2L215.5,142.6z M214.7,142.6l-1.6,2.7l0.6,0.3
c0.3,0.2,0.6,0.2,0.9,0.1c0.3-0.1,0.6-0.3,0.7-0.6l0.3-0.5c0.2-0.3,0.2-0.6,0.2-1c-0.1-0.3-0.3-0.6-0.6-0.8L214.7,142.6z"/>
<path class="st8" d="M216.2,145.9c0.2-0.4,0.5-0.6,0.8-0.8c0.3-0.1,0.7-0.1,1,0.1c0.4,0.2,0.6,0.5,0.6,0.9c0.1,0.4,0,0.7-0.2,1.1
l0,0.1c-0.2,0.4-0.5,0.6-0.8,0.8c-0.3,0.1-0.7,0.1-1-0.1c-0.4-0.2-0.6-0.5-0.6-0.9C215.9,146.7,216,146.3,216.2,145.9L216.2,145.9
z M216.7,146.2c-0.2,0.3-0.2,0.5-0.2,0.8c0,0.2,0.1,0.4,0.4,0.6c0.2,0.1,0.4,0.1,0.7,0c0.2-0.1,0.4-0.3,0.6-0.6l0-0.1
c0.2-0.3,0.2-0.5,0.2-0.8c0-0.2-0.1-0.4-0.4-0.6c-0.2-0.1-0.4-0.1-0.7,0C217,145.7,216.8,145.9,216.7,146.2L216.7,146.2z"/>
<path class="st8" d="M219.5,149.1c0.2,0.1,0.3,0.1,0.5,0.1c0.2,0,0.3-0.1,0.4-0.2l0.4,0.2l0,0c-0.1,0.2-0.3,0.4-0.6,0.4
c-0.3,0.1-0.6,0-0.9-0.2c-0.4-0.2-0.6-0.5-0.6-0.9c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.8-0.7s0.7-0.1,1.1,0.1
c0.2,0.1,0.4,0.3,0.5,0.4s0.2,0.3,0.2,0.5l-0.3,0.6l-0.4-0.2l0.1-0.5c0-0.1-0.1-0.2-0.1-0.3c-0.1-0.1-0.2-0.2-0.3-0.2
c-0.2-0.1-0.5-0.2-0.7,0c-0.2,0.1-0.4,0.3-0.5,0.6l-0.1,0.1c-0.2,0.3-0.2,0.5-0.2,0.7C219.2,148.8,219.3,149,219.5,149.1z"/>
<path class="st8" d="M222.7,146.8l0.2-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9-0.4l-0.2-0.2l0.2-0.3l1.1,0.6l-0.2,0.3l-0.3-0.1l-1.1,0.4
l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1-0.6l0.2-0.3l0.3,0.1l-0.2-1.2l-0.3-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3
L222.7,146.8z"/>
<path class="st8" d="M224.8,152.7c-0.4-0.2-0.6-0.5-0.6-0.8s0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7s0.7-0.1,1,0.1
c0.3,0.2,0.5,0.5,0.6,0.8c0.1,0.3,0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7c0,0.2,0.2,0.4,0.4,0.5
c0.2,0.1,0.3,0.1,0.5,0.2c0.1,0,0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0S225.1,152.8,224.8,152.7z M226.1,150.3
c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.4,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C226.4,150.6,226.3,150.4,226.1,150.3z"/>
<path class="st8" d="M227.6,151.2l0.2-0.3l0.8,0.5l-0.2,0.4c0.1-0.1,0.3-0.1,0.4-0.2c0.1,0,0.3,0,0.4,0.1c0,0,0.1,0,0.1,0.1
c0,0,0.1,0.1,0.1,0.1l-0.3,0.4l-0.3-0.2c-0.1-0.1-0.2-0.1-0.4-0.1c-0.1,0-0.2,0.1-0.3,0.1l-0.8,1.5l0.3,0.3l-0.2,0.3l-1.2-0.7
l0.2-0.3l0.4,0.2l1.1-1.8L227.6,151.2z"/>
</g>
<g>
<path class="st8" d="M194,130.3l0.2-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9-0.4l-0.2-0.2l0.2-0.3l1.1,0.6l-0.2,0.3l-0.3-0.1l-1.1,0.4
l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1-0.6l0.2-0.3l0.3,0.1l-0.2-1.2l-0.3-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3
L194,130.3z"/>
<path class="st8" d="M197.1,136.1c-0.2,0.1-0.3,0.1-0.5,0.1c-0.2,0-0.3,0-0.5-0.1c-0.3-0.2-0.4-0.4-0.5-0.6c-0.1-0.3,0-0.6,0.2-1
l0.7-1.1l-0.3-0.2l0.2-0.3l0.3,0.2l0.5,0.3l-0.9,1.5c-0.2,0.3-0.2,0.5-0.2,0.6c0,0.1,0.1,0.3,0.3,0.4c0.2,0.1,0.3,0.1,0.5,0.1
c0.1,0,0.3-0.1,0.4-0.1l0.9-1.5l-0.3-0.3l0.2-0.3l0.3,0.2l0.5,0.3l-1.3,2.2l0.3,0.2l-0.2,0.3l-0.7-0.4L197.1,136.1z"/>
<path class="st8" d="M201,137.3c-0.2,0.4-0.5,0.6-0.8,0.7s-0.6,0.1-0.9-0.1c-0.2-0.1-0.3-0.2-0.4-0.3c-0.1-0.1-0.1-0.3-0.1-0.5
l-0.2,0.3l-0.4-0.2l1.9-3.3l-0.3-0.3l0.2-0.3l0.9,0.5l-0.8,1.4c0.1-0.1,0.3-0.1,0.5-0.1s0.3,0.1,0.5,0.2c0.3,0.2,0.5,0.5,0.5,0.8
S201.3,136.8,201,137.3L201,137.3z M200.6,136.9c0.2-0.3,0.3-0.6,0.3-0.8c0-0.2-0.1-0.4-0.3-0.6c-0.1-0.1-0.3-0.1-0.4-0.1
c-0.1,0-0.3,0.1-0.4,0.1l-0.7,1.1c0,0.2,0,0.3,0.1,0.4c0.1,0.1,0.2,0.2,0.3,0.3c0.2,0.1,0.4,0.1,0.6,0
C200.2,137.4,200.4,137.2,200.6,136.9L200.6,136.9z"/>
<path class="st8" d="M202,139.4c-0.4-0.2-0.6-0.5-0.6-0.8c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7
c0.4-0.1,0.7-0.1,1,0.1c0.3,0.2,0.5,0.5,0.6,0.8c0.1,0.3,0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7
c0,0.2,0.2,0.4,0.4,0.5c0.2,0.1,0.3,0.1,0.5,0.2c0.1,0,0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0C202.4,139.6,202.2,139.6,202,139.4z
M203.2,137.1c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.3,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C203.5,137.4,203.4,137.2,203.2,137.1z"/>
<path class="st8" d="M205.3,136.8l0.2-0.3l0.9,0.5l-1.9,3.3l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3L205.3,136.8z"/>
<path class="st8" d="M206.1,141.8c-0.4-0.2-0.6-0.5-0.6-0.8c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7
c0.4-0.1,0.7-0.1,1,0.1c0.3,0.2,0.5,0.5,0.6,0.8c0.1,0.3,0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7
c0,0.2,0.2,0.4,0.4,0.5c0.2,0.1,0.3,0.1,0.5,0.2c0.1,0,0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0C206.5,142,206.3,141.9,206.1,141.8z
M207.3,139.5c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.3,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C207.6,139.7,207.5,139.6,207.3,139.5z"/>
<path class="st8" d="M210.2,139.9l-0.4,0.6l0.5,0.3l-0.2,0.3l-0.5-0.3l-0.9,1.6c-0.1,0.1-0.1,0.2-0.1,0.3c0,0.1,0.1,0.1,0.2,0.2
c0,0,0.1,0,0.1,0.1c0.1,0,0.1,0,0.1,0.1l-0.1,0.4c-0.1,0-0.1,0-0.2,0c-0.1,0-0.2-0.1-0.3-0.1c-0.2-0.1-0.3-0.3-0.4-0.4
c0-0.2,0-0.4,0.1-0.6l0.9-1.6l-0.4-0.2l0.2-0.3l0.4,0.2l0.4-0.6L210.2,139.9z"/>
</g>
<polygon class="st16" points="147.1,296.4 111.4,275.9 111.4,234.7 147.1,214.1 182.7,234.7 182.7,275.9 "/>
<polygon class="st13" points="129.3,224.4 147.1,214.1 182.7,234.7 182.7,255.3 "/>
<polygon class="st7" points="163.5,226.9 157.4,237.4 179.9,250.4 179.9,236.3 "/>
<g>
<path class="st8" d="M164.2,232.3c0.5,0.3,0.7,0.6,0.8,1c0.1,0.4,0,0.9-0.2,1.3l-0.3,0.5c-0.3,0.4-0.6,0.7-1,0.8
c-0.4,0.1-0.9,0-1.3-0.2l-1.4-0.8l0.2-0.3l0.4,0.2l1.6-2.7l-0.3-0.3l0.2-0.3l0.4,0.2L164.2,232.3z M163.4,232.3l-1.6,2.7l0.6,0.3
c0.3,0.2,0.6,0.2,0.9,0.1c0.3-0.1,0.6-0.3,0.7-0.6l0.3-0.5c0.2-0.3,0.3-0.6,0.2-1c-0.1-0.3-0.3-0.6-0.6-0.8L163.4,232.3z"/>
<path class="st8" d="M164.9,235.6c0.2-0.4,0.5-0.6,0.8-0.8c0.3-0.1,0.7-0.1,1,0.1c0.4,0.2,0.6,0.5,0.6,0.9c0.1,0.4,0,0.7-0.2,1.1
l0,0.1c-0.2,0.4-0.5,0.6-0.8,0.8c-0.3,0.1-0.7,0.1-1-0.1c-0.4-0.2-0.6-0.5-0.6-0.9C164.6,236.4,164.7,236,164.9,235.6L164.9,235.6
z M165.4,235.9c-0.2,0.3-0.2,0.5-0.2,0.8c0,0.2,0.1,0.4,0.4,0.6c0.2,0.1,0.4,0.1,0.7,0c0.2-0.1,0.4-0.3,0.6-0.6l0-0.1
c0.2-0.3,0.2-0.5,0.2-0.8c0-0.2-0.1-0.4-0.4-0.6c-0.2-0.1-0.4-0.1-0.7,0C165.8,235.4,165.6,235.6,165.4,235.9L165.4,235.9z"/>
<path class="st8" d="M168.2,238.8c0.2,0.1,0.3,0.1,0.5,0.1c0.2,0,0.3-0.1,0.4-0.2l0.4,0.2l0,0c-0.1,0.2-0.3,0.4-0.6,0.4
c-0.3,0.1-0.6,0-0.9-0.2c-0.4-0.2-0.6-0.5-0.6-0.9c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.8-0.7
c0.3-0.1,0.7-0.1,1.1,0.1c0.2,0.1,0.4,0.3,0.5,0.4s0.2,0.3,0.2,0.5L170,238l-0.4-0.2l0.1-0.5c0-0.1-0.1-0.2-0.1-0.3
c-0.1-0.1-0.2-0.2-0.3-0.2c-0.2-0.1-0.5-0.2-0.7,0c-0.2,0.1-0.4,0.3-0.5,0.6l-0.1,0.1c-0.2,0.3-0.2,0.5-0.2,0.7
C167.9,238.5,168,238.7,168.2,238.8z"/>
<path class="st8" d="M171.4,236.5l0.2-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9-0.4l-0.2-0.2l0.2-0.3l1.1,0.6l-0.2,0.3L173,239l-1.1,0.4
l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1-0.6l0.2-0.3l0.3,0.1l-0.2-1.2l-0.3-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3
L171.4,236.5z"/>
<path class="st8" d="M173.6,242.4c-0.4-0.2-0.6-0.5-0.6-0.8c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7
c0.4-0.1,0.7-0.1,1,0.1c0.3,0.2,0.5,0.5,0.6,0.8c0.1,0.3,0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7
c0,0.2,0.2,0.4,0.4,0.5c0.2,0.1,0.3,0.1,0.5,0.2s0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0C174,242.6,173.8,242.5,173.6,242.4z
M174.8,240c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.4,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C175.1,240.3,175,240.1,174.8,240z"/>
<path class="st8" d="M176.3,240.9l0.2-0.3l0.8,0.5l-0.2,0.4c0.1-0.1,0.3-0.1,0.4-0.2c0.2,0,0.3,0,0.4,0.1c0,0,0.1,0,0.1,0.1
c0,0,0.1,0.1,0.1,0.1l-0.3,0.4l-0.3-0.2c-0.1-0.1-0.2-0.1-0.4-0.1c-0.1,0-0.2,0.1-0.3,0.1l-0.8,1.5l0.3,0.3l-0.2,0.3l-1.2-0.7
l0.2-0.3l0.4,0.2l1.1-1.8L176.3,240.9z"/>
</g>
<g>
<path class="st8" d="M142.7,220l0.2-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9-0.4l-0.2-0.2l0.2-0.3l1.1,0.6l-0.2,0.3l-0.3-0.1l-1.1,0.4
l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1-0.6l0.2-0.3l0.3,0.1l-0.2-1.2l-0.3-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3
L142.7,220z"/>
<path class="st8" d="M145.8,225.8c-0.2,0.1-0.3,0.1-0.5,0.1s-0.3,0-0.5-0.1c-0.3-0.2-0.4-0.4-0.5-0.6s0-0.6,0.2-1l0.7-1.1
l-0.3-0.2l0.2-0.3l0.3,0.2l0.5,0.3l-0.9,1.5c-0.2,0.3-0.2,0.5-0.2,0.6s0.1,0.3,0.3,0.4c0.2,0.1,0.3,0.1,0.5,0.1
c0.1,0,0.3-0.1,0.4-0.1l0.9-1.5l-0.3-0.3l0.2-0.3l0.3,0.2l0.5,0.3l-1.3,2.2l0.3,0.2l-0.2,0.3l-0.7-0.4L145.8,225.8z"/>
<path class="st8" d="M149.8,227c-0.2,0.4-0.5,0.6-0.8,0.7c-0.3,0.1-0.6,0.1-0.9-0.1c-0.2-0.1-0.3-0.2-0.4-0.3
c-0.1-0.1-0.1-0.3-0.1-0.5l-0.2,0.3l-0.4-0.2l1.9-3.3l-0.3-0.3l0.2-0.3l0.9,0.5l-0.8,1.4c0.1-0.1,0.3-0.1,0.5-0.1
c0.2,0,0.3,0.1,0.5,0.2c0.3,0.2,0.5,0.5,0.5,0.8C150.1,226.1,150,226.5,149.8,227L149.8,227z M149.3,226.6
c0.2-0.3,0.3-0.6,0.3-0.8c0-0.2-0.1-0.4-0.3-0.6c-0.1-0.1-0.3-0.1-0.4-0.1c-0.1,0-0.3,0.1-0.4,0.1l-0.7,1.1c0,0.2,0,0.3,0.1,0.4
s0.2,0.2,0.3,0.3c0.2,0.1,0.4,0.1,0.6,0C149,227.1,149.1,227,149.3,226.6L149.3,226.6z"/>
<path class="st8" d="M150.7,229.1c-0.4-0.2-0.6-0.5-0.6-0.8c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7
c0.4-0.1,0.7-0.1,1,0.1c0.3,0.2,0.5,0.5,0.6,0.8s0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7s0.2,0.4,0.4,0.5
c0.2,0.1,0.3,0.1,0.5,0.2c0.1,0,0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0C151.1,229.3,150.9,229.3,150.7,229.1z M151.9,226.8
c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.4,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C152.2,227.1,152.1,226.9,151.9,226.8z"/>
<path class="st8" d="M154,226.5l0.2-0.3l0.9,0.5l-1.9,3.3l0.3,0.3l-0.2,0.3l-1.2-0.7l0.2-0.3l0.4,0.2l1.7-3L154,226.5z"/>
<path class="st8" d="M154.8,231.5c-0.4-0.2-0.6-0.5-0.6-0.8c-0.1-0.4,0-0.7,0.2-1.1l0.1-0.1c0.2-0.4,0.5-0.6,0.9-0.7
c0.4-0.1,0.7-0.1,1,0.1c0.3,0.2,0.5,0.5,0.6,0.8s0,0.6-0.2,1l-0.2,0.3l-1.7-1l0,0c-0.1,0.2-0.2,0.5-0.2,0.7s0.2,0.4,0.4,0.5
c0.2,0.1,0.3,0.1,0.5,0.2c0.1,0,0.3,0,0.4,0l0,0.4c-0.1,0-0.3,0-0.5,0C155.2,231.7,155,231.7,154.8,231.5z M156.1,229.2
c-0.2-0.1-0.3-0.1-0.5,0c-0.2,0.1-0.4,0.2-0.5,0.4l0,0l1.2,0.7l0-0.1c0.1-0.2,0.1-0.4,0.1-0.5
C156.4,229.5,156.3,229.3,156.1,229.2z"/>
<path class="st8" d="M158.9,229.6l-0.4,0.6l0.5,0.3l-0.2,0.3l-0.5-0.3l-0.9,1.6c-0.1,0.1-0.1,0.2-0.1,0.3c0,0.1,0.1,0.1,0.2,0.2
c0,0,0.1,0,0.1,0.1c0.1,0,0.1,0,0.1,0.1l-0.1,0.4c-0.1,0-0.1,0-0.2,0c-0.1,0-0.2-0.1-0.3-0.1c-0.2-0.1-0.3-0.3-0.4-0.4
c0-0.2,0-0.4,0.1-0.6l0.9-1.6l-0.4-0.2l0.2-0.3l0.4,0.2l0.4-0.6L158.9,229.6z"/>
</g>
</g>
<g id="service">
</g>
<g id="pods">
</g>
<g id="IP">
</g>
<g id="deployments">
</g>
<g id="containers_x2F_volumes">
</g>
<g id="labels_x2F_selectors">
</g>
<g id="description">
<g>
<path class="st42" d="M374.4,188.6L374.4,188.6l-2.8,6.8h-0.8l-2.8-6.8l0,0l0.1,3.5v2.5l1,0.2v0.7h-3.1v-0.7l1-0.2v-6.7l-1-0.2
v-0.7h1h1.5l2.7,6.9h0l2.7-6.9h2.4v0.7l-1,0.2v6.7l1,0.2v0.7h-3.1v-0.7l1-0.2V192L374.4,188.6z"/>
<path class="st42" d="M381.5,195.4c0-0.2-0.1-0.3-0.1-0.5s0-0.3,0-0.4c-0.2,0.3-0.5,0.5-0.8,0.7s-0.7,0.3-1.1,0.3
c-0.7,0-1.2-0.2-1.5-0.5s-0.5-0.8-0.5-1.4c0-0.6,0.2-1.1,0.7-1.4s1.2-0.5,2-0.5h1.2v-0.7c0-0.4-0.1-0.7-0.4-0.9s-0.6-0.3-1-0.3
c-0.3,0-0.5,0-0.8,0.1s-0.4,0.2-0.5,0.3l-0.1,0.7h-0.9v-1.2c0.3-0.2,0.6-0.4,1-0.6s0.9-0.2,1.3-0.2c0.7,0,1.3,0.2,1.7,0.6
s0.7,0.9,0.7,1.6v3.1c0,0.1,0,0.2,0,0.2s0,0.2,0,0.2l0.5,0.1v0.7H381.5z M379.6,194.6c0.4,0,0.7-0.1,1-0.3s0.5-0.4,0.7-0.7v-1
h-1.2c-0.5,0-0.8,0.1-1.1,0.3s-0.4,0.5-0.4,0.8c0,0.3,0.1,0.5,0.3,0.6S379.3,194.6,379.6,194.6z"/>
<path class="st42" d="M388.8,191.1h-0.9l-0.2-0.8c-0.1-0.1-0.3-0.2-0.5-0.3s-0.5-0.1-0.7-0.1c-0.4,0-0.7,0.1-0.9,0.3
s-0.3,0.4-0.3,0.7c0,0.2,0.1,0.4,0.3,0.6s0.5,0.3,1.1,0.4c0.8,0.2,1.4,0.4,1.8,0.7s0.6,0.7,0.6,1.2c0,0.6-0.2,1-0.7,1.4
s-1,0.5-1.8,0.5c-0.5,0-0.9-0.1-1.3-0.2s-0.7-0.3-1-0.5l0-1.4h0.9l0.2,0.8c0.1,0.1,0.3,0.2,0.5,0.3s0.5,0.1,0.7,0.1
c0.4,0,0.7-0.1,1-0.2s0.3-0.4,0.3-0.7c0-0.3-0.1-0.5-0.3-0.6s-0.6-0.3-1.1-0.4c-0.8-0.2-1.3-0.4-1.7-0.7s-0.6-0.7-0.6-1.2
c0-0.5,0.2-1,0.7-1.3s1-0.5,1.7-0.5c0.5,0,0.9,0.1,1.3,0.2s0.7,0.3,1,0.5L388.8,191.1z"/>
<path class="st42" d="M392.1,187.5v1.5h1.2v0.9h-1.2v3.8c0,0.3,0.1,0.5,0.2,0.6s0.3,0.2,0.5,0.2c0.1,0,0.2,0,0.3,0s0.2,0,0.3-0.1
l0.2,0.8c-0.1,0.1-0.3,0.1-0.5,0.2s-0.4,0.1-0.6,0.1c-0.5,0-0.8-0.1-1.1-0.4s-0.4-0.7-0.4-1.3v-3.8h-1v-0.9h1v-1.5H392.1z"/>
<path class="st42" d="M397.4,195.5c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3v-0.3c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,1.9-0.9
c0.9,0,1.5,0.3,1.9,0.8s0.7,1.2,0.7,2.1v0.7h-4.1l0,0c0,0.6,0.2,1.1,0.5,1.5s0.7,0.6,1.2,0.6c0.4,0,0.7-0.1,1-0.2s0.5-0.3,0.8-0.5
l0.5,0.8c-0.2,0.2-0.5,0.4-0.9,0.6S397.9,195.5,397.4,195.5z M397.2,189.9c-0.4,0-0.7,0.2-1,0.5s-0.4,0.7-0.5,1.2l0,0h2.9v-0.2
c0-0.5-0.1-0.8-0.4-1.1S397.7,189.9,397.2,189.9z"/>
<path class="st42" d="M400.9,189.8v-0.7h2l0.1,0.9c0.2-0.3,0.4-0.6,0.7-0.8s0.6-0.3,0.9-0.3c0.1,0,0.2,0,0.3,0s0.2,0,0.2,0
l-0.2,1.1l-0.7,0c-0.3,0-0.6,0.1-0.8,0.2s-0.4,0.3-0.5,0.6v3.6l1,0.2v0.7h-3.1v-0.7l1-0.2V190L400.9,189.8z"/>
</g>
<g>
<path class="st42" d="M366,230.1l1-0.2v-4.5l-1-0.2v-0.7h2l0.1,0.9c0.2-0.3,0.5-0.6,0.8-0.8s0.7-0.3,1.1-0.3
c0.7,0,1.2,0.2,1.6,0.6s0.6,1,0.6,1.9v3.1l1,0.2v0.7H370v-0.7l1-0.2v-3.1c0-0.6-0.1-1-0.3-1.2s-0.6-0.4-1-0.4
c-0.3,0-0.6,0.1-0.9,0.2s-0.5,0.4-0.6,0.7v3.7l1,0.2v0.7H366V230.1z"/>
<path class="st42" d="M373.9,227.6c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,2.1-0.9c0.9,0,1.6,0.3,2.1,0.9s0.8,1.4,0.8,2.3v0.1
c0,0.9-0.3,1.7-0.8,2.3s-1.2,0.9-2.1,0.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3V227.6z M375,227.7c0,0.7,0.1,1.2,0.4,1.7
s0.7,0.7,1.3,0.7c0.5,0,1-0.2,1.2-0.7s0.4-1,0.4-1.7v-0.1c0-0.7-0.1-1.2-0.4-1.7s-0.7-0.7-1.3-0.7s-1,0.2-1.3,0.7s-0.4,1-0.4,1.7
V227.7z"/>
<path class="st42" d="M384.9,230c-0.2,0.3-0.5,0.5-0.8,0.7s-0.6,0.2-1,0.2c-0.8,0-1.4-0.3-1.8-0.8s-0.7-1.3-0.7-2.2v-0.1
c0-1,0.2-1.8,0.7-2.5s1-0.9,1.8-0.9c0.4,0,0.7,0.1,1,0.2s0.5,0.3,0.7,0.6v-2.6l-1-0.2v-0.7h1h1.2v8.2l1,0.2v0.7h-2L384.9,230z
M381.7,227.9c0,0.6,0.1,1.1,0.4,1.5s0.7,0.6,1.2,0.6c0.3,0,0.6-0.1,0.9-0.2s0.4-0.4,0.6-0.7v-2.9c-0.1-0.3-0.3-0.5-0.6-0.6
s-0.5-0.2-0.9-0.2c-0.6,0-1,0.2-1.2,0.7s-0.4,1.1-0.4,1.8V227.9z"/>
<path class="st42" d="M390.7,230.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3v-0.3c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,1.9-0.9
c0.9,0,1.5,0.3,1.9,0.8s0.7,1.2,0.7,2.1v0.7H389l0,0c0,0.6,0.2,1.1,0.5,1.5s0.7,0.6,1.2,0.6c0.4,0,0.7-0.1,1-0.2s0.5-0.3,0.8-0.5
l0.5,0.8c-0.2,0.2-0.5,0.4-0.9,0.6S391.2,230.9,390.7,230.9z M390.5,225.3c-0.4,0-0.7,0.2-1,0.5s-0.4,0.7-0.5,1.2l0,0h2.9v-0.2
c0-0.5-0.1-0.8-0.4-1.1S391,225.3,390.5,225.3z"/>
<path class="st42" d="M397.1,232.5l1-0.2v-7l-1-0.2v-0.7h1.9l0.1,0.8c0.2-0.3,0.5-0.5,0.8-0.7s0.7-0.2,1.1-0.2
c0.8,0,1.4,0.3,1.8,0.9s0.7,1.4,0.7,2.5v0.1c0,0.9-0.2,1.7-0.7,2.2s-1,0.8-1.8,0.8c-0.4,0-0.7-0.1-1-0.2s-0.5-0.3-0.8-0.6v2.2
l1,0.2v0.7h-3.1V232.5z M402.3,227.7c0-0.7-0.1-1.3-0.4-1.8s-0.7-0.7-1.3-0.7c-0.3,0-0.6,0.1-0.8,0.2s-0.4,0.4-0.6,0.6v3.1
c0.1,0.3,0.3,0.5,0.6,0.6s0.5,0.2,0.9,0.2c0.5,0,1-0.2,1.2-0.6s0.4-0.9,0.4-1.6V227.7z"/>
<path class="st42" d="M404.5,225.2v-0.7h2l0.1,0.9c0.2-0.3,0.4-0.6,0.7-0.8s0.6-0.3,0.9-0.3c0.1,0,0.2,0,0.3,0s0.2,0,0.2,0
l-0.2,1.1l-0.7,0c-0.3,0-0.6,0.1-0.8,0.2s-0.4,0.3-0.5,0.6v3.6l1,0.2v0.7h-3.1v-0.7l1-0.2v-4.5L404.5,225.2z"/>
<path class="st42" d="M409.3,227.6c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,2.1-0.9c0.9,0,1.6,0.3,2.1,0.9s0.8,1.4,0.8,2.3v0.1
c0,0.9-0.3,1.7-0.8,2.3s-1.2,0.9-2.1,0.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3V227.6z M410.5,227.7c0,0.7,0.1,1.2,0.4,1.7
s0.7,0.7,1.3,0.7c0.5,0,1-0.2,1.2-0.7s0.4-1,0.4-1.7v-0.1c0-0.7-0.1-1.2-0.4-1.7s-0.7-0.7-1.3-0.7s-1,0.2-1.3,0.7s-0.4,1-0.4,1.7
V227.7z"/>
<path class="st42" d="M418.9,230c0.4,0,0.7-0.1,1-0.4s0.4-0.5,0.4-0.9h1l0,0c0,0.5-0.2,1-0.7,1.5s-1.1,0.6-1.8,0.6
c-0.9,0-1.6-0.3-2.1-0.9s-0.7-1.4-0.7-2.3v-0.2c0-0.9,0.2-1.7,0.7-2.3s1.2-0.9,2.1-0.9c0.5,0,1,0.1,1.4,0.3s0.7,0.4,1,0.7l0.1,1.4
h-0.9l-0.3-1c-0.1-0.1-0.3-0.2-0.5-0.3s-0.5-0.1-0.7-0.1c-0.6,0-1,0.2-1.3,0.7s-0.4,1-0.4,1.6v0.2c0,0.6,0.1,1.2,0.4,1.6
S418.3,230,418.9,230z"/>
<path class="st42" d="M425.4,230.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3v-0.3c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,1.9-0.9
c0.9,0,1.5,0.3,1.9,0.8s0.7,1.2,0.7,2.1v0.7h-4.1l0,0c0,0.6,0.2,1.1,0.5,1.5s0.7,0.6,1.2,0.6c0.4,0,0.7-0.1,1-0.2s0.5-0.3,0.8-0.5
l0.5,0.8c-0.2,0.2-0.5,0.4-0.9,0.6S426,230.9,425.4,230.9z M425.3,225.3c-0.4,0-0.7,0.2-1,0.5s-0.4,0.7-0.5,1.2l0,0h2.9v-0.2
c0-0.5-0.1-0.8-0.4-1.1S425.7,225.3,425.3,225.3z"/>
<path class="st42" d="M433.8,226.5H433l-0.2-0.8c-0.1-0.1-0.3-0.2-0.5-0.3s-0.5-0.1-0.7-0.1c-0.4,0-0.7,0.1-0.9,0.3
s-0.3,0.4-0.3,0.7c0,0.2,0.1,0.4,0.3,0.6s0.5,0.3,1.1,0.4c0.8,0.2,1.4,0.4,1.8,0.7s0.6,0.7,0.6,1.2c0,0.6-0.2,1-0.7,1.4
s-1,0.5-1.8,0.5c-0.5,0-0.9-0.1-1.3-0.2s-0.7-0.3-1-0.5l0-1.4h0.9l0.2,0.8c0.1,0.1,0.3,0.2,0.5,0.3s0.5,0.1,0.7,0.1
c0.4,0,0.7-0.1,1-0.2s0.3-0.4,0.3-0.7c0-0.3-0.1-0.5-0.3-0.6s-0.6-0.3-1.1-0.4c-0.8-0.2-1.3-0.4-1.7-0.7s-0.6-0.7-0.6-1.2
c0-0.5,0.2-1,0.7-1.3s1-0.5,1.7-0.5c0.5,0,0.9,0.1,1.3,0.2s0.7,0.3,1,0.5L433.8,226.5z"/>
<path class="st42" d="M440,226.5h-0.9l-0.2-0.8c-0.1-0.1-0.3-0.2-0.5-0.3s-0.5-0.1-0.7-0.1c-0.4,0-0.7,0.1-0.9,0.3
s-0.3,0.4-0.3,0.7c0,0.2,0.1,0.4,0.3,0.6s0.5,0.3,1.1,0.4c0.8,0.2,1.4,0.4,1.8,0.7s0.6,0.7,0.6,1.2c0,0.6-0.2,1-0.7,1.4
s-1,0.5-1.8,0.5c-0.5,0-0.9-0.1-1.3-0.2s-0.7-0.3-1-0.5l0-1.4h0.9l0.2,0.8c0.1,0.1,0.3,0.2,0.5,0.3s0.5,0.1,0.7,0.1
c0.4,0,0.7-0.1,1-0.2s0.3-0.4,0.3-0.7c0-0.3-0.1-0.5-0.3-0.6s-0.6-0.3-1.1-0.4c-0.8-0.2-1.3-0.4-1.7-0.7s-0.6-0.7-0.6-1.2
c0-0.5,0.2-1,0.7-1.3s1-0.5,1.7-0.5c0.5,0,0.9,0.1,1.3,0.2s0.7,0.3,1,0.5L440,226.5z"/>
<path class="st42" d="M444.1,230.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3v-0.3c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,1.9-0.9
c0.9,0,1.5,0.3,1.9,0.8s0.7,1.2,0.7,2.1v0.7h-4.1l0,0c0,0.6,0.2,1.1,0.5,1.5s0.7,0.6,1.2,0.6c0.4,0,0.7-0.1,1-0.2s0.5-0.3,0.8-0.5
l0.5,0.8c-0.2,0.2-0.5,0.4-0.9,0.6S444.6,230.9,444.1,230.9z M443.9,225.3c-0.4,0-0.7,0.2-1,0.5s-0.4,0.7-0.5,1.2l0,0h2.9v-0.2
c0-0.5-0.1-0.8-0.4-1.1S444.4,225.3,443.9,225.3z"/>
<path class="st42" d="M452.5,226.5h-0.9l-0.2-0.8c-0.1-0.1-0.3-0.2-0.5-0.3s-0.5-0.1-0.7-0.1c-0.4,0-0.7,0.1-0.9,0.3
s-0.3,0.4-0.3,0.7c0,0.2,0.1,0.4,0.3,0.6s0.5,0.3,1.1,0.4c0.8,0.2,1.4,0.4,1.8,0.7s0.6,0.7,0.6,1.2c0,0.6-0.2,1-0.7,1.4
s-1,0.5-1.8,0.5c-0.5,0-0.9-0.1-1.3-0.2s-0.7-0.3-1-0.5l0-1.4h0.9l0.2,0.8c0.1,0.1,0.3,0.2,0.5,0.3s0.5,0.1,0.7,0.1
c0.4,0,0.7-0.1,1-0.2s0.3-0.4,0.3-0.7c0-0.3-0.1-0.5-0.3-0.6s-0.6-0.3-1.1-0.4c-0.8-0.2-1.3-0.4-1.7-0.7s-0.6-0.7-0.6-1.2
c0-0.5,0.2-1,0.7-1.3s1-0.5,1.7-0.5c0.5,0,0.9,0.1,1.3,0.2s0.7,0.3,1,0.5L452.5,226.5z"/>
</g>
<g>
<path class="st42" d="M374.4,111.8v0.7l-1,0.2v7.6h-1.2l-4.1-6.6l0,0v5.7l1,0.2v0.7h-3.1v-0.7l1-0.2v-6.7l-1-0.2v-0.7h1h1.2
l4.1,6.6l0,0v-5.7l-1-0.2v-0.7h2.1H374.4z"/>
<path class="st42" d="M375.3,117.1c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,2.1-0.9c0.9,0,1.6,0.3,2.1,0.9s0.8,1.4,0.8,2.3v0.1
c0,0.9-0.3,1.7-0.8,2.3s-1.2,0.9-2.1,0.9c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3V117.1z M376.5,117.2c0,0.7,0.1,1.2,0.4,1.7
s0.7,0.7,1.3,0.7c0.5,0,1-0.2,1.2-0.7s0.4-1,0.4-1.7v-0.1c0-0.7-0.1-1.2-0.4-1.7s-0.7-0.7-1.3-0.7s-1,0.2-1.3,0.7s-0.4,1-0.4,1.7
V117.2z"/>
<path class="st42" d="M386.3,119.5c-0.2,0.3-0.5,0.5-0.8,0.7s-0.6,0.2-1,0.2c-0.8,0-1.4-0.3-1.8-0.8s-0.7-1.3-0.7-2.2v-0.1
c0-1,0.2-1.8,0.7-2.5s1-0.9,1.8-0.9c0.4,0,0.7,0.1,1,0.2s0.5,0.3,0.7,0.6v-2.6l-1-0.2v-0.7h1h1.2v8.2l1,0.2v0.7h-2L386.3,119.5z
M383.2,117.4c0,0.6,0.1,1.1,0.4,1.5s0.7,0.6,1.2,0.6c0.3,0,0.6-0.1,0.9-0.2s0.4-0.4,0.6-0.7v-2.9c-0.1-0.3-0.3-0.5-0.6-0.6
s-0.5-0.2-0.9-0.2c-0.6,0-1,0.2-1.2,0.7s-0.4,1.1-0.4,1.8V117.4z"/>
<path class="st42" d="M392.2,120.4c-0.9,0-1.6-0.3-2.1-0.9s-0.8-1.4-0.8-2.3V117c0-0.9,0.3-1.7,0.8-2.3s1.2-0.9,1.9-0.9
c0.9,0,1.5,0.3,1.9,0.8s0.7,1.2,0.7,2.1v0.7h-4.1l0,0c0,0.6,0.2,1.1,0.5,1.5s0.7,0.6,1.2,0.6c0.4,0,0.7-0.1,1-0.2s0.5-0.3,0.8-0.5
l0.5,0.8c-0.2,0.2-0.5,0.4-0.9,0.6S392.7,120.4,392.2,120.4z M392,114.8c-0.4,0-0.7,0.2-1,0.5s-0.4,0.7-0.5,1.2l0,0h2.9v-0.2
c0-0.5-0.1-0.8-0.4-1.1S392.5,114.8,392,114.8z"/>
</g>
<g>
<g>
<path class="st42" d="M122.4,356.6l1-0.2v-6.9l-1-0.2v-1.2h4v1.2l-1,0.2v2.6h0.8l1.9-2.7l-0.6-0.1v-1.2h3.8v1.2l-1,0.2l-2.4,3.2
l2.7,3.8l1,0.2v1.2h-3.8v-1.2l0.6-0.1l-1.9-2.8h-1.1v2.7l1,0.2v1.2h-4V356.6z"/>
<path class="st42" d="M137,356.9c-0.2,0.3-0.5,0.6-0.9,0.8c-0.3,0.2-0.7,0.3-1.2,0.3c-0.8,0-1.4-0.2-1.8-0.7
c-0.4-0.5-0.6-1.2-0.6-2.3v-3l-0.8-0.2v-1.2h0.8h1.9v4.3c0,0.5,0.1,0.9,0.3,1.1c0.2,0.2,0.4,0.3,0.8,0.3c0.3,0,0.6,0,0.8-0.1
s0.4-0.2,0.5-0.4V352l-0.8-0.2v-1.2h0.8h1.9v5.8l0.9,0.2v1.2h-2.6L137,356.9z"/>
<path class="st42" d="M147.4,354.4c0,1.1-0.2,1.9-0.7,2.6c-0.5,0.6-1.2,1-2.1,1c-0.4,0-0.8-0.1-1.1-0.3c-0.3-0.2-0.6-0.4-0.8-0.8
l-0.1,0.9H141v-9l-1-0.2v-1.2h3v3.9c0.2-0.3,0.5-0.5,0.7-0.7c0.3-0.2,0.6-0.2,1-0.2c0.9,0,1.6,0.3,2.1,1c0.5,0.7,0.7,1.6,0.7,2.8
V354.4z M145.5,354.3c0-0.7-0.1-1.3-0.3-1.7c-0.2-0.4-0.6-0.6-1.1-0.6c-0.3,0-0.6,0.1-0.8,0.2c-0.2,0.1-0.4,0.3-0.5,0.5v3
c0.1,0.2,0.3,0.4,0.5,0.5c0.2,0.1,0.5,0.2,0.8,0.2c0.5,0,0.8-0.2,1-0.5s0.3-0.9,0.3-1.5V354.3z"/>
<path class="st42" d="M151.8,358c-1,0-1.9-0.3-2.5-1c-0.6-0.7-0.9-1.5-0.9-2.5v-0.3c0-1.1,0.3-1.9,0.9-2.6c0.6-0.7,1.4-1,2.4-1
c1,0,1.7,0.3,2.3,0.9c0.5,0.6,0.8,1.4,0.8,2.4v1.1h-4.3l0,0c0,0.5,0.2,0.9,0.5,1.2c0.3,0.3,0.7,0.5,1.1,0.5c0.4,0,0.8,0,1.1-0.1
c0.3-0.1,0.6-0.2,0.9-0.4l0.5,1.2c-0.3,0.2-0.7,0.4-1.2,0.6C152.9,357.9,152.4,358,151.8,358z M151.6,352c-0.4,0-0.6,0.1-0.8,0.4
c-0.2,0.3-0.3,0.6-0.4,1.1l0,0h2.4v-0.2c0-0.4-0.1-0.7-0.3-1S152,352,151.6,352z"/>
<path class="st42" d="M155.7,356.6l0.9-0.2V352l-1-0.2v-1.2h2.8l0.1,1c0.2-0.4,0.4-0.7,0.7-0.9c0.3-0.2,0.6-0.3,0.9-0.3
c0.1,0,0.2,0,0.3,0c0.1,0,0.2,0,0.3,0.1l-0.2,1.8l-0.8,0c-0.3,0-0.5,0.1-0.7,0.2c-0.2,0.1-0.3,0.3-0.4,0.5v3.5l0.9,0.2v1.2h-3.8
V356.6z"/>
<path class="st42" d="M161.2,356.6l0.9-0.2V352l-1-0.2v-1.2h2.8l0.1,1c0.2-0.4,0.5-0.7,0.9-0.9s0.7-0.3,1.2-0.3
c0.7,0,1.3,0.2,1.7,0.7s0.6,1.2,0.6,2.1v3.1l0.9,0.2v1.2h-3.7v-1.2l0.8-0.2v-3.1c0-0.5-0.1-0.8-0.3-1c-0.2-0.2-0.5-0.3-0.9-0.3
c-0.3,0-0.5,0.1-0.7,0.2c-0.2,0.1-0.4,0.3-0.5,0.4v3.9l0.8,0.2v1.2h-3.7V356.6z"/>
<path class="st42" d="M173.3,358c-1,0-1.9-0.3-2.5-1c-0.6-0.7-0.9-1.5-0.9-2.5v-0.3c0-1.1,0.3-1.9,0.9-2.6c0.6-0.7,1.4-1,2.4-1
c1,0,1.7,0.3,2.3,0.9c0.5,0.6,0.8,1.4,0.8,2.4v1.1h-4.3l0,0c0,0.5,0.2,0.9,0.5,1.2c0.3,0.3,0.7,0.5,1.1,0.5c0.4,0,0.8,0,1.1-0.1
c0.3-0.1,0.6-0.2,0.9-0.4l0.5,1.2c-0.3,0.2-0.7,0.4-1.2,0.6C174.4,357.9,173.9,358,173.3,358z M173.1,352c-0.4,0-0.6,0.1-0.8,0.4
c-0.2,0.3-0.3,0.6-0.4,1.1l0,0h2.4v-0.2c0-0.4-0.1-0.7-0.3-1S173.5,352,173.1,352z"/>
<path class="st42" d="M180,348.9v1.8h1.3v1.4H180v3.7c0,0.3,0.1,0.5,0.2,0.6c0.1,0.1,0.3,0.2,0.5,0.2c0.1,0,0.2,0,0.3,0
c0.1,0,0.2,0,0.3-0.1l0.2,1.4c-0.2,0.1-0.4,0.1-0.6,0.1c-0.2,0-0.4,0-0.7,0c-0.7,0-1.2-0.2-1.5-0.6c-0.4-0.4-0.5-0.9-0.5-1.7V352
H177v-1.4h1.1v-1.8H180z"/>
<path class="st42" d="M185.6,358c-1,0-1.9-0.3-2.5-1c-0.6-0.7-0.9-1.5-0.9-2.5v-0.3c0-1.1,0.3-1.9,0.9-2.6c0.6-0.7,1.4-1,2.4-1
c1,0,1.7,0.3,2.3,0.9c0.5,0.6,0.8,1.4,0.8,2.4v1.1h-4.3l0,0c0,0.5,0.2,0.9,0.5,1.2c0.3,0.3,0.7,0.5,1.1,0.5c0.4,0,0.8,0,1.1-0.1
c0.3-0.1,0.6-0.2,0.9-0.4l0.5,1.2c-0.3,0.2-0.7,0.4-1.2,0.6C186.7,357.9,186.2,358,185.6,358z M185.5,352c-0.4,0-0.6,0.1-0.8,0.4
c-0.2,0.3-0.3,0.6-0.4,1.1l0,0h2.4v-0.2c0-0.4-0.1-0.7-0.3-1S185.9,352,185.5,352z"/>
<path class="st42" d="M189.6,356.6l0.9-0.2V352l-1-0.2v-1.2h2.8l0.1,1c0.2-0.4,0.4-0.7,0.7-0.9c0.3-0.2,0.6-0.3,0.9-0.3
c0.1,0,0.2,0,0.3,0c0.1,0,0.2,0,0.3,0.1l-0.2,1.8l-0.8,0c-0.3,0-0.5,0.1-0.7,0.2c-0.2,0.1-0.3,0.3-0.4,0.5v3.5l0.9,0.2v1.2h-3.8
V356.6z"/>
<path class="st42" d="M201,353.1h-1.3l-0.2-0.9c-0.1-0.1-0.3-0.2-0.5-0.3c-0.2-0.1-0.4-0.1-0.7-0.1c-0.3,0-0.6,0.1-0.8,0.2
s-0.3,0.3-0.3,0.6c0,0.2,0.1,0.4,0.3,0.5c0.2,0.1,0.6,0.3,1.1,0.4c0.9,0.2,1.5,0.4,2,0.8c0.4,0.3,0.6,0.8,0.6,1.4
c0,0.6-0.3,1.2-0.8,1.6c-0.6,0.4-1.3,0.6-2.2,0.6c-0.6,0-1.1-0.1-1.5-0.2c-0.5-0.2-0.9-0.4-1.2-0.7l0-1.6h1.4l0.3,0.9
c0.1,0.1,0.3,0.2,0.5,0.2s0.4,0.1,0.6,0.1c0.4,0,0.7-0.1,0.9-0.2c0.2-0.1,0.3-0.3,0.3-0.6c0-0.2-0.1-0.4-0.3-0.6
c-0.2-0.2-0.6-0.3-1.1-0.4c-0.8-0.2-1.5-0.4-1.9-0.8c-0.4-0.3-0.6-0.8-0.6-1.4c0-0.6,0.2-1.1,0.7-1.6s1.2-0.7,2.1-0.7
c0.6,0,1.1,0.1,1.6,0.2c0.5,0.2,0.9,0.3,1.2,0.6L201,353.1z"/>
<path class="st42" d="M208.9,356.5c0.3,0,0.6-0.1,0.8-0.3c0.2-0.2,0.3-0.5,0.3-0.8h1.8l0,0c0,0.7-0.3,1.3-0.8,1.8
c-0.6,0.5-1.3,0.7-2.1,0.7c-1.1,0-1.9-0.3-2.5-1s-0.9-1.5-0.9-2.6v-0.2c0-1.1,0.3-1.9,0.9-2.6s1.4-1,2.5-1c0.6,0,1.1,0.1,1.6,0.3
c0.5,0.2,0.8,0.4,1.1,0.7l0,1.9h-1.6l-0.3-1.1c-0.1-0.1-0.2-0.2-0.4-0.2c-0.1-0.1-0.3-0.1-0.5-0.1c-0.5,0-0.9,0.2-1.2,0.6
s-0.3,0.9-0.3,1.5v0.2c0,0.6,0.1,1.2,0.3,1.6C208,356.3,208.4,356.5,208.9,356.5z"/>
<path class="st42" d="M212.5,348.6v-1.2h3v9l0.9,0.2v1.2h-3.8v-1.2l0.9-0.2v-7.6L212.5,348.6z"/>
<path class="st42" d="M222,356.9c-0.2,0.3-0.5,0.6-0.9,0.8c-0.3,0.2-0.7,0.3-1.2,0.3c-0.8,0-1.4-0.2-1.8-0.7
c-0.4-0.5-0.6-1.2-0.6-2.3v-3l-0.8-0.2v-1.2h0.8h1.9v4.3c0,0.5,0.1,0.9,0.3,1.1c0.2,0.2,0.4,0.3,0.8,0.3c0.3,0,0.6,0,0.8-0.1
s0.4-0.2,0.5-0.4V352l-0.8-0.2v-1.2h0.8h1.9v5.8l0.9,0.2v1.2h-2.6L222,356.9z"/>
<path class="st42" d="M231.4,353.1h-1.3l-0.2-0.9c-0.1-0.1-0.3-0.2-0.5-0.3c-0.2-0.1-0.4-0.1-0.7-0.1c-0.3,0-0.6,0.1-0.8,0.2
s-0.3,0.3-0.3,0.6c0,0.2,0.1,0.4,0.3,0.5c0.2,0.1,0.6,0.3,1.1,0.4c0.9,0.2,1.5,0.4,2,0.8c0.4,0.3,0.6,0.8,0.6,1.4
c0,0.6-0.3,1.2-0.8,1.6c-0.6,0.4-1.3,0.6-2.2,0.6c-0.6,0-1.1-0.1-1.5-0.2c-0.5-0.2-0.9-0.4-1.2-0.7l0-1.6h1.4l0.3,0.9
c0.1,0.1,0.3,0.2,0.5,0.2s0.4,0.1,0.6,0.1c0.4,0,0.7-0.1,0.9-0.2c0.2-0.1,0.3-0.3,0.3-0.6c0-0.2-0.1-0.4-0.3-0.6
c-0.2-0.2-0.6-0.3-1.1-0.4c-0.8-0.2-1.5-0.4-1.9-0.8c-0.4-0.3-0.6-0.8-0.6-1.4c0-0.6,0.2-1.1,0.7-1.6s1.2-0.7,2.1-0.7
c0.6,0,1.1,0.1,1.6,0.2c0.5,0.2,0.9,0.3,1.2,0.6L231.4,353.1z"/>
<path class="st42" d="M235.4,348.9v1.8h1.3v1.4h-1.3v3.7c0,0.3,0.1,0.5,0.2,0.6c0.1,0.1,0.3,0.2,0.5,0.2c0.1,0,0.2,0,0.3,0
c0.1,0,0.2,0,0.3-0.1l0.2,1.4c-0.2,0.1-0.4,0.1-0.6,0.1c-0.2,0-0.4,0-0.7,0c-0.7,0-1.2-0.2-1.5-0.6c-0.4-0.4-0.5-0.9-0.5-1.7V352
h-1.1v-1.4h1.1v-1.8H235.4z"/>
<path class="st42" d="M241,358c-1,0-1.9-0.3-2.5-1c-0.6-0.7-0.9-1.5-0.9-2.5v-0.3c0-1.1,0.3-1.9,0.9-2.6c0.6-0.7,1.4-1,2.4-1
c1,0,1.7,0.3,2.3,0.9c0.5,0.6,0.8,1.4,0.8,2.4v1.1h-4.3l0,0c0,0.5,0.2,0.9,0.5,1.2c0.3,0.3,0.7,0.5,1.1,0.5c0.4,0,0.8,0,1.1-0.1
c0.3-0.1,0.6-0.2,0.9-0.4l0.5,1.2c-0.3,0.2-0.7,0.4-1.2,0.6C242.1,357.9,241.6,358,241,358z M240.8,352c-0.4,0-0.6,0.1-0.8,0.4
c-0.2,0.3-0.3,0.6-0.4,1.1l0,0h2.4v-0.2c0-0.4-0.1-0.7-0.3-1S241.2,352,240.8,352z"/>
<path class="st42" d="M245,356.6l0.9-0.2V352l-1-0.2v-1.2h2.8l0.1,1c0.2-0.4,0.4-0.7,0.7-0.9c0.3-0.2,0.6-0.3,0.9-0.3
c0.1,0,0.2,0,0.3,0c0.1,0,0.2,0,0.3,0.1l-0.2,1.8l-0.8,0c-0.3,0-0.5,0.1-0.7,0.2c-0.2,0.1-0.3,0.3-0.4,0.5v3.5l0.9,0.2v1.2H245
V356.6z"/>
</g>
</g>
<line class="st62" x1="360.6" y1="191.7" x2="201.6" y2="191.7"/>
<line class="st62" x1="360.6" y1="228.8" x2="160" y2="228.8"/>
<line class="st62" x1="360.6" y1="118.3" x2="243.4" y2="118.3"/>
</g>
<g id="Layer_14">
</g>
</svg>
.st63{fill:none;stroke:#011F38;stroke-width:0.2813;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}</style>
<symbol viewBox="-68.6 -66.9 137.2 133.9" id="master_x5F_level1_1_">
<g id="svg_1">
<g id="svg_2">
<line id="svg_3" y2="0.7" x2="0" y1="-11.1" x1="0" class="st0"/>
<line id="svg_4" y2="-5.2" x2="-5.9" y1="-5.2" x1="5.9" class="st0"/>
</g>
<polygon id="svg_5" points="-29.2,-63.9 -65.6,-18.3 -52.6,38.6 0,63.9 52.6,38.6 65.6,-18.3 29.2,-63.9 " class="st1"/>
</g>
</symbol>
<symbol viewBox="-81 -93 162 186.1" id="node_high_level">
<polygon id="svg_6" points="-80,-46 -80,46 0,92 80,46 80,-46 0,-92 " class="st2"/>
<g id="Isolation_Mode_3_"/>
</symbol>
<symbol viewBox="-87.5 -100.6 175.1 201.1" id="node_x5F_empty">
<use transform="matrix(1.0808,0,0,1.0808,-0.00003292006,-0.00003749943) " y="-93" x="-81" id="XMLID_201_" height="186.1" width="162" xlink:href="#node_high_level"/>
<g id="svg_7">
<polygon id="svg_8" points="76.8,-28.1 -14,-80.3 0,-88.3 76.7,-44.4 " class="st3"/>
<polygon id="svg_9" points="76.8,-28.1 32.1,-53.8 38.8,-66.1 76.7,-44.4 " class="st4"/>
</g>
</symbol>
<symbol viewBox="-87.6 -101 175.2 202" id="node_x5F_new">
<polygon id="svg_10" points="0,-100 -86.6,-50 -86.6,50 0,100 86.6,50 86.6,-50 " class="st5"/>
<polygon id="svg_11" points="-86.6,-20.2 -86.6,-50 0,-100 25.8,-85.1 " class="st6"/>
<polygon id="svg_12" points="-40.8,-70.7 -32.9,-57 15.7,-85.1 0,-94.3 " class="st7"/>
<text id="svg_13" font-family="&#x27;RobotoSlab-Regular'" font-size="11.3632px" class="st8" transform="matrix(0.866,-0.5,-0.5,-0.866,-33.9256,-70.7388) ">Docker</text>
<text id="svg_14" font-family="&#x27;RobotoSlab-Regular'" font-size="11.3632px" class="st8" transform="matrix(0.866,-0.5,-0.5,-0.866,-76.0668,-46.4087) ">Kubelt</text>
</symbol>
<g>
<title>Layer 1</title>
<g id="CLUSTER">
<g class="st9" id="XMLID_296_">
<g id="svg_15">
<linearGradient y2="185.2931" x2="343.0902" y1="185.2931" x1="28.6348" gradientUnits="userSpaceOnUse" id="SVGID_1_">
<stop stop-color="#326DE6" offset="0"/>
<stop stop-color="#10FFC6" offset="1"/>
</linearGradient>
<polygon id="svg_16" points="311.9,92.7 343.1,229.2 255.8,338.6 115.9,338.6 28.6,229.2 59.8,92.7 185.9,32 " class="st10"/>
</g>
</g>
</g>
<g id="master">
<use id="svg_17" transform="matrix(0.4,0,0,-0.4,185.8606,187.2514) " y="-66.9" x="-68.6" height="133.9" width="137.2" xlink:href="#master_x5F_level1_1_"/>
<g id="master_x5F_level1"/>
</g>
<g id="description">
<g id="svg_72">
<path id="svg_73" d="m374.4,188.6l0,0l-2.8,6.8l-0.8,0l-2.8,-6.8l0,0l0.1,3.5l0,2.5l1,0.2l0,0.7l-3.1,0l0,-0.7l1,-0.2l0,-6.7l-1,-0.2l0,-0.7l1,0l1.5,0l2.7,6.9l0,0l2.7,-6.9l2.4,0l0,0.7l-1,0.2l0,6.7l1,0.2l0,0.7l-3.1,0l0,-0.7l1,-0.2l0,-2.6l0.2,-3.4z" class="st42"/>
<path id="svg_74" d="m381.5,195.4c0,-0.2 -0.1,-0.3 -0.1,-0.5s0,-0.3 0,-0.4c-0.2,0.3 -0.5,0.5 -0.8,0.7s-0.7,0.3 -1.1,0.3c-0.7,0 -1.2,-0.2 -1.5,-0.5s-0.5,-0.8 -0.5,-1.4c0,-0.6 0.2,-1.1 0.7,-1.4s1.2,-0.5 2,-0.5l1.2,0l0,-0.7c0,-0.4 -0.1,-0.7 -0.4,-0.9s-0.6,-0.3 -1,-0.3c-0.3,0 -0.5,0 -0.8,0.1s-0.4,0.2 -0.5,0.3l-0.1,0.7l-0.9,0l0,-1.2c0.3,-0.2 0.6,-0.4 1,-0.6s0.9,-0.2 1.3,-0.2c0.7,0 1.3,0.2 1.7,0.6s0.7,0.9 0.7,1.6l0,3.1c0,0.1 0,0.2 0,0.2s0,0.2 0,0.2l0.5,0.1l0,0.7l-1.4,0zm-1.9,-0.8c0.4,0 0.7,-0.1 1,-0.3s0.5,-0.4 0.7,-0.7l0,-1l-1.2,0c-0.5,0 -0.8,0.1 -1.1,0.3s-0.4,0.5 -0.4,0.8c0,0.3 0.1,0.5 0.3,0.6s0.4,0.3 0.7,0.3z" class="st42"/>
<path id="svg_75" d="m388.8,191.1l-0.9,0l-0.2,-0.8c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3s-0.5,-0.1 -0.7,-0.1c-0.4,0 -0.7,0.1 -0.9,0.3s-0.3,0.4 -0.3,0.7c0,0.2 0.1,0.4 0.3,0.6s0.5,0.3 1.1,0.4c0.8,0.2 1.4,0.4 1.8,0.7s0.6,0.7 0.6,1.2c0,0.6 -0.2,1 -0.7,1.4s-1,0.5 -1.8,0.5c-0.5,0 -0.9,-0.1 -1.3,-0.2s-0.7,-0.3 -1,-0.5l0,-1.4l0.9,0l0.2,0.8c0.1,0.1 0.3,0.2 0.5,0.3s0.5,0.1 0.7,0.1c0.4,0 0.7,-0.1 1,-0.2s0.3,-0.4 0.3,-0.7c0,-0.3 -0.1,-0.5 -0.3,-0.6s-0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.3,-0.4 -1.7,-0.7s-0.6,-0.7 -0.6,-1.2c0,-0.5 0.2,-1 0.7,-1.3s1,-0.5 1.7,-0.5c0.5,0 0.9,0.1 1.3,0.2s0.7,0.3 1,0.5l-0.1,1.2z" class="st42"/>
<path id="svg_76" d="m392.1,187.5l0,1.5l1.2,0l0,0.9l-1.2,0l0,3.8c0,0.3 0.1,0.5 0.2,0.6s0.3,0.2 0.5,0.2c0.1,0 0.2,0 0.3,0s0.2,0 0.3,-0.1l0.2,0.8c-0.1,0.1 -0.3,0.1 -0.5,0.2s-0.4,0.1 -0.6,0.1c-0.5,0 -0.8,-0.1 -1.1,-0.4s-0.4,-0.7 -0.4,-1.3l0,-3.8l-1,0l0,-0.9l1,0l0,-1.5l1.1,0l0,-0.1z" class="st42"/>
<path id="svg_77" d="m397.4,195.5c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.3c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 1.9,-0.9c0.9,0 1.5,0.3 1.9,0.8s0.7,1.2 0.7,2.1l0,0.7l-4.1,0l0,0c0,0.6 0.2,1.1 0.5,1.5s0.7,0.6 1.2,0.6c0.4,0 0.7,-0.1 1,-0.2s0.5,-0.3 0.8,-0.5l0.5,0.8c-0.2,0.2 -0.5,0.4 -0.9,0.6s-0.9,0.3 -1.4,0.3zm-0.2,-5.6c-0.4,0 -0.7,0.2 -1,0.5s-0.4,0.7 -0.5,1.2l0,0l2.9,0l0,-0.2c0,-0.5 -0.1,-0.8 -0.4,-1.1s-0.5,-0.4 -1,-0.4z" class="st42"/>
<path id="svg_78" d="m400.9,189.8l0,-0.7l2,0l0.1,0.9c0.2,-0.3 0.4,-0.6 0.7,-0.8s0.6,-0.3 0.9,-0.3c0.1,0 0.2,0 0.3,0s0.2,0 0.2,0l-0.2,1.1l-0.7,0c-0.3,0 -0.6,0.1 -0.8,0.2s-0.4,0.3 -0.5,0.6l0,3.6l1,0.2l0,0.7l-3.1,0l0,-0.7l1,-0.2l0,-4.4l-0.9,-0.2z" class="st42"/>
</g>
<g id="svg_79">
<path id="svg_80" d="m366,230.1l1,-0.2l0,-4.5l-1,-0.2l0,-0.7l2,0l0.1,0.9c0.2,-0.3 0.5,-0.6 0.8,-0.8s0.7,-0.3 1.1,-0.3c0.7,0 1.2,0.2 1.6,0.6s0.6,1 0.6,1.9l0,3.1l1,0.2l0,0.7l-3.2,0l0,-0.7l1,-0.2l0,-3.1c0,-0.6 -0.1,-1 -0.3,-1.2s-0.6,-0.4 -1,-0.4c-0.3,0 -0.6,0.1 -0.9,0.2s-0.5,0.4 -0.6,0.7l0,3.7l1,0.2l0,0.7l-3.2,0l0,-0.6z" class="st42"/>
<path id="svg_81" d="m373.9,227.6c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 2.1,-0.9c0.9,0 1.6,0.3 2.1,0.9s0.8,1.4 0.8,2.3l0,0.1c0,0.9 -0.3,1.7 -0.8,2.3s-1.2,0.9 -2.1,0.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.1zm1.1,0.1c0,0.7 0.1,1.2 0.4,1.7s0.7,0.7 1.3,0.7c0.5,0 1,-0.2 1.2,-0.7s0.4,-1 0.4,-1.7l0,-0.1c0,-0.7 -0.1,-1.2 -0.4,-1.7s-0.7,-0.7 -1.3,-0.7s-1,0.2 -1.3,0.7s-0.4,1 -0.4,1.7l0,0.1l0.1,0z" class="st42"/>
<path id="svg_82" d="m384.9,230c-0.2,0.3 -0.5,0.5 -0.8,0.7s-0.6,0.2 -1,0.2c-0.8,0 -1.4,-0.3 -1.8,-0.8s-0.7,-1.3 -0.7,-2.2l0,-0.1c0,-1 0.2,-1.8 0.7,-2.5s1,-0.9 1.8,-0.9c0.4,0 0.7,0.1 1,0.2s0.5,0.3 0.7,0.6l0,-2.6l-1,-0.2l0,-0.7l1,0l1.2,0l0,8.2l1,0.2l0,0.7l-2,0l-0.1,-0.8zm-3.2,-2.1c0,0.6 0.1,1.1 0.4,1.5s0.7,0.6 1.2,0.6c0.3,0 0.6,-0.1 0.9,-0.2s0.4,-0.4 0.6,-0.7l0,-2.9c-0.1,-0.3 -0.3,-0.5 -0.6,-0.6s-0.5,-0.2 -0.9,-0.2c-0.6,0 -1,0.2 -1.2,0.7s-0.4,1.1 -0.4,1.8l0,0z" class="st42"/>
<path id="svg_83" d="m390.7,230.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.3c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 1.9,-0.9c0.9,0 1.5,0.3 1.9,0.8s0.7,1.2 0.7,2.1l0,0.7l-4.1,0l0,0c0,0.6 0.2,1.1 0.5,1.5s0.7,0.6 1.2,0.6c0.4,0 0.7,-0.1 1,-0.2s0.5,-0.3 0.8,-0.5l0.5,0.8c-0.2,0.2 -0.5,0.4 -0.9,0.6s-0.9,0.3 -1.4,0.3zm-0.2,-5.6c-0.4,0 -0.7,0.2 -1,0.5s-0.4,0.7 -0.5,1.2l0,0l2.9,0l0,-0.2c0,-0.5 -0.1,-0.8 -0.4,-1.1s-0.5,-0.4 -1,-0.4z" class="st42"/>
<path id="svg_84" d="m397.1,232.5l1,-0.2l0,-7l-1,-0.2l0,-0.7l1.9,0l0.1,0.8c0.2,-0.3 0.5,-0.5 0.8,-0.7s0.7,-0.2 1.1,-0.2c0.8,0 1.4,0.3 1.8,0.9s0.7,1.4 0.7,2.5l0,0.1c0,0.9 -0.2,1.7 -0.7,2.2s-1,0.8 -1.8,0.8c-0.4,0 -0.7,-0.1 -1,-0.2s-0.5,-0.3 -0.8,-0.6l0,2.2l1,0.2l0,0.7l-3.1,0l0,-0.6zm5.2,-4.8c0,-0.7 -0.1,-1.3 -0.4,-1.8s-0.7,-0.7 -1.3,-0.7c-0.3,0 -0.6,0.1 -0.8,0.2s-0.4,0.4 -0.6,0.6l0,3.1c0.1,0.3 0.3,0.5 0.6,0.6s0.5,0.2 0.9,0.2c0.5,0 1,-0.2 1.2,-0.6s0.4,-0.9 0.4,-1.6l0,0z" class="st42"/>
<path id="svg_85" d="m404.5,225.2l0,-0.7l2,0l0.1,0.9c0.2,-0.3 0.4,-0.6 0.7,-0.8s0.6,-0.3 0.9,-0.3c0.1,0 0.2,0 0.3,0s0.2,0 0.2,0l-0.2,1.1l-0.7,0c-0.3,0 -0.6,0.1 -0.8,0.2s-0.4,0.3 -0.5,0.6l0,3.6l1,0.2l0,0.7l-3.1,0l0,-0.7l1,-0.2l0,-4.5l-0.9,-0.1z" class="st42"/>
<path id="svg_86" d="m409.3,227.6c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 2.1,-0.9c0.9,0 1.6,0.3 2.1,0.9s0.8,1.4 0.8,2.3l0,0.1c0,0.9 -0.3,1.7 -0.8,2.3s-1.2,0.9 -2.1,0.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.1zm1.2,0.1c0,0.7 0.1,1.2 0.4,1.7s0.7,0.7 1.3,0.7c0.5,0 1,-0.2 1.2,-0.7s0.4,-1 0.4,-1.7l0,-0.1c0,-0.7 -0.1,-1.2 -0.4,-1.7s-0.7,-0.7 -1.3,-0.7s-1,0.2 -1.3,0.7s-0.4,1 -0.4,1.7l0,0.1l0.1,0z" class="st42"/>
<path id="svg_87" d="m418.9,230c0.4,0 0.7,-0.1 1,-0.4s0.4,-0.5 0.4,-0.9l1,0l0,0c0,0.5 -0.2,1 -0.7,1.5s-1.1,0.6 -1.8,0.6c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.7,-1.4 -0.7,-2.3l0,-0.2c0,-0.9 0.2,-1.7 0.7,-2.3s1.2,-0.9 2.1,-0.9c0.5,0 1,0.1 1.4,0.3s0.7,0.4 1,0.7l0.1,1.4l-0.9,0l-0.3,-1c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3s-0.5,-0.1 -0.7,-0.1c-0.6,0 -1,0.2 -1.3,0.7s-0.4,1 -0.4,1.6l0,0.2c0,0.6 0.1,1.2 0.4,1.6s0.7,0.7 1.3,0.7z" class="st42"/>
<path id="svg_88" d="m425.4,230.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.3c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 1.9,-0.9c0.9,0 1.5,0.3 1.9,0.8s0.7,1.2 0.7,2.1l0,0.7l-4.1,0l0,0c0,0.6 0.2,1.1 0.5,1.5s0.7,0.6 1.2,0.6c0.4,0 0.7,-0.1 1,-0.2s0.5,-0.3 0.8,-0.5l0.5,0.8c-0.2,0.2 -0.5,0.4 -0.9,0.6s-0.8,0.3 -1.4,0.3zm-0.1,-5.6c-0.4,0 -0.7,0.2 -1,0.5s-0.4,0.7 -0.5,1.2l0,0l2.9,0l0,-0.2c0,-0.5 -0.1,-0.8 -0.4,-1.1s-0.6,-0.4 -1,-0.4z" class="st42"/>
<path id="svg_89" d="m433.8,226.5l-0.8,0l-0.2,-0.8c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3s-0.5,-0.1 -0.7,-0.1c-0.4,0 -0.7,0.1 -0.9,0.3s-0.3,0.4 -0.3,0.7c0,0.2 0.1,0.4 0.3,0.6s0.5,0.3 1.1,0.4c0.8,0.2 1.4,0.4 1.8,0.7s0.6,0.7 0.6,1.2c0,0.6 -0.2,1 -0.7,1.4s-1,0.5 -1.8,0.5c-0.5,0 -0.9,-0.1 -1.3,-0.2s-0.7,-0.3 -1,-0.5l0,-1.4l0.9,0l0.2,0.8c0.1,0.1 0.3,0.2 0.5,0.3s0.5,0.1 0.7,0.1c0.4,0 0.7,-0.1 1,-0.2s0.3,-0.4 0.3,-0.7c0,-0.3 -0.1,-0.5 -0.3,-0.6s-0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.3,-0.4 -1.7,-0.7s-0.6,-0.7 -0.6,-1.2c0,-0.5 0.2,-1 0.7,-1.3s1,-0.5 1.7,-0.5c0.5,0 0.9,0.1 1.3,0.2s0.7,0.3 1,0.5l-0.2,1.2z" class="st42"/>
<path id="svg_90" d="m440,226.5l-0.9,0l-0.2,-0.8c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3s-0.5,-0.1 -0.7,-0.1c-0.4,0 -0.7,0.1 -0.9,0.3s-0.3,0.4 -0.3,0.7c0,0.2 0.1,0.4 0.3,0.6s0.5,0.3 1.1,0.4c0.8,0.2 1.4,0.4 1.8,0.7s0.6,0.7 0.6,1.2c0,0.6 -0.2,1 -0.7,1.4s-1,0.5 -1.8,0.5c-0.5,0 -0.9,-0.1 -1.3,-0.2s-0.7,-0.3 -1,-0.5l0,-1.4l0.9,0l0.2,0.8c0.1,0.1 0.3,0.2 0.5,0.3s0.5,0.1 0.7,0.1c0.4,0 0.7,-0.1 1,-0.2s0.3,-0.4 0.3,-0.7c0,-0.3 -0.1,-0.5 -0.3,-0.6s-0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.3,-0.4 -1.7,-0.7s-0.6,-0.7 -0.6,-1.2c0,-0.5 0.2,-1 0.7,-1.3s1,-0.5 1.7,-0.5c0.5,0 0.9,0.1 1.3,0.2s0.7,0.3 1,0.5l-0.1,1.2z" class="st42"/>
<path id="svg_91" d="m444.1,230.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.3c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 1.9,-0.9c0.9,0 1.5,0.3 1.9,0.8s0.7,1.2 0.7,2.1l0,0.7l-4.1,0l0,0c0,0.6 0.2,1.1 0.5,1.5s0.7,0.6 1.2,0.6c0.4,0 0.7,-0.1 1,-0.2s0.5,-0.3 0.8,-0.5l0.5,0.8c-0.2,0.2 -0.5,0.4 -0.9,0.6s-0.9,0.3 -1.4,0.3zm-0.2,-5.6c-0.4,0 -0.7,0.2 -1,0.5s-0.4,0.7 -0.5,1.2l0,0l2.9,0l0,-0.2c0,-0.5 -0.1,-0.8 -0.4,-1.1s-0.5,-0.4 -1,-0.4z" class="st42"/>
<path id="svg_92" d="m452.5,226.5l-0.9,0l-0.2,-0.8c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3s-0.5,-0.1 -0.7,-0.1c-0.4,0 -0.7,0.1 -0.9,0.3s-0.3,0.4 -0.3,0.7c0,0.2 0.1,0.4 0.3,0.6s0.5,0.3 1.1,0.4c0.8,0.2 1.4,0.4 1.8,0.7s0.6,0.7 0.6,1.2c0,0.6 -0.2,1 -0.7,1.4s-1,0.5 -1.8,0.5c-0.5,0 -0.9,-0.1 -1.3,-0.2s-0.7,-0.3 -1,-0.5l0,-1.4l0.9,0l0.2,0.8c0.1,0.1 0.3,0.2 0.5,0.3s0.5,0.1 0.7,0.1c0.4,0 0.7,-0.1 1,-0.2s0.3,-0.4 0.3,-0.7c0,-0.3 -0.1,-0.5 -0.3,-0.6s-0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.3,-0.4 -1.7,-0.7s-0.6,-0.7 -0.6,-1.2c0,-0.5 0.2,-1 0.7,-1.3s1,-0.5 1.7,-0.5c0.5,0 0.9,0.1 1.3,0.2s0.7,0.3 1,0.5l-0.1,1.2z" class="st42"/>
</g>
<g id="svg_93">
<path id="svg_94" d="m374.4,111.8l0,0.7l-1,0.2l0,7.6l-1.2,0l-4.1,-6.6l0,0l0,5.7l1,0.2l0,0.7l-3.1,0l0,-0.7l1,-0.2l0,-6.7l-1,-0.2l0,-0.7l1,0l1.2,0l4.1,6.6l0,0l0,-5.7l-1,-0.2l0,-0.7l2.1,0l1,0z" class="st42"/>
<path id="svg_95" d="m375.3,117.1c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 2.1,-0.9c0.9,0 1.6,0.3 2.1,0.9s0.8,1.4 0.8,2.3l0,0.1c0,0.9 -0.3,1.7 -0.8,2.3s-1.2,0.9 -2.1,0.9c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.1zm1.2,0.1c0,0.7 0.1,1.2 0.4,1.7s0.7,0.7 1.3,0.7c0.5,0 1,-0.2 1.2,-0.7s0.4,-1 0.4,-1.7l0,-0.1c0,-0.7 -0.1,-1.2 -0.4,-1.7s-0.7,-0.7 -1.3,-0.7s-1,0.2 -1.3,0.7s-0.4,1 -0.4,1.7l0,0.1l0.1,0z" class="st42"/>
<path id="svg_96" d="m386.3,119.5c-0.2,0.3 -0.5,0.5 -0.8,0.7s-0.6,0.2 -1,0.2c-0.8,0 -1.4,-0.3 -1.8,-0.8s-0.7,-1.3 -0.7,-2.2l0,-0.1c0,-1 0.2,-1.8 0.7,-2.5s1,-0.9 1.8,-0.9c0.4,0 0.7,0.1 1,0.2s0.5,0.3 0.7,0.6l0,-2.6l-1,-0.2l0,-0.7l1,0l1.2,0l0,8.2l1,0.2l0,0.7l-2,0l-0.1,-0.8zm-3.1,-2.1c0,0.6 0.1,1.1 0.4,1.5s0.7,0.6 1.2,0.6c0.3,0 0.6,-0.1 0.9,-0.2s0.4,-0.4 0.6,-0.7l0,-2.9c-0.1,-0.3 -0.3,-0.5 -0.6,-0.6s-0.5,-0.2 -0.9,-0.2c-0.6,0 -1,0.2 -1.2,0.7s-0.4,1.1 -0.4,1.8l0,0z" class="st42"/>
<path id="svg_97" d="m392.2,120.4c-0.9,0 -1.6,-0.3 -2.1,-0.9s-0.8,-1.4 -0.8,-2.3l0,-0.2c0,-0.9 0.3,-1.7 0.8,-2.3s1.2,-0.9 1.9,-0.9c0.9,0 1.5,0.3 1.9,0.8s0.7,1.2 0.7,2.1l0,0.7l-4.1,0l0,0c0,0.6 0.2,1.1 0.5,1.5s0.7,0.6 1.2,0.6c0.4,0 0.7,-0.1 1,-0.2s0.5,-0.3 0.8,-0.5l0.5,0.8c-0.2,0.2 -0.5,0.4 -0.9,0.6s-0.9,0.2 -1.4,0.2zm-0.2,-5.6c-0.4,0 -0.7,0.2 -1,0.5s-0.4,0.7 -0.5,1.2l0,0l2.9,0l0,-0.2c0,-0.5 -0.1,-0.8 -0.4,-1.1s-0.5,-0.4 -1,-0.4z" class="st42"/>
</g>
<g id="svg_98">
<g id="svg_99">
<path id="svg_100" d="m128.4,356.6l1,-0.2l0,-6.9l-1,-0.2l0,-1.2l4,0l0,1.2l-1,0.2l0,2.6l0.8,0l1.9,-2.7l-0.6,-0.1l0,-1.2l3.8,0l0,1.2l-1,0.2l-2.4,3.2l2.7,3.8l1,0.2l0,1.2l-3.8,0l0,-1.2l0.6,-0.1l-1.9,-2.8l-1.1,0l0,2.7l1,0.2l0,1.2l-4,0l0,-1.3z" class="st42"/>
<path id="svg_101" d="m143,356.9c-0.2,0.3 -0.5,0.6 -0.9,0.8c-0.3,0.2 -0.7,0.3 -1.2,0.3c-0.8,0 -1.4,-0.2 -1.8,-0.7c-0.4,-0.5 -0.6,-1.2 -0.6,-2.3l0,-3l-0.8,-0.2l0,-1.2l0.8,0l1.9,0l0,4.3c0,0.5 0.1,0.9 0.3,1.1c0.2,0.2 0.4,0.3 0.8,0.3c0.3,0 0.6,0 0.8,-0.1s0.4,-0.2 0.5,-0.4l0,-3.8l-0.8,-0.2l0,-1.2l0.8,0l1.9,0l0,5.8l0.9,0.2l0,1.2l-2.6,0l0,-0.9z" class="st42"/>
<path id="svg_102" d="m153.4,354.4c0,1.1 -0.2,1.9 -0.7,2.6c-0.5,0.6 -1.2,1 -2.1,1c-0.4,0 -0.8,-0.1 -1.1,-0.3c-0.3,-0.2 -0.6,-0.4 -0.8,-0.8l-0.1,0.9l-1.6,0l0,-9l-1,-0.2l0,-1.2l3,0l0,3.9c0.2,-0.3 0.5,-0.5 0.7,-0.7c0.3,-0.2 0.6,-0.2 1,-0.2c0.9,0 1.6,0.3 2.1,1c0.5,0.7 0.7,1.6 0.7,2.8l0,0.2l-0.1,0zm-1.9,-0.1c0,-0.7 -0.1,-1.3 -0.3,-1.7c-0.2,-0.4 -0.6,-0.6 -1.1,-0.6c-0.3,0 -0.6,0.1 -0.8,0.2c-0.2,0.1 -0.4,0.3 -0.5,0.5l0,3c0.1,0.2 0.3,0.4 0.5,0.5c0.2,0.1 0.5,0.2 0.8,0.2c0.5,0 0.8,-0.2 1,-0.5s0.3,-0.9 0.3,-1.5l0,-0.1l0.1,0z" class="st42"/>
<path id="svg_103" d="m157.8,358c-1,0 -1.9,-0.3 -2.5,-1c-0.6,-0.7 -0.9,-1.5 -0.9,-2.5l0,-0.3c0,-1.1 0.3,-1.9 0.9,-2.6c0.6,-0.7 1.4,-1 2.4,-1c1,0 1.7,0.3 2.3,0.9c0.5,0.6 0.8,1.4 0.8,2.4l0,1.1l-4.3,0l0,0c0,0.5 0.2,0.9 0.5,1.2c0.3,0.3 0.7,0.5 1.1,0.5c0.4,0 0.8,0 1.1,-0.1c0.3,-0.1 0.6,-0.2 0.9,-0.4l0.5,1.2c-0.3,0.2 -0.7,0.4 -1.2,0.6c-0.5,-0.1 -1,0 -1.6,0zm-0.2,-6c-0.4,0 -0.6,0.1 -0.8,0.4c-0.2,0.3 -0.3,0.6 -0.4,1.1l0,0l2.4,0l0,-0.2c0,-0.4 -0.1,-0.7 -0.3,-1s-0.5,-0.3 -0.9,-0.3z" class="st42"/>
<path id="svg_104" d="m161.7,356.6l0.9,-0.2l0,-4.4l-1,-0.2l0,-1.2l2.8,0l0.1,1c0.2,-0.4 0.4,-0.7 0.7,-0.9c0.3,-0.2 0.6,-0.3 0.9,-0.3c0.1,0 0.2,0 0.3,0c0.1,0 0.2,0 0.3,0.1l-0.2,1.8l-0.8,0c-0.3,0 -0.5,0.1 -0.7,0.2c-0.2,0.1 -0.3,0.3 -0.4,0.5l0,3.5l0.9,0.2l0,1.2l-3.8,0l0,-1.3z" class="st42"/>
<path id="svg_105" d="m167.2,356.6l0.9,-0.2l0,-4.4l-1,-0.2l0,-1.2l2.8,0l0.1,1c0.2,-0.4 0.5,-0.7 0.9,-0.9s0.7,-0.3 1.2,-0.3c0.7,0 1.3,0.2 1.7,0.7s0.6,1.2 0.6,2.1l0,3.1l0.9,0.2l0,1.2l-3.7,0l0,-1.2l0.8,-0.2l0,-3.1c0,-0.5 -0.1,-0.8 -0.3,-1c-0.2,-0.2 -0.5,-0.3 -0.9,-0.3c-0.3,0 -0.5,0.1 -0.7,0.2c-0.2,0.1 -0.4,0.3 -0.5,0.4l0,3.9l0.8,0.2l0,1.2l-3.7,0l0,-1.2l0.1,0z" class="st42"/>
<path id="svg_106" d="m179.3,358c-1,0 -1.9,-0.3 -2.5,-1c-0.6,-0.7 -0.9,-1.5 -0.9,-2.5l0,-0.3c0,-1.1 0.3,-1.9 0.9,-2.6c0.6,-0.7 1.4,-1 2.4,-1c1,0 1.7,0.3 2.3,0.9c0.5,0.6 0.8,1.4 0.8,2.4l0,1.1l-4.3,0l0,0c0,0.5 0.2,0.9 0.5,1.2c0.3,0.3 0.7,0.5 1.1,0.5c0.4,0 0.8,0 1.1,-0.1c0.3,-0.1 0.6,-0.2 0.9,-0.4l0.5,1.2c-0.3,0.2 -0.7,0.4 -1.2,0.6c-0.5,-0.1 -1,0 -1.6,0zm-0.2,-6c-0.4,0 -0.6,0.1 -0.8,0.4c-0.2,0.3 -0.3,0.6 -0.4,1.1l0,0l2.4,0l0,-0.2c0,-0.4 -0.1,-0.7 -0.3,-1s-0.5,-0.3 -0.9,-0.3z" class="st42"/>
<path id="svg_107" d="m186,348.9l0,1.8l1.3,0l0,1.4l-1.3,0l0,3.7c0,0.3 0.1,0.5 0.2,0.6c0.1,0.1 0.3,0.2 0.5,0.2c0.1,0 0.2,0 0.3,0c0.1,0 0.2,0 0.3,-0.1l0.2,1.4c-0.2,0.1 -0.4,0.1 -0.6,0.1c-0.2,0 -0.4,0 -0.7,0c-0.7,0 -1.2,-0.2 -1.5,-0.6c-0.4,-0.4 -0.5,-0.9 -0.5,-1.7l0,-3.7l-1.2,0l0,-1.4l1.1,0l0,-1.8l1.9,0l0,0.1z" class="st42"/>
<path id="svg_108" d="m191.6,358c-1,0 -1.9,-0.3 -2.5,-1c-0.6,-0.7 -0.9,-1.5 -0.9,-2.5l0,-0.3c0,-1.1 0.3,-1.9 0.9,-2.6c0.6,-0.7 1.4,-1 2.4,-1c1,0 1.7,0.3 2.3,0.9c0.5,0.6 0.8,1.4 0.8,2.4l0,1.1l-4.3,0l0,0c0,0.5 0.2,0.9 0.5,1.2c0.3,0.3 0.7,0.5 1.1,0.5c0.4,0 0.8,0 1.1,-0.1c0.3,-0.1 0.6,-0.2 0.9,-0.4l0.5,1.2c-0.3,0.2 -0.7,0.4 -1.2,0.6c-0.5,-0.1 -1,0 -1.6,0zm-0.1,-6c-0.4,0 -0.6,0.1 -0.8,0.4c-0.2,0.3 -0.3,0.6 -0.4,1.1l0,0l2.4,0l0,-0.2c0,-0.4 -0.1,-0.7 -0.3,-1s-0.5,-0.3 -0.9,-0.3z" class="st42"/>
<path stroke="null" style="vector-effect: non-scaling-stroke;" id="svg_110" d="m201,353.1l-1.3,0l-0.2,-0.9c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3c-0.2,-0.1 -0.4,-0.1 -0.7,-0.1c-0.3,0 -0.6,0.1 -0.8,0.2s-0.3,0.3 -0.3,0.6c0,0.2 0.1,0.4 0.3,0.5c0.2,0.1 0.6,0.3 1.1,0.4c0.9,0.2 1.5,0.4 2,0.8c0.4,0.3 0.6,0.8 0.6,1.4c0,0.6 -0.3,1.2 -0.8,1.6c-0.6,0.4 -1.3,0.6 -2.2,0.6c-0.6,0 -1.1,-0.1 -1.5,-0.2c-0.5,-0.2 -0.9,-0.4 -1.2,-0.7l0,-1.6l1.4,0l0.3,0.9c0.1,0.1 0.3,0.2 0.5,0.2s0.4,0.1 0.6,0.1c0.4,0 0.7,-0.1 0.9,-0.2c0.2,-0.1 0.3,-0.3 0.3,-0.6c0,-0.2 -0.1,-0.4 -0.3,-0.6c-0.2,-0.2 -0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.5,-0.4 -1.9,-0.8c-0.4,-0.3 -0.6,-0.8 -0.6,-1.4c0,-0.6 0.2,-1.1 0.7,-1.6s1.2,-0.7 2.1,-0.7c0.6,0 1.1,0.1 1.6,0.2c0.5,0.2 0.9,0.3 1.2,0.6l-0.2,2z" class="st42"/>
<path id="svg_111" d="m208.9,356.5c0.3,0 0.6,-0.1 0.8,-0.3c0.2,-0.2 0.3,-0.5 0.3,-0.8l1.8,0l0,0c0,0.7 -0.3,1.3 -0.8,1.8c-0.6,0.5 -1.3,0.7 -2.1,0.7c-1.1,0 -1.9,-0.3 -2.5,-1s-0.9,-1.5 -0.9,-2.6l0,-0.2c0,-1.1 0.3,-1.9 0.9,-2.6s1.4,-1 2.5,-1c0.6,0 1.1,0.1 1.6,0.3c0.5,0.2 0.8,0.4 1.1,0.7l0,1.9l-1.6,0l-0.3,-1.1c-0.1,-0.1 -0.2,-0.2 -0.4,-0.2c-0.1,-0.1 -0.3,-0.1 -0.5,-0.1c-0.5,0 -0.9,0.2 -1.2,0.6s-0.3,0.9 -0.3,1.5l0,0.2c0,0.6 0.1,1.2 0.3,1.6c0.4,0.4 0.8,0.6 1.3,0.6z" class="st42"/>
<path fill="black" id="svg_112" d="m212.5,348.6l0,-1.2l3,0l0,9l0.9,0.2l0,1.2l-3.8,0l0,-1.2l0.9,-0.2l0,-7.6l-1,-0.2z" class="st42"/>
<path id="svg_113" d="m222,356.9c-0.2,0.3 -0.5,0.6 -0.9,0.8c-0.3,0.2 -0.7,0.3 -1.2,0.3c-0.8,0 -1.4,-0.2 -1.8,-0.7c-0.4,-0.5 -0.6,-1.2 -0.6,-2.3l0,-3l-0.8,-0.2l0,-1.2l0.8,0l1.9,0l0,4.3c0,0.5 0.1,0.9 0.3,1.1c0.2,0.2 0.4,0.3 0.8,0.3c0.3,0 0.6,0 0.8,-0.1s0.4,-0.2 0.5,-0.4l0,-3.8l-0.8,-0.2l0,-1.2l0.8,0l1.9,0l0,5.8l0.9,0.2l0,1.2l-2.6,0l0,-0.9z" class="st42"/>
<path id="svg_114" d="m231.4,353.1l-1.3,0l-0.2,-0.9c-0.1,-0.1 -0.3,-0.2 -0.5,-0.3c-0.2,-0.1 -0.4,-0.1 -0.7,-0.1c-0.3,0 -0.6,0.1 -0.8,0.2s-0.3,0.3 -0.3,0.6c0,0.2 0.1,0.4 0.3,0.5c0.2,0.1 0.6,0.3 1.1,0.4c0.9,0.2 1.5,0.4 2,0.8c0.4,0.3 0.6,0.8 0.6,1.4c0,0.6 -0.3,1.2 -0.8,1.6c-0.6,0.4 -1.3,0.6 -2.2,0.6c-0.6,0 -1.1,-0.1 -1.5,-0.2c-0.5,-0.2 -0.9,-0.4 -1.2,-0.7l0,-1.6l1.4,0l0.3,0.9c0.1,0.1 0.3,0.2 0.5,0.2s0.4,0.1 0.6,0.1c0.4,0 0.7,-0.1 0.9,-0.2c0.2,-0.1 0.3,-0.3 0.3,-0.6c0,-0.2 -0.1,-0.4 -0.3,-0.6c-0.2,-0.2 -0.6,-0.3 -1.1,-0.4c-0.8,-0.2 -1.5,-0.4 -1.9,-0.8c-0.4,-0.3 -0.6,-0.8 -0.6,-1.4c0,-0.6 0.2,-1.1 0.7,-1.6s1.2,-0.7 2.1,-0.7c0.6,0 1.1,0.1 1.6,0.2c0.5,0.2 0.9,0.3 1.2,0.6l-0.2,2z" class="st42"/>
<path id="svg_115" d="m235.4,348.9l0,1.8l1.3,0l0,1.4l-1.3,0l0,3.7c0,0.3 0.1,0.5 0.2,0.6c0.1,0.1 0.3,0.2 0.5,0.2c0.1,0 0.2,0 0.3,0c0.1,0 0.2,0 0.3,-0.1l0.2,1.4c-0.2,0.1 -0.4,0.1 -0.6,0.1c-0.2,0 -0.4,0 -0.7,0c-0.7,0 -1.2,-0.2 -1.5,-0.6c-0.4,-0.4 -0.5,-0.9 -0.5,-1.7l0,-3.7l-1.1,0l0,-1.4l1.1,0l0,-1.8l1.8,0l0,0.1z" class="st42"/>
<path id="svg_116" d="m241,358c-1,0 -1.9,-0.3 -2.5,-1c-0.6,-0.7 -0.9,-1.5 -0.9,-2.5l0,-0.3c0,-1.1 0.3,-1.9 0.9,-2.6c0.6,-0.7 1.4,-1 2.4,-1c1,0 1.7,0.3 2.3,0.9c0.5,0.6 0.8,1.4 0.8,2.4l0,1.1l-4.3,0l0,0c0,0.5 0.2,0.9 0.5,1.2c0.3,0.3 0.7,0.5 1.1,0.5c0.4,0 0.8,0 1.1,-0.1c0.3,-0.1 0.6,-0.2 0.9,-0.4l0.5,1.2c-0.3,0.2 -0.7,0.4 -1.2,0.6c-0.5,-0.1 -1,0 -1.6,0zm-0.2,-6c-0.4,0 -0.6,0.1 -0.8,0.4c-0.2,0.3 -0.3,0.6 -0.4,1.1l0,0l2.4,0l0,-0.2c0,-0.4 -0.1,-0.7 -0.3,-1s-0.5,-0.3 -0.9,-0.3z" class="st42"/>
<path id="svg_117" d="m245,356.6l0.9,-0.2l0,-4.4l-1,-0.2l0,-1.2l2.8,0l0.1,1c0.2,-0.4 0.4,-0.7 0.7,-0.9c0.3,-0.2 0.6,-0.3 0.9,-0.3c0.1,0 0.2,0 0.3,0c0.1,0 0.2,0 0.3,0.1l-0.2,1.8l-0.8,0c-0.3,0 -0.5,0.1 -0.7,0.2c-0.2,0.1 -0.3,0.3 -0.4,0.5l0,3.5l0.9,0.2l0,1.2l-3.8,0l0,-1.3z" class="st42"/>
</g>
</g>
<line id="svg_118" y2="191.7" x2="201.6" y1="191.7" x1="360.6" class="st62"/>
<line id="svg_119" y2="228.8" x2="160" y1="228.8" x1="360.6" class="st62"/>
<line id="svg_120" y2="118.3" x2="243.4" y1="118.3" x1="360.6" class="st62"/>
</g>
<g id="Node">
<g id="Node_x5F_level3_x5F_1">
<g id="Isolation_Mode"/>
</g>
<polygon id="svg_18" points="182.7,139.9 147.1,160.4 111.4,139.9 111.4,98.7 147.1,78.2 182.7,98.7 " class="st16"/>
<polygon id="svg_19" points="129.3,150.2 147.1,160.4 182.7,139.9 182.7,119.3 " class="st13"/>
<polygon id="svg_20" points="163.5,147.7 157.4,137.2 179.9,124.2 179.9,138.3 " class="st7"/>
<g id="svg_21">
<path id="svg_22" d="m162.5,139.3c0.4,-0.3 0.9,-0.3 1.3,-0.2c0.4,0.1 0.8,0.4 1,0.8l0.3,0.5c0.3,0.4 0.3,0.9 0.2,1.3c-0.1,0.4 -0.4,0.8 -0.8,1l-1.4,0.8l-0.2,-0.3l0.3,-0.3l-1.6,-2.7l-0.4,0.2l-0.2,-0.3l0.4,-0.2l1.1,-0.6zm-0.4,0.7l1.6,2.7l0.6,-0.3c0.3,-0.2 0.5,-0.4 0.6,-0.8c0.1,-0.3 0,-0.6 -0.2,-1l-0.3,-0.5c-0.2,-0.3 -0.4,-0.5 -0.7,-0.6c-0.3,-0.1 -0.6,-0.1 -0.9,0.1l-0.7,0.4z" class="st8"/>
<path id="svg_23" d="m165.7,140.3c-0.2,-0.4 -0.3,-0.8 -0.2,-1.1c0.1,-0.4 0.3,-0.6 0.6,-0.9c0.4,-0.2 0.7,-0.2 1.1,-0.1c0.3,0.1 0.6,0.4 0.9,0.8l0,0.1c0.2,0.4 0.3,0.8 0.2,1.1c-0.1,0.4 -0.3,0.6 -0.6,0.8c-0.4,0.2 -0.7,0.2 -1.1,0.1c-0.4,-0.1 -0.7,-0.3 -0.9,-0.8l0,0zm0.5,-0.2c0.2,0.3 0.3,0.5 0.6,0.6c0.2,0.1 0.4,0.1 0.7,0c0.2,-0.1 0.3,-0.3 0.4,-0.6c0,-0.2 -0.1,-0.5 -0.2,-0.8l0,-0.1c-0.2,-0.3 -0.3,-0.5 -0.6,-0.6c-0.2,-0.1 -0.4,-0.1 -0.7,0c-0.2,0.1 -0.3,0.3 -0.4,0.6c-0.1,0.3 0,0.6 0.2,0.9l0,0z" class="st8"/>
<path id="svg_24" d="m170.1,139.1c0.2,-0.1 0.3,-0.2 0.3,-0.4c0.1,-0.2 0.1,-0.3 0,-0.5l0.4,-0.2l0,0c0.1,0.2 0.2,0.5 0,0.8c-0.1,0.3 -0.3,0.5 -0.6,0.7c-0.4,0.2 -0.7,0.3 -1.1,0.1c-0.3,-0.1 -0.6,-0.4 -0.8,-0.7l-0.1,-0.1c-0.2,-0.4 -0.3,-0.7 -0.2,-1.1s0.3,-0.6 0.6,-0.9c0.2,-0.1 0.4,-0.2 0.6,-0.2c0.2,0 0.4,0 0.6,0l0.3,0.6l-0.4,0.2l-0.3,-0.3c-0.1,0 -0.2,0 -0.3,0c-0.1,0 -0.2,0.1 -0.3,0.1c-0.2,0.1 -0.4,0.3 -0.4,0.6c0,0.2 0.1,0.5 0.2,0.7l0.1,0.1c0.2,0.3 0.3,0.5 0.5,0.6c0.5,0 0.7,0 0.9,-0.1z" class="st8"/>
<path id="svg_25" d="m169.8,135.2l-0.2,-0.3l0.9,-0.5l1.2,2.1l0.3,-0.2l0.2,-1l-0.3,0.1l-0.2,-0.3l1.1,-0.6l0.2,0.3l-0.3,0.2l-0.2,1.2l1.4,0.6l0.3,-0.1l0.2,0.3l-1.1,0.6l-0.2,-0.3l0.2,-0.2l-1.2,-0.5l-0.3,0.2l0.5,0.8l0.4,-0.2l0.2,0.3l-1.2,0.7l-0.2,-0.3l0.3,-0.3l-1.7,-3l-0.3,0.4z" class="st8"/>
<path id="svg_26" d="m175.9,136.3c-0.4,0.2 -0.7,0.3 -1.1,0.1s-0.6,-0.4 -0.9,-0.7l-0.1,-0.1c-0.2,-0.4 -0.3,-0.7 -0.2,-1.1c0.1,-0.4 0.3,-0.6 0.6,-0.8c0.3,-0.2 0.7,-0.2 1,-0.1s0.5,0.3 0.7,0.7l0.2,0.3l-1.7,1l0,0c0.1,0.2 0.3,0.4 0.5,0.5c0.2,0.1 0.4,0.1 0.6,-0.1c0.2,-0.1 0.3,-0.2 0.4,-0.3c0.1,-0.1 0.2,-0.2 0.2,-0.4l0.4,0.2c0,0.1 -0.1,0.3 -0.2,0.4c0,0.1 -0.2,0.2 -0.4,0.4zm-1.4,-2.3c-0.2,0.1 -0.3,0.2 -0.3,0.4s0,0.4 0.1,0.6l0,0l1.2,-0.7l0,-0.1c-0.1,-0.2 -0.2,-0.3 -0.4,-0.4c-0.2,0.1 -0.4,0.1 -0.6,0.2z" class="st8"/>
<path id="svg_27" d="m176,133.1l-0.2,-0.3l0.8,-0.5l0.3,0.3c0,-0.2 0,-0.3 0.1,-0.5c0.1,-0.1 0.2,-0.2 0.3,-0.3c0,0 0.1,0 0.1,-0.1c0,0 0.1,0 0.1,0l0.2,0.5l-0.3,0.1c-0.1,0.1 -0.2,0.2 -0.3,0.3c-0.1,0.1 -0.1,0.2 -0.1,0.4l0.8,1.5l0.4,-0.2l0.2,0.3l-1.2,0.7l-0.2,-0.3l0.3,-0.3l-1.1,-1.8l-0.2,0.2z" class="st8"/>
</g>
<g id="svg_28">
<path id="svg_29" d="m141.1,151.8l-0.2,-0.3l0.9,-0.5l1.2,2.1l0.3,-0.2l0.2,-1l-0.3,0.1l-0.2,-0.3l1.1,-0.6l0.2,0.3l-0.3,0.2l-0.2,1.2l1.4,0.6l0.3,-0.1l0.2,0.3l-1.1,0.6l-0.2,-0.3l0.2,-0.2l-1.2,-0.5l-0.3,0.2l0.5,0.8l0.4,-0.2l0.2,0.3l-1.2,0.7l-0.2,-0.3l0.3,-0.3l-1.7,-3l-0.3,0.4z" class="st8"/>
<path id="svg_30" d="m147.6,152c0,0.2 0,0.3 -0.1,0.5c-0.1,0.1 -0.2,0.3 -0.4,0.4c-0.3,0.2 -0.5,0.2 -0.8,0.1c-0.3,-0.1 -0.5,-0.3 -0.7,-0.7l-0.7,-1.1l-0.3,0.1l-0.2,-0.3l0.3,-0.2l0.5,-0.3l0.9,1.5c0.2,0.3 0.3,0.4 0.4,0.5s0.3,0 0.5,-0.1c0.2,-0.1 0.3,-0.2 0.4,-0.3c0.1,-0.1 0.1,-0.3 0.1,-0.4l-0.9,-1.5l-0.4,0.1l-0.2,-0.3l0.3,-0.2l0.5,-0.3l1.3,2.2l0.3,-0.1l0.2,0.3l-0.7,0.4l-0.3,-0.3z" class="st8"/>
<path id="svg_31" d="m150.6,149.2c0.2,0.4 0.3,0.7 0.2,1c0,0.3 -0.2,0.6 -0.5,0.8c-0.2,0.1 -0.3,0.1 -0.5,0.2c-0.2,0 -0.3,0 -0.5,-0.1l0.1,0.4l-0.4,0.2l-1.9,-3.3l-0.4,0.2l-0.2,-0.3l0.9,-0.5l0.8,1.4c0,-0.2 0.1,-0.3 0.1,-0.4c0.1,-0.1 0.2,-0.2 0.4,-0.3c0.3,-0.2 0.6,-0.2 1,0s0.7,0.2 0.9,0.7l0,0zm-0.5,0.2c-0.2,-0.3 -0.4,-0.5 -0.6,-0.6c-0.2,-0.1 -0.4,-0.1 -0.7,0c-0.1,0.1 -0.2,0.2 -0.3,0.3c-0.1,0.1 -0.1,0.3 -0.1,0.4l0.7,1.1c0.1,0.1 0.3,0.1 0.4,0.1c0.1,0 0.3,0 0.4,-0.1c0.2,-0.1 0.3,-0.3 0.4,-0.5c0.1,-0.2 0,-0.4 -0.2,-0.7l0,0z" class="st8"/>
<path id="svg_32" d="m153,149.5c-0.4,0.2 -0.7,0.3 -1.1,0.1s-0.6,-0.4 -0.9,-0.7l-0.1,-0.1c-0.2,-0.4 -0.3,-0.7 -0.2,-1.1c0.1,-0.4 0.3,-0.6 0.6,-0.8c0.3,-0.2 0.7,-0.2 1,-0.1s0.5,0.3 0.7,0.7l0.2,0.3l-1.7,1l0,0c0.1,0.2 0.3,0.4 0.5,0.5c0.2,0.1 0.4,0.1 0.6,-0.1c0.2,-0.1 0.3,-0.2 0.4,-0.3c0.1,-0.1 0.2,-0.2 0.2,-0.4l0.4,0.2c0,0.1 -0.1,0.3 -0.2,0.4s-0.2,0.2 -0.4,0.4zm-1.4,-2.3c-0.2,0.1 -0.3,0.2 -0.3,0.4c0,0.2 0,0.4 0.1,0.6l0,0l1.2,-0.7l0,-0.1c-0.1,-0.2 -0.2,-0.3 -0.4,-0.4s-0.4,0.1 -0.6,0.2z" class="st8"/>
<path id="svg_33" d="m152.4,145.2l-0.2,-0.3l0.9,-0.5l1.9,3.3l0.4,-0.2l0.2,0.3l-1.2,0.7l-0.2,-0.3l0.3,-0.3l-1.7,-3l-0.4,0.3z" class="st8"/>
<path id="svg_34" d="m157.1,147.1c-0.4,0.2 -0.7,0.3 -1.1,0.1s-0.6,-0.4 -0.9,-0.7l-0.1,-0.1c-0.2,-0.4 -0.3,-0.7 -0.2,-1.1c0.1,-0.4 0.3,-0.6 0.6,-0.8c0.3,-0.2 0.7,-0.2 1,-0.1s0.5,0.3 0.7,0.7l0.2,0.3l-1.7,1l0,0c0.1,0.2 0.3,0.4 0.5,0.5c0.2,0.1 0.4,0.1 0.6,-0.1c0.2,-0.1 0.3,-0.2 0.4,-0.3c0.1,-0.1 0.2,-0.2 0.2,-0.4l0.4,0.2c0,0.1 -0.1,0.3 -0.2,0.4s-0.2,0.3 -0.4,0.4zm-1.4,-2.3c-0.2,0.1 -0.3,0.2 -0.3,0.4s0,0.4 0.1,0.6l0,0l1.2,-0.7l0,-0.1c-0.1,-0.2 -0.2,-0.3 -0.4,-0.4c-0.2,0.1 -0.4,0.1 -0.6,0.2z" class="st8"/>
<path id="svg_35" d="m157.5,142.6l0.4,0.6l0.5,-0.3l0.2,0.3l-0.5,0.3l0.9,1.6c0.1,0.1 0.1,0.2 0.2,0.2c0.1,0 0.2,0 0.2,0c0,0 0.1,-0.1 0.1,-0.1c0,0 0.1,-0.1 0.1,-0.1l0.2,0.3c0,0.1 -0.1,0.1 -0.2,0.2c-0.1,0.1 -0.2,0.1 -0.2,0.2c-0.2,0.1 -0.4,0.1 -0.6,0.1c-0.2,0 -0.3,-0.2 -0.5,-0.4l-0.9,-1.6l-0.4,0.2l-0.2,-0.3l0.4,-0.2l-0.4,-0.6l0.7,-0.4z" class="st8"/>
</g>
<polygon id="svg_36" points="225.1,160.4 189.4,139.9 189.4,98.7 225.1,78.2 260.7,98.7 260.7,139.9 " class="st16"/>
<polygon id="svg_37" points="189.4,119.3 189.4,139.9 225.1,160.4 242.9,150.2 " class="st13"/>
<polygon id="svg_38" points="208.7,147.7 214.8,137.2 237.3,150.2 225.1,157.2 " class="st7"/>
<g id="svg_39">
<path id="svg_40" d="m215.5,142.6c0.5,0.3 0.7,0.6 0.8,1c0.1,0.4 0,0.9 -0.2,1.3l-0.3,0.5c-0.3,0.4 -0.6,0.7 -1,0.8c-0.4,0.1 -0.9,0 -1.3,-0.2l-1.4,-0.8l0.2,-0.3l0.4,0.2l1.6,-2.7l-0.3,-0.3l0.2,-0.3l0.4,0.2l0.9,0.6zm-0.8,0l-1.6,2.7l0.6,0.3c0.3,0.2 0.6,0.2 0.9,0.1c0.3,-0.1 0.6,-0.3 0.7,-0.6l0.3,-0.5c0.2,-0.3 0.2,-0.6 0.2,-1c-0.1,-0.3 -0.3,-0.6 -0.6,-0.8l-0.5,-0.2z" class="st8"/>
<path id="svg_41" d="m216.2,145.9c0.2,-0.4 0.5,-0.6 0.8,-0.8c0.3,-0.1 0.7,-0.1 1,0.1c0.4,0.2 0.6,0.5 0.6,0.9c0.1,0.4 0,0.7 -0.2,1.1l0,0.1c-0.2,0.4 -0.5,0.6 -0.8,0.8c-0.3,0.1 -0.7,0.1 -1,-0.1c-0.4,-0.2 -0.6,-0.5 -0.6,-0.9c-0.1,-0.4 0,-0.8 0.2,-1.2l0,0zm0.5,0.3c-0.2,0.3 -0.2,0.5 -0.2,0.8c0,0.2 0.1,0.4 0.4,0.6c0.2,0.1 0.4,0.1 0.7,0c0.2,-0.1 0.4,-0.3 0.6,-0.6l0,-0.1c0.2,-0.3 0.2,-0.5 0.2,-0.8c0,-0.2 -0.1,-0.4 -0.4,-0.6c-0.2,-0.1 -0.4,-0.1 -0.7,0c-0.3,0.2 -0.5,0.4 -0.6,0.7l0,0z" class="st8"/>
<path id="svg_42" d="m219.5,149.1c0.2,0.1 0.3,0.1 0.5,0.1c0.2,0 0.3,-0.1 0.4,-0.2l0.4,0.2l0,0c-0.1,0.2 -0.3,0.4 -0.6,0.4c-0.3,0.1 -0.6,0 -0.9,-0.2c-0.4,-0.2 -0.6,-0.5 -0.6,-0.9c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.8,-0.7s0.7,-0.1 1.1,0.1c0.2,0.1 0.4,0.3 0.5,0.4s0.2,0.3 0.2,0.5l-0.3,0.6l-0.4,-0.2l0.1,-0.5c0,-0.1 -0.1,-0.2 -0.1,-0.3c-0.1,-0.1 -0.2,-0.2 -0.3,-0.2c-0.2,-0.1 -0.5,-0.2 -0.7,0c-0.2,0.1 -0.4,0.3 -0.5,0.6l-0.1,0.1c-0.2,0.3 -0.2,0.5 -0.2,0.7c0.1,0.4 0.2,0.6 0.4,0.7z" class="st8"/>
<path id="svg_43" d="m222.7,146.8l0.2,-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9,-0.4l-0.2,-0.2l0.2,-0.3l1.1,0.6l-0.2,0.3l-0.3,-0.1l-1.1,0.4l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1,-0.6l0.2,-0.3l0.3,0.1l-0.2,-1.2l-0.3,-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.5,-0.2z" class="st8"/>
<path id="svg_44" d="m224.8,152.7c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8s0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7s0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8c0.1,0.3 0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7c0,0.2 0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2c0.1,0 0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0s-0.4,-0.2 -0.7,-0.3zm1.3,-2.4c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.4,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.2 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_45" d="m227.6,151.2l0.2,-0.3l0.8,0.5l-0.2,0.4c0.1,-0.1 0.3,-0.1 0.4,-0.2c0.1,0 0.3,0 0.4,0.1c0,0 0.1,0 0.1,0.1c0,0 0.1,0.1 0.1,0.1l-0.3,0.4l-0.3,-0.2c-0.1,-0.1 -0.2,-0.1 -0.4,-0.1c-0.1,0 -0.2,0.1 -0.3,0.1l-0.8,1.5l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.1,-1.8l-0.3,-0.4z" class="st8"/>
</g>
<g id="svg_46">
<path id="svg_47" d="m194,130.3l0.2,-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9,-0.4l-0.2,-0.2l0.2,-0.3l1.1,0.6l-0.2,0.3l-0.3,-0.1l-1.1,0.4l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1,-0.6l0.2,-0.3l0.3,0.1l-0.2,-1.2l-0.3,-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.5,-0.2z" class="st8"/>
<path id="svg_48" d="m197.1,136.1c-0.2,0.1 -0.3,0.1 -0.5,0.1c-0.2,0 -0.3,0 -0.5,-0.1c-0.3,-0.2 -0.4,-0.4 -0.5,-0.6c-0.1,-0.3 0,-0.6 0.2,-1l0.7,-1.1l-0.3,-0.2l0.2,-0.3l0.3,0.2l0.5,0.3l-0.9,1.5c-0.2,0.3 -0.2,0.5 -0.2,0.6c0,0.1 0.1,0.3 0.3,0.4c0.2,0.1 0.3,0.1 0.5,0.1c0.1,0 0.3,-0.1 0.4,-0.1l0.9,-1.5l-0.3,-0.3l0.2,-0.3l0.3,0.2l0.5,0.3l-1.3,2.2l0.3,0.2l-0.2,0.3l-0.7,-0.4l0.1,-0.5z" class="st8"/>
<path id="svg_49" d="m201,137.3c-0.2,0.4 -0.5,0.6 -0.8,0.7s-0.6,0.1 -0.9,-0.1c-0.2,-0.1 -0.3,-0.2 -0.4,-0.3c-0.1,-0.1 -0.1,-0.3 -0.1,-0.5l-0.2,0.3l-0.4,-0.2l1.9,-3.3l-0.3,-0.3l0.2,-0.3l0.9,0.5l-0.8,1.4c0.1,-0.1 0.3,-0.1 0.5,-0.1s0.3,0.1 0.5,0.2c0.3,0.2 0.5,0.5 0.5,0.8s-0.3,0.7 -0.6,1.2l0,0zm-0.4,-0.4c0.2,-0.3 0.3,-0.6 0.3,-0.8c0,-0.2 -0.1,-0.4 -0.3,-0.6c-0.1,-0.1 -0.3,-0.1 -0.4,-0.1c-0.1,0 -0.3,0.1 -0.4,0.1l-0.7,1.1c0,0.2 0,0.3 0.1,0.4c0.1,0.1 0.2,0.2 0.3,0.3c0.2,0.1 0.4,0.1 0.6,0c0.1,0.1 0.3,-0.1 0.5,-0.4l0,0z" class="st8"/>
<path id="svg_50" d="m202,139.4c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7c0.4,-0.1 0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8c0.1,0.3 0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7c0,0.2 0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2c0.1,0 0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0c-0.3,-0.1 -0.5,-0.1 -0.7,-0.3zm1.2,-2.3c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.3,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.2 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_51" d="m205.3,136.8l0.2,-0.3l0.9,0.5l-1.9,3.3l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.4,-0.3z" class="st8"/>
<path id="svg_52" d="m206.1,141.8c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7c0.4,-0.1 0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8c0.1,0.3 0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7c0,0.2 0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2c0.1,0 0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0c-0.3,-0.1 -0.5,-0.2 -0.7,-0.3zm1.2,-2.3c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.3,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.3 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_53" d="m210.2,139.9l-0.4,0.6l0.5,0.3l-0.2,0.3l-0.5,-0.3l-0.9,1.6c-0.1,0.1 -0.1,0.2 -0.1,0.3c0,0.1 0.1,0.1 0.2,0.2c0,0 0.1,0 0.1,0.1c0.1,0 0.1,0 0.1,0.1l-0.1,0.4c-0.1,0 -0.1,0 -0.2,0c-0.1,0 -0.2,-0.1 -0.3,-0.1c-0.2,-0.1 -0.3,-0.3 -0.4,-0.4c0,-0.2 0,-0.4 0.1,-0.6l0.9,-1.6l-0.4,-0.2l0.2,-0.3l0.4,0.2l0.4,-0.6l0.6,0z" class="st8"/>
</g>
<polygon id="svg_54" points="147.1,296.4 111.4,275.9 111.4,234.7 147.1,214.1 182.7,234.7 182.7,275.9 " class="st16"/>
<polygon id="svg_55" points="129.3,224.4 147.1,214.1 182.7,234.7 182.7,255.3 " class="st13"/>
<polygon id="svg_56" points="163.5,226.9 157.4,237.4 179.9,250.4 179.9,236.3 " class="st7"/>
<g id="svg_57">
<path id="svg_58" d="m164.2,232.3c0.5,0.3 0.7,0.6 0.8,1c0.1,0.4 0,0.9 -0.2,1.3l-0.3,0.5c-0.3,0.4 -0.6,0.7 -1,0.8c-0.4,0.1 -0.9,0 -1.3,-0.2l-1.4,-0.8l0.2,-0.3l0.4,0.2l1.6,-2.7l-0.3,-0.3l0.2,-0.3l0.4,0.2l0.9,0.6zm-0.8,0l-1.6,2.7l0.6,0.3c0.3,0.2 0.6,0.2 0.9,0.1c0.3,-0.1 0.6,-0.3 0.7,-0.6l0.3,-0.5c0.2,-0.3 0.3,-0.6 0.2,-1c-0.1,-0.3 -0.3,-0.6 -0.6,-0.8l-0.5,-0.2z" class="st8"/>
<path id="svg_59" d="m164.9,235.6c0.2,-0.4 0.5,-0.6 0.8,-0.8c0.3,-0.1 0.7,-0.1 1,0.1c0.4,0.2 0.6,0.5 0.6,0.9c0.1,0.4 0,0.7 -0.2,1.1l0,0.1c-0.2,0.4 -0.5,0.6 -0.8,0.8c-0.3,0.1 -0.7,0.1 -1,-0.1c-0.4,-0.2 -0.6,-0.5 -0.6,-0.9c-0.1,-0.4 0,-0.8 0.2,-1.2l0,0zm0.5,0.3c-0.2,0.3 -0.2,0.5 -0.2,0.8c0,0.2 0.1,0.4 0.4,0.6c0.2,0.1 0.4,0.1 0.7,0c0.2,-0.1 0.4,-0.3 0.6,-0.6l0,-0.1c0.2,-0.3 0.2,-0.5 0.2,-0.8c0,-0.2 -0.1,-0.4 -0.4,-0.6c-0.2,-0.1 -0.4,-0.1 -0.7,0c-0.2,0.2 -0.4,0.4 -0.6,0.7l0,0z" class="st8"/>
<path id="svg_60" d="m168.2,238.8c0.2,0.1 0.3,0.1 0.5,0.1c0.2,0 0.3,-0.1 0.4,-0.2l0.4,0.2l0,0c-0.1,0.2 -0.3,0.4 -0.6,0.4c-0.3,0.1 -0.6,0 -0.9,-0.2c-0.4,-0.2 -0.6,-0.5 -0.6,-0.9c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.8,-0.7c0.3,-0.1 0.7,-0.1 1.1,0.1c0.2,0.1 0.4,0.3 0.5,0.4s0.2,0.3 0.2,0.5l-0.3,0.7l-0.4,-0.2l0.1,-0.5c0,-0.1 -0.1,-0.2 -0.1,-0.3c-0.1,-0.1 -0.2,-0.2 -0.3,-0.2c-0.2,-0.1 -0.5,-0.2 -0.7,0c-0.2,0.1 -0.4,0.3 -0.5,0.6l-0.1,0.1c-0.2,0.3 -0.2,0.5 -0.2,0.7c0.1,0.3 0.2,0.5 0.4,0.6z" class="st8"/>
<path id="svg_61" d="m171.4,236.5l0.2,-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9,-0.4l-0.2,-0.2l0.2,-0.3l1.1,0.6l-0.2,0.3l-0.4,0l-1.1,0.4l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1,-0.6l0.2,-0.3l0.3,0.1l-0.2,-1.2l-0.3,-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.4,-0.3z" class="st8"/>
<path id="svg_62" d="m173.6,242.4c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7c0.4,-0.1 0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8c0.1,0.3 0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7c0,0.2 0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2s0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0c-0.3,-0.1 -0.5,-0.2 -0.7,-0.3zm1.2,-2.4c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.4,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.2 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_63" d="m176.3,240.9l0.2,-0.3l0.8,0.5l-0.2,0.4c0.1,-0.1 0.3,-0.1 0.4,-0.2c0.2,0 0.3,0 0.4,0.1c0,0 0.1,0 0.1,0.1c0,0 0.1,0.1 0.1,0.1l-0.3,0.4l-0.3,-0.2c-0.1,-0.1 -0.2,-0.1 -0.4,-0.1c-0.1,0 -0.2,0.1 -0.3,0.1l-0.8,1.5l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.1,-1.8l-0.3,-0.4z" class="st8"/>
</g>
<g id="svg_64">
<path id="svg_65" d="m142.7,220l0.2,-0.3l0.9,0.5l-1.2,2.1l0.3,0.2l0.9,-0.4l-0.2,-0.2l0.2,-0.3l1.1,0.6l-0.2,0.3l-0.3,-0.1l-1.1,0.4l0.2,1.5l0.3,0.2l-0.2,0.3l-1.1,-0.6l0.2,-0.3l0.3,0.1l-0.2,-1.2l-0.3,-0.2l-0.5,0.8l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.5,-0.2z" class="st8"/>
<path id="svg_66" d="m145.8,225.8c-0.2,0.1 -0.3,0.1 -0.5,0.1s-0.3,0 -0.5,-0.1c-0.3,-0.2 -0.4,-0.4 -0.5,-0.6s0,-0.6 0.2,-1l0.7,-1.1l-0.3,-0.2l0.2,-0.3l0.3,0.2l0.5,0.3l-0.9,1.5c-0.2,0.3 -0.2,0.5 -0.2,0.6s0.1,0.3 0.3,0.4c0.2,0.1 0.3,0.1 0.5,0.1c0.1,0 0.3,-0.1 0.4,-0.1l0.9,-1.5l-0.3,-0.3l0.2,-0.3l0.3,0.2l0.5,0.3l-1.3,2.2l0.3,0.2l-0.2,0.3l-0.7,-0.4l0.1,-0.5z" class="st8"/>
<path id="svg_67" d="m149.8,227c-0.2,0.4 -0.5,0.6 -0.8,0.7c-0.3,0.1 -0.6,0.1 -0.9,-0.1c-0.2,-0.1 -0.3,-0.2 -0.4,-0.3c-0.1,-0.1 -0.1,-0.3 -0.1,-0.5l-0.2,0.3l-0.4,-0.2l1.9,-3.3l-0.3,-0.3l0.2,-0.3l0.9,0.5l-0.8,1.4c0.1,-0.1 0.3,-0.1 0.5,-0.1c0.2,0 0.3,0.1 0.5,0.2c0.3,0.2 0.5,0.5 0.5,0.8c-0.3,0.3 -0.4,0.7 -0.6,1.2l0,0zm-0.5,-0.4c0.2,-0.3 0.3,-0.6 0.3,-0.8c0,-0.2 -0.1,-0.4 -0.3,-0.6c-0.1,-0.1 -0.3,-0.1 -0.4,-0.1c-0.1,0 -0.3,0.1 -0.4,0.1l-0.7,1.1c0,0.2 0,0.3 0.1,0.4s0.2,0.2 0.3,0.3c0.2,0.1 0.4,0.1 0.6,0c0.2,0.1 0.3,0 0.5,-0.4l0,0z" class="st8"/>
<path id="svg_68" d="m150.7,229.1c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7c0.4,-0.1 0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8s0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7s0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2c0.1,0 0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0c-0.3,-0.1 -0.5,-0.1 -0.7,-0.3zm1.2,-2.3c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.4,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.2 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_69" d="m154,226.5l0.2,-0.3l0.9,0.5l-1.9,3.3l0.3,0.3l-0.2,0.3l-1.2,-0.7l0.2,-0.3l0.4,0.2l1.7,-3l-0.4,-0.3z" class="st8"/>
<path id="svg_70" d="m154.8,231.5c-0.4,-0.2 -0.6,-0.5 -0.6,-0.8c-0.1,-0.4 0,-0.7 0.2,-1.1l0.1,-0.1c0.2,-0.4 0.5,-0.6 0.9,-0.7c0.4,-0.1 0.7,-0.1 1,0.1c0.3,0.2 0.5,0.5 0.6,0.8s0,0.6 -0.2,1l-0.2,0.3l-1.7,-1l0,0c-0.1,0.2 -0.2,0.5 -0.2,0.7s0.2,0.4 0.4,0.5c0.2,0.1 0.3,0.1 0.5,0.2c0.1,0 0.3,0 0.4,0l0,0.4c-0.1,0 -0.3,0 -0.5,0c-0.3,-0.1 -0.5,-0.1 -0.7,-0.3zm1.3,-2.3c-0.2,-0.1 -0.3,-0.1 -0.5,0c-0.2,0.1 -0.4,0.2 -0.5,0.4l0,0l1.2,0.7l0,-0.1c0.1,-0.2 0.1,-0.4 0.1,-0.5c0,-0.2 -0.1,-0.4 -0.3,-0.5z" class="st8"/>
<path id="svg_71" d="m158.9,229.6l-0.4,0.6l0.5,0.3l-0.2,0.3l-0.5,-0.3l-0.9,1.6c-0.1,0.1 -0.1,0.2 -0.1,0.3c0,0.1 0.1,0.1 0.2,0.2c0,0 0.1,0 0.1,0.1c0.1,0 0.1,0 0.1,0.1l-0.1,0.4c-0.1,0 -0.1,0 -0.2,0c-0.1,0 -0.2,-0.1 -0.3,-0.1c-0.2,-0.1 -0.3,-0.3 -0.4,-0.4c0,-0.2 0,-0.4 0.1,-0.6l0.9,-1.6l-0.4,-0.2l0.2,-0.3l0.4,0.2l0.4,-0.6l0.6,0z" class="st8"/>
</g>
</g>
<g id="service"/>
<g id="pods"/>
<g id="IP"/>
<g id="deployments"/>
<g id="containers_x2F_volumes"/>
<g id="labels_x2F_selectors"/>
<g id="Layer_14"/>
</g>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 41 KiB

docs/tutorials/stateful-application/gce-volume.yaml
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: mysql-disk
fsType: ext4
docs/tutorials/stateful-application/mysql-deployment.yaml
+51
View File
@@ -0,0 +1,51 @@
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mysql
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
# Use secret in real usage
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
docs/tutorials/stateful-application/run-stateful-application.md
+220
View File
@@ -0,0 +1,220 @@
---
---
{% capture overview %}
This page shows you how to run a single-instance stateful application
in Kubernetes using a PersistentVolume and a Deployment. The
application is MySQL.
{% endcapture %}
{% capture objectives %}
* Create a PersistentVolume referencing a disk in your environment.
* Create a MySQL Deployment.
* Expose MySQL to other pods in the cluster at a known DNS name.
{% endcapture %}
{% capture prerequisites %}
* {% include task-tutorial-prereqs.md %}
* For data persistence we will create a Persistent Volume that
references a disk in your
environment. See
[here](/docs/user-guide/persistent-volumes/#types-of-persistent-volumes) for
the types of environments supported. This Tutorial will demonstrate
`GCEPersistentDisk` but any type will work. `GCEPersistentDisk`
volumes only work on Google Compute Engine.
{% endcapture %}
{% capture lessoncontent %}
### Set up a disk in your environment
You can use any type of persistent volume for your stateful app. See
[Types of Persistent Volumes](/docs/user-guide/persistent-volumes/#types-of-persistent-volumes)
for a list of supported environment disks. For Google Compute Engine, run:
```
gcloud compute disks create --size=20GB mysql-disk
```
Next create a PersistentVolume that points to the `mysql-disk`
disk just created. Here is a configuration file for a PersistentVolume
that points to the Compute Engine disk above:
{% include code.html language="yaml" file="gce-volume.yaml" ghlink="/docs/tutorials/stateful-application/gce-volume.yaml" %}
Notice that the `pdName: mysql-disk` line matches the name of the disk
in the Compute Engine environment. See the
[Persistent Volumes](/docs/user-guide/persistent-volumes/)
for details on writing a PersistentVolume configuration file for other
environments.
Create the persistent volume:
```
kubectl create -f http://k8s.io/docs/tutorials/stateful-application/gce-volume.yaml
```
### Deploy MySQL
You can run a stateful application by creating a Kubernetes Deployment
and connecting it to an existing PersistentVolume using a
PersistentVolumeClaim. For example, this YAML file describes a
Deployment that runs MySQL and references the PersistentVolumeClaim. The file
defines a volume mount for /var/lib/mysql, and then creates a
PersistentVolumeClaim that looks for a 20G volume. This claim is
satisfied by any volume that meets the requirements, in this case, the
volume created above.
Note: The password is defined in the config yaml, and this is insecure. See
[Kubernetes Secrets](/docs/user-guide/secrets/)
for a secure solution.
{% include code.html language="yaml" file="mysql-deployment.yaml" ghlink="/docs/tutorials/stateful-application/mysql-deployment.yaml" %}
1. Deploy the contents of the YAML file:
kubectl create -f http://k8s.io/docs/tutorials/stateful-application/mysql-deployment.yaml
1. Display information about the Deployment:
kubectl describe deployment mysql
Name: mysql
Namespace: default
CreationTimestamp: Tue, 01 Nov 2016 11:18:45 -0700
Labels: app=mysql
Selector: app=mysql
Replicas: 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: Recreate
MinReadySeconds: 0
OldReplicaSets: <none>
NewReplicaSet: mysql-63082529 (1/1 replicas created)
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
33s 33s 1 {deployment-controller } Normal ScalingReplicaSet Scaled up replica set mysql-63082529 to 1
1. List the pods created by the Deployment:
kubectl get pods -l app=mysql
NAME READY STATUS RESTARTS AGE
mysql-63082529-2z3ki 1/1 Running 0 3m
1. Inspect the Persistent Volume:
kubectl describe pv mysql-pv
Name: mysql-pv
Labels: <none>
Status: Bound
Claim: default/mysql-pv-claim
Reclaim Policy: Retain
Access Modes: RWO
Capacity: 20Gi
Message:
Source:
Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
PDName: mysql-disk
FSType: ext4
Partition: 0
ReadOnly: false
No events.
1. Inspect the PersistentVolumeClaim:
kubectl describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
Status: Bound
Volume: mysql-pv
Labels: <none>
Capacity: 20Gi
Access Modes: RWO
No events.
### Accessing the MySQL instance
The preceding YAML file creates a service that
allows other Pods in the cluster to access the database. The Service option
`clusterIP: None` lets the Service DNS name resolve directly to the
Pod's IP address. This is optimal when you have only one Pod
behind a Service and you don't intend to increase the number of Pods.
Run a MySQL client to connect to the server:
```
kubectl run -it --rm --image=mysql:5.6 mysql-client -- mysql -h mysql -ppassword
```
This command creates a new Pod in the cluster running a mysql client
and connects it to the server through the Service. If it connects, you
know your stateful MySQL database is up and running.
```
Waiting for pod default/mysql-client-274442439-zyp6i to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.
mysql>
```
### Updating
The image or any other part of the Deployment can be updated as usual
with the `kubectl apply` command. Here are some precautions that are
specific to stateful apps:
* Don't scale the app. This setup is for single-instance apps
only. The underlying PersistentVolume can only be mounted to one
Pod. For clustered stateful apps, see the
[StatefulSet documentation](/docs/user-guide/petset/).
* Use `strategy:` `type: Recreate` in the Deployment configuration
YAML file. This instructs Kubernetes to _not_ use rolling
updates. Rolling updates will not work, as you cannot have more than
one Pod running at a time. The `Recreate` strategy will stop the
first pod before creating a new one with the updated configuration.
### Deleting a deployment
Delete the deployed objects by name:
```
kubectl delete deployment,svc mysql
kubectl delete pvc mysql-pv-claim
kubectl delete pv mysql-pv
```
Also, if you are using Compute Engine disks:
```
gcloud compute disks delete mysql-disk
```
{% endcapture %}
{% capture whatsnext %}
* Learn more about [Deployment objects](/docs/user-guide/deployments/).
* Learn more about [Deploying applications](/docs/user-guide/deploying-applications/)
* [kubectl run documentation](/docs/user-guide/kubectl/kubectl_run/)
* [Volumes](/docs/user-guide/volumes/) and [Persistent Volumes](/docs/user-guide/persistent-volumes/)
{% endcapture %}
{% include templates/tutorial.md %}
docs/user-guide/accessing-the-cluster.md
+3 -2
View File
@@ -129,7 +129,7 @@ To use it,
* Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., `import "k8s.io/client-go/1.4/pkg/api/v1"` is correct.
The Go client can use the same [kubeconfig file](/docs/user-guide/kubeconfig-file)
as the kubectl CLI does to locate and authenticate to the apiserver. See this [example](https://github.com/kubernetes/client-go/examples/out-of-cluster.go):
as the kubectl CLI does to locate and authenticate to the apiserver. See this [example](https://github.com/kubernetes/client-go/blob/master/examples/out-of-cluster/main.go):
```golang
import (
@@ -183,7 +183,8 @@ From within a pod the recommended ways to connect to API are:
in any container of the pod can access it. See this [example of using kubectl proxy
in a pod](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/kubectl-container/).
- use the Go client library, and create a client using the `client.NewInCluster()` factory.
This handles locating and authenticating to the apiserver. [example](https://github.com/kubernetes/client-go/examples/in-cluster.go)
This handles locating and authenticating to the apiserver. See this [example of using Go client
library in a pod](https://github.com/kubernetes/client-go/blob/master/examples/in-cluster/main.go).
In each case, the credentials of the pod are used to communicate securely with the apiserver.
docs/user-guide/configuring-containers.md
+1 -1
View File
@@ -12,7 +12,7 @@ assignees:
In addition to the imperative-style commands, such as `kubectl run` and `kubectl expose`, described [elsewhere](/docs/user-guide/quick-start), Kubernetes supports declarative configuration. Oftentimes, configuration files are preferable to imperative commands, since they can be checked into version control and changes to the files can be code reviewed, which is especially important for more complex configurations, producing a more robust, reliable and archival system.
In the declarative style, all configuration is stored in YAML or JSON configuration files using Kubernetes's API resource schemas as the configuration schemas. `kubectl` can create, update, delete, and get API resources. The `apiVersion` (currently 'v1'?), resource `kind`, and resource `name` are used by `kubectl` to construct the appropriate API path to invoke for the specified operation.
In the declarative style, all configuration is stored in YAML or JSON configuration files using Kubernetes's API resource schemas as the configuration schemas. `kubectl` can create, update, delete, and get API resources. The `apiVersion` (currently `v1`?), resource `kind`, and resource `name` are used by `kubectl` to construct the appropriate API path to invoke for the specified operation.
## Launching a container using a configuration file
docs/user-guide/scheduled-jobs.md → docs/user-guide/cron-jobs.md
+54 -48
View File
@@ -9,16 +9,22 @@ assignees:
* TOC
{:toc}
## What is a _Scheduled Job_?
## What is a _Cron Job_?
A _Scheduled Job_ manages time based [Jobs](/docs/user-guide/jobs/), namely:
A _Cron Job_ manages time based [Jobs](/docs/user-guide/jobs/), namely:
* Once at a specified point in time
* Repeatedly at a specified point in time
One ScheduledJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically
One CronJob object is like one line of a _crontab_ (cron table) file. It runs a job periodically
on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format.
**Note:**: The question mark (`?`) in the schedule has the same meaning as an asterisk `*`,
that is, it stands for any of available value for a given field.
**Note:**: ScheduledJob resource was introduced in Kubernetes version 1.4, but starting
from version 1.5 its current name is CronJob.
A typical use case is:
* Schedule a job execution at a given point in time.
@@ -26,41 +32,41 @@ A typical use case is:
### Prerequisites
You need a working Kubernetes cluster at version >= 1.4, with batch/v2alpha1 API turned on by passing
`--runtime-config=batch/v2alpha1` while bringing up the API server (see [Turn on or off an API version
for your cluster](/docs/admin/cluster-management/#turn-on-or-off-an-api-version-for-your-cluster) for
more). You cannot use Scheduled Jobs on a hosted Kubernetes provider that has disabled alpha resources.
You need a working Kubernetes cluster at version >= 1.4 (for ScheduledJob), >= 1.5 (for CronJobs),
with batch/v2alpha1 API turned on by passing `--runtime-config=batch/v2alpha1` while bringing up
the API server (see [Turn on or off an API version for your cluster](/docs/admin/cluster-management/#turn-on-or-off-an-api-version-for-your-cluster)
for more). You cannot use Cron Jobs on a hosted Kubernetes provider that has disabled alpha resources.
## Creating a Scheduled Job
## Creating a Cron Job
Here is an example Scheduled Job. Every minute, it runs a simple job to print current time and then say
Here is an example Cron Job. Every minute, it runs a simple job to print current time and then say
hello.
{% include code.html language="yaml" file="sj.yaml" ghlink="/docs/user-guide/sj.yaml" %}
{% include code.html language="yaml" file="cronjob.yaml" ghlink="/docs/user-guide/cronjob.yaml" %}
Run the example scheduled job by downloading the example file and then running this command:
Run the example cron job by downloading the example file and then running this command:
```shell
$ kubectl create -f ./sj.yaml
scheduledjob "hello" created
$ kubectl create -f ./cronjob.yaml
cronjob "hello" created
```
Alternatively, use `kubectl run` to create a scheduled job without writing full config:
Alternatively, use `kubectl run` to create a cron job without writing full config:
```shell
$ kubectl run hello --schedule="0/1 * * * ?" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster"
scheduledjob "hello" created
$ kubectl run hello --schedule="*/1 * * * *" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster"
cronjob "hello" created
```
After creating the scheduled job, get its status using this command:
After creating the cron job, get its status using this command:
```shell
$ kubectl get scheduledjob hello
$ kubectl get cronjob hello
NAME SCHEDULE SUSPEND ACTIVE LAST-SCHEDULE
hello 0/1 * * * ? False 0 <none>
hello */1 * * * * False 0 <none>
```
As you can see above, there's no active job yet, and no job has been scheduled, either.
As you can see above, there's no active job yet, and no job has been scheduled, either.
Watch for the job to be created in around one minute:
@@ -70,16 +76,16 @@ NAME DESIRED SUCCESSFUL AGE
hello-4111706356 1 1 2s
```
Now you've seen one running job scheduled by "hello". We can stop watching it and get the scheduled job again:
Now you've seen one running job scheduled by "hello". We can stop watching it and get the cron job again:
```shell
$ kubectl get scheduledjob hello
$ kubectl get cronjob hello
NAME SCHEDULE SUSPEND ACTIVE LAST-SCHEDULE
hello 0/1 * * * ? False 0 Mon, 29 Aug 2016 14:34:00 -0700
hello */1 * * * * False 0 Mon, 29 Aug 2016 14:34:00 -0700
```
You should see that "hello" successfully scheduled a job at the time specified in `LAST-SCHEDULE`. There are
currently 0 active jobs, meaning that the job that's scheduled is completed or failed.
currently 0 active jobs, meaning that the job that's scheduled is completed or failed.
Now, find the pods created by the job last scheduled and view the standard output of one of the pods. Note that
your job name and pod name would be different.
@@ -96,17 +102,17 @@ Mon Aug 29 21:34:09 UTC 2016
Hello from the Kubernetes cluster
```
## Deleting a Scheduled Job
## Deleting a Cron Job
Once you don't need a scheduled job anymore, simply delete it with `kubectl`:
Once you don't need a cron job anymore, simply delete it with `kubectl`:
```shell
$ kubectl delete scheduledjob hello
scheduledjob "hello" deleted
$ kubectl delete cronjob hello
cronjob "hello" deleted
```
This stops new jobs from being created. However, running jobs won't be stopped, and no jobs or their pods will
be deleted. To clean up those jobs and pods, you need to list all jobs created by the scheduled job, and delete them all:
be deleted. To clean up those jobs and pods, you need to list all jobs created by the cron job, and delete them all:
```shell
$ kubectl get jobs
@@ -121,34 +127,34 @@ job "hello-1202039034" deleted
...
```
Once the jobs are deleted, the pods created by them are deleted as well. Note that all jobs created by scheduled
job "hello" will be prefixed "hello-". You can delete them at once with `kubectl delete jobs --all`, if you want to
Once the jobs are deleted, the pods created by them are deleted as well. Note that all jobs created by cron
job "hello" will be prefixed "hello-". You can delete them at once with `kubectl delete jobs --all`, if you want to
delete all jobs in the current namespace (not just the ones created by "hello".)
## Scheduled Job Limitations
## Cron Job Limitations
A scheduled job creates a job object _about_ once per execution time of its schedule. We say "about" because there
A cron job creates a job object _about_ once per execution time of its schedule. We say "about" because there
are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare,
but do not completely prevent them. Therefore, jobs should be _idempotent_.
The job is responsible for retrying pods, parallelism among pods it creates, and determining the success or failure
of the set of pods. A scheduled job does not examine pods at all.
of the set of pods. A cron job does not examine pods at all.
## Writing a Scheduled Job Spec
## Writing a Cron Job Spec
As with all other Kubernetes configs, a scheduled job needs `apiVersion`, `kind`, and `metadata` fields. For general
information about working with config files, see [deploying applications](/docs/user-guide/deploying-applications),
As with all other Kubernetes configs, a cron job needs `apiVersion`, `kind`, and `metadata` fields. For general
information about working with config files, see [deploying applications](/docs/user-guide/deploying-applications),
[configuring containers](/docs/user-guide/configuring-containers), and
[using kubectl to manage resources](/docs/user-guide/working-with-resources) documents.
A scheduled job also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status).
A cron job also needs a [`.spec` section](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status).
**Note:** All modifications to a scheduled job, especially its `.spec`, will be applied only to the next run.
**Note:** All modifications to a cron job, especially its `.spec`, will be applied only to the next run.
### Schedule
### Schedule
The `.spec.schedule` is a required field of the `.spec`. It takes a [Cron](https://en.wikipedia.org/wiki/Cron) format
string, e.g. `0 * * * *` or `@hourly`, as schedule time of its jobs to be created and executed.
string, e.g. `0 * * * *` or `@hourly`, as schedule time of its jobs to be created and executed.
### Job Template
@@ -159,22 +165,22 @@ as a [Job](/docs/user-guide/jobs), except it is nested and does not have an `api
### Starting Deadline Seconds
The `.spec.startingDeadlineSeconds` field is optional. It stands for the deadline (in seconds) for starting the job
if it misses its scheduled time for any reason. Missed jobs executions will be counted as failed ones. If not specified,
there's no deadline.
if it misses its scheduled time for any reason. Missed jobs executions will be counted as failed ones. If not specified,
there's no deadline.
### Concurrency Policy
The `.spec.concurrencyPolicy` field is also optional. It specifies how to treat concurrent executions of a job
created by this scheduled job. Only one of the following concurrent policies may be specified:
created by this cron job. Only one of the following concurrent policies may be specified:
* `Allow` (default): allows concurrently running jobs
* `Forbid`: forbids concurrent runs, skipping next run if previous hasn't finished yet
* `Replace`: cancels currently running job and replaces it with a new one
Note that concurrency policy only applies to the jobs created by the same scheduled job. If there are multiple
scheduled jobs, their respective jobs are always allowed to run concurrently.
Note that concurrency policy only applies to the jobs created by the same cron job. If there are multiple
cron jobs, their respective jobs are always allowed to run concurrently.
### Suspend
The `.spec.suspend` field is also optional. If set to `true`, all subsequent executions will be suspended. It does not
apply to already started executions. Defaults to false.
The `.spec.suspend` field is also optional. If set to `true`, all subsequent executions will be suspended. It does not
apply to already started executions. Defaults to false.
docs/user-guide/sj.yaml → docs/user-guide/cronjob.yaml
+2 -2
View File
@@ -1,9 +1,9 @@
apiVersion: batch/v2alpha1
kind: ScheduledJob
kind: CronJob
metadata:
name: hello
spec:
schedule: 0/1 * * * ?
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
docs/user-guide/federation/federated-ingress.md
+62 -12
View File
@@ -64,12 +64,12 @@ healthy backend service endpoint at all times, even in the event of
pod, cluster,
availability zone or regional outages.
Note that in the
case of Google Cloud, the logical L7 load balancer is not a single physical device (which
would present both a single point of failure, and a single global
network routing choke point), but rather a [truly global, highly available
load balancing managed service](https://cloud.google.com/load-balancing/),
globally reachable via a single, static IP address.
Note that in the case of Google Cloud, the logical L7 load balancer is
not a single physical device (which would present both a single point
of failure, and a single global network routing choke point), but
rather a
[truly global, highly available load balancing managed service](https://cloud.google.com/load-balancing/),
globally reachable via a single, static IP address.
Clients inside your federated Kubernetes clusters (i.e. Pods) will be
automatically routed to the cluster-local shard of the Federated Service
@@ -86,13 +86,13 @@ You can create a federated ingress in any of the usual ways, for example using k
``` shell
kubectl --context=federation-cluster create -f myingress.yaml
```
For example ingress YAML configurations, see the [Ingress User Guide](/docs/user-guide/ingress/)
The '--context=federation-cluster' flag tells kubectl to submit the
request to the Federation API endpoint, with the appropriate
credentials. If you have not yet configured such a context, visit the
[federation admin guide](/docs/admin/federation/) or one of the
[administration tutorials](https://github.com/kelseyhightower/kubernetes-cluster-federation)
to find out how to do so. TODO: Update links
to find out how to do so.
As described above, the Federated Ingress will automatically create
and maintain matching Kubernetes ingresses in all of the clusters
@@ -147,17 +147,28 @@ Events:
2m 2m 1 {loadbalancer-controller } Normal CREATE ip: 130.211.5.194
```
Note the address of your Federated Ingress
Note that:
1. the address of your Federated Ingress
corresponds with the address of all of the
underlying Kubernetes ingresses (once these have been allocated - this
may take up to a few minutes).
Note also that we have not yet provisioned any backend Pods to receive
2. we have not yet provisioned any backend Pods to receive
the network traffic directed to this ingress (i.e. 'Service
Endpoints' behind the service backing the Ingress), so the Federated Ingress does not yet consider these to
be healthy shards and will not direct traffic to any of these clusters.
3. the federation control system will
automatically reconfigure the load balancer controllers in all of the
clusters in your federation to make them consistent, and allow
them to share global load balancers. But this reconfiguration can
only complete successfully if there are no pre-existing Ingresses in
those clusters (this is a safety feature to prevent accidental
breakage of existing ingresses). So to ensure that your federated
ingresses function correctly, either start with new, empty clusters, or make
sure that you delete (and recreate if necessary) all pre-existing
Ingresses in the clusters comprising your federation.
## Adding backend services and pods
#Adding backend services and pods
To render the underlying ingress shards healthy, we need to add
backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but
@@ -175,6 +186,16 @@ kubectl --context=federation-cluster create -f services/nginx.yaml
kubectl --context=federation-cluster create -f myreplicaset.yaml
```
Note that in order for your federated ingress to work correctly on
Google Cloud, the node ports of all of the underlying cluster-local
services need to be identical. If you're using a federated service
this is easy to do. Simply pick a node port that is not already
being used in any of your clusters, and add that to the spec of your
federated service. If you do not specify a node port for your
federated service, each cluster will choose it's own node port for
its cluster-local shard of the service, and these will probably end
up being different, which is not what you want.
You can verify this by checking in each of the underlying clusters, for example:
``` shell
@@ -258,6 +279,35 @@ Check that:
`service-controller` or `replicaset-controller`,
errors in the output of `kubectl logs federation-controller-manager --namespace federation`).
#### I can create a federated ingress successfully, but request load is not correctly distributed across the underlying clusters
Check that:
1. the services underlying your federated ingress in each cluster have
identical node ports. See [above](#creating_a_federated_ingress) for further explanation.
2. the load balancer controllers in each of your clusters are of the
correct type ("GLBC") and have been correctly reconfigured by the
federation control plane to share a global GCE load balancer (this
should happen automatically). If they of the correct type, and
have been correctly reconfigured, the UID data item in the GLBC
configmap in each cluster will be identical across all clusters.
See
[the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid)
for further details.
If this is not the case, check the logs of your federation
controller manager to determine why this automated reconfiguration
might be failing.
3. no ingresses have been manually created in any of your clusters before the above
reconfiguration of the load balancer controller completed
successfully. Ingresses created before the reconfiguration of
your GLBC will interfere with the behavior of your federated
ingresses created after the reconfiguration (see
[the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid)
for further information. To remedy this,
delete any ingresses created before the cluster joined the
federation (and had it's GLBC reconfigured), and recreate them if
necessary.
#### This troubleshooting guide did not help me solve my problem
Please use one of our [support channels](http://kubernetes.io/docs/troubleshooting/) to seek assistance.
docs/user-guide/images.md
+1 -1
View File
@@ -5,7 +5,7 @@ assignees:
---
Each container in a pod has its own image. Currently, the only type of image supported is a [Docker Image](https://docs.docker.com/userguide/dockerimages/).
Each container in a pod has its own image. Currently, the only type of image supported is a [Docker Image](https://docs.docker.com/engine/tutorials/dockerimages/).
You create your Docker image and push it to a registry before referring to it in a Kubernetes pod.
docs/user-guide/ingress.md
+12 -12
View File
@@ -53,18 +53,18 @@ Make sure you review the [beta limitations](https://github.com/kubernetes/contri
A minimal Ingress might look like:
```yaml
01. apiVersion: extensions/v1beta1
02. kind: Ingress
03. metadata:
04. name: test-ingress
05. spec:
06. rules:
07. - http:
08. paths:
09. - path: /testpath
10. backend:
11. serviceName: test
12. servicePort: 80
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
spec:
rules:
- http:
paths:
- path: /testpath
backend:
serviceName: test
servicePort: 80
```
*POSTing this to the API server will have no effect if you have not configured an [Ingress controller](#ingress-controllers).*
docs/user-guide/jobs.md
+2 -2
View File
@@ -380,6 +380,6 @@ driver, and then cleans up.
An advantage of this approach is that the overall process gets the completion guarantee of a Job
object, but complete control over what pods are created and how work is assigned to them.
## Scheduled Jobs
## Cron Jobs
Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [scheduled job documents](http://kubernetes.io/docs/user-guide/scheduled-jobs/)
Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [cron job documents](http://kubernetes.io/docs/user-guide/cron-jobs/)
docs/user-guide/kubectl-cheatsheet.md
+2 -2
View File
@@ -141,7 +141,7 @@ $ kubectl rolling-update frontend-v1 frontend-v2 --rollback # Abort exist
$ cat pod.json | kubectl replace -f - # Replace a pod based on the JSON passed into stdin
# Force replace, delete and then re-create the resource. Will cause a service outage.
$ kubectl replace --force -f ./pod.json
$ kubectl replace --force -f ./pod.json
# Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000
$ kubectl expose rc nginx --port=80 --target-port=8000
@@ -251,7 +251,7 @@ Resource type | Abbreviated alias
`replicasets` |`rs`
`replicationcontrollers` |`rc`
`resourcequotas` |`quota`
`scheduledjob` |
`cronjob` |
`secrets` |
`serviceaccount` |`sa`
`services` |`svc`
docs/user-guide/kubectl-conventions.md
+2 -2
View File
@@ -40,7 +40,7 @@ In order for `kubectl run` to satisfy infrastructure as code:
version 1.5 of kuberentes this generator is deprecated, with a plan to be
removed in 1.6. Please use `job/v1` instead.
* Job - use `job/v1`.
* ScheduledJob - use `scheduledjob/v2alpha1`.
* CronJob - use `cronjob/v2alpha1`.
Additionally, if you didn't specify a generator flag, other flags will suggest using
a specific generator. Below table shows which flags force using specific generators,
@@ -52,7 +52,7 @@ depending on your cluster version:
| Replication Controller | `--generator=run/v1` | `--generator=run/v1` | `--generator=run/v1` | `--restart=Always` |
| Deployment | `--restart=Always` | `--restart=Always` | `--restart=Always` | N/A |
| Job | `--restart=OnFailure` | `--restart=OnFailure` | `--restart=OnFailure` OR `--restart=Never` | N/A |
| Scheduled Job | `--schedule=<cron>` | N/A | N/A | N/A |
| Cron Job | `--schedule=<cron>` | N/A | N/A | N/A |
Note that these flags will use a default generator only when you have not specified
any flag. This also means that combining `--generator` with other flags won't
docs/user-guide/kubectl/kubectl_run.md
+1 -1
View File
@@ -50,7 +50,7 @@ kubectl run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>
# Start the perl container to compute π to 2000 places and print it out.
kubectl run pi --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
# Start the scheduled job to compute π to 2000 places and print it out every 5 minutes.
# Start the cron job to compute π to 2000 places and print it out every 5 minutes.
kubectl run pi --schedule="0/5 * * * ?" --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
```
docs/user-guide/node-selection/index.md
+1 -1
View File
@@ -173,7 +173,7 @@ on node N if node N has a label with key `failure-domain.beta.kubernetes.io/zone
such that there is at least one node in the cluster with key `failure-domain.beta.kubernetes.io/zone` and
value V that is running a pod that has a label with key "security" and value "S1".) The pod anti-affinity
rule says that the pod cannot schedule onto a node if that node is already running a pod with label
having key "security" and value "S2". (If the `topologyKey` were `failure-domain.beta.kuberntes.io/zone` then
having key "security" and value "S2". (If the `topologyKey` were `failure-domain.beta.kubernetes.io/zone` then
it would mean that the pod cannot schedule onto a node if that node is in the same zone as a pod with
label having key "security" and value "S2".) See the [design doc](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/design/podaffinity.md).
for many more examples of pod affinity and anti-affinity, both the `requiredDuringSchedulingIgnoredDuringExecution`
docs/user-guide/persistent-volumes/volumes/local-01.yaml
+1 -1
View File
@@ -10,4 +10,4 @@ spec:
accessModes:
- ReadWriteOnce
hostPath:
path: "/somepath/data01"
path: "/tmp/data01"
docs/user-guide/persistent-volumes/walkthrough.md
+2 -2
View File
@@ -27,7 +27,7 @@ for ease of development and testing. You'll create a local `HostPath` for this
support local storage on the host at this time. There is no guarantee your pod ends up on the correct node where the `HostPath` resides.
```shell
# This will be nginx's webroot
# This will be nginx's webroot; execute this on the node where your pod will run.
$ mkdir /tmp/data01
$ echo 'I love Kubernetes storage!' > /tmp/data01/index.html
```
@@ -125,4 +125,4 @@ I love Kubernetes storage!
Hopefully this simple guide is enough to get you started with PersistentVolumes. If you have any questions, join the team on [Slack](/docs/troubleshooting/#slack) and ask!
Enjoy!
Enjoy!
docs/user-guide/petset/bootstrapping/index.md
+1 -1
View File
@@ -88,7 +88,7 @@ vm-1 # printf "GET / HTTP/1.0\r\n\r\n" | netcat vm-0.ub 80
It's worth exploring what just happened. Init containers run sequentially *before* the application container. In this example we used the init container to copy shared libraries from the rootfs, while preserving user installed packages across container restart.
```yaml
pod.alpha.kubernetes.io/init-containers: '[
pod.beta.kubernetes.io/init-containers: '[
{
"name": "rootfs",
"image": "ubuntu:15.10",
docs/user-guide/petset/bootstrapping/petset_peers.yaml
+1 -1
View File
@@ -29,7 +29,7 @@ spec:
app: nginx
annotations:
pod.alpha.kubernetes.io/initialized: "true"
pod.alpha.kubernetes.io/init-containers: '[
pod.beta.kubernetes.io/init-containers: '[
{
"name": "peerfinder",
"image": "gcr.io/google_containers/peer-finder:0.1",
docs/user-guide/petset/bootstrapping/petset_vm.yaml
+1 -1
View File
@@ -27,7 +27,7 @@ spec:
app: ub
annotations:
pod.alpha.kubernetes.io/initialized: "true"
pod.alpha.kubernetes.io/init-containers: '[
pod.beta.kubernetes.io/init-containers: '[
{
"name": "rootfs",
"image": "ubuntu:15.10",
docs/user-guide/pod-states.md
+2 -2
View File
@@ -66,8 +66,8 @@ The possible values for RestartPolicy are `Always`, `OnFailure`, or `Never`. If
Three types of controllers are currently available:
- Use a [`Job`](/docs/user-guide/jobs/) for pods which are expected to terminate (e.g. batch computations).
- Use a [`ReplicationController`](/docs/user-guide/replication-controller/) for pods which are not expected to
terminate (e.g. web servers).
- Use a [`ReplicationController`](/docs/user-guide/replication-controller/) or [`Deployment`](/docs/user-guide/deployments/)
for pods which are not expected to terminate (e.g. web servers).
- Use a [`DaemonSet`](/docs/admin/daemons/): Use for pods which need to run 1 per machine because they provide a
machine-specific system service.
If you are unsure whether to use ReplicationController or Daemon, then see [Daemon Set versus
docs/user-guide/pods/init-container.md
+169
View File
@@ -0,0 +1,169 @@
---
assignees:
- erictune
---
* TOC
{:toc}
In addition to having one or more main containers (or **app containers**), a
pod can also have one or more **init containers** which run before the app
containers. Init containers allow you to reduce and reorganize setup scripts
and "glue code".
## Overview
An init container is exactly like a regular container, except that it always
runs to completion and each init container must complete successfully before
the next one is started. If the init container fails, Kubernetes will restart
the pod until the init container succeeds. If a pod is marked as `RestartNever`,
the pod will fail if the init container fails.
You specify a container as an init container by adding an annotation
The annotation key is `pod.beta.kubernetes.io/init-containers`. The annotation
value is a JSON array of [objects of type `v1.Container`
](http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_container)
Once the feature exits beta, the init containers will be specified on the Pod
Spec alongside the app `containers` array.
The status of the init containers is returned as another annotation -
`pod.beta.kubernetes.io/init-container-statuses` -- as an array of the
container statuses (similar to the `status.containerStatuses` field).
Init containers support all of the same features as normal containers,
including resource limits, volumes, and security settings. The resource
requests and limits for an init container are [handled slightly differently](
#resources). Init containers do not support readiness probes since they will
run to completion before the pod can be ready.
An init container has all of the fields of an app container.
If you specify multiple init containers for a pod, those containers run one at
a time in sequential order. Each must succeed before the next can run. Once all
init containers have run to completion, Kubernetes initializes the pod and runs
the application containers as usual.
## What are Init Containers Good For?
Because init containers have separate images from application containers, they
have some advantages for start-up related code. These include:
* they can contain utilities that are not desirable to include in the app container
image for security reasons,
* they can contain utilities or custom code for setup that is not present in an app
image. (No need to make an image `FROM` another image just to use a tool like
`sed`, `awk`, `python`, `dig`, etc during setup).
* the application image builder and the deployer roles can work independently without
the need to jointly build a single app image.
Because init containers have different filesystem view (Linux namespaces) from
app containers, they can be given access to Secrets that the app containers are
not able to access.
Since init containers run to completion before any app containers start, and
since app containers run in parallel, they provide an easier way to block or
delay the startup of application containers until some precondition is met.
Because init containers run in sequence and there can be multiple init containers,
they can be composed easily.
Here are some ideas for how to use init containers:
- Wait for a service to be created with a shell command like:
`for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; exit 1`
- Register this pod with a remote server with a command like:
`curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(POD_NAME)&ip=$(POD_IP)'`
using `POD_NAME` and `POD_IP` from the downward API.
- Wait for some time before starting the app container with a command like `sleep 60`.
- Clone a git repository into a volume
- Place values like a POD_IP into a configuration file, and run a template tool (e.g. jinja)
to generate a configuration file to be consumed by the main app contianer.
```
Complete usage examples can be found in the [PetSets
guide](docs/user-guide/petset/bootstrapping/index.md) and the [Production Pods
guide](/docs/user-guide/production-pods.md#handling-initialization).
## Detailed Behavior
Each pod may have 0..N init containers defined along with the existing
1..M app containers.
On startup of the pod, after the network and volumes are initialized, the init
containers are started in order. Each container must exit successfully before
the next is invoked. If a container fails to start (due to the runtime) or
exits with failure, it is retried according to the pod RestartPolicy, except
when the pod restart policy is RestartPolicyAlways, in which case just the init
containers use RestartPolicyOnFailure.
A pod cannot be ready until all init containers have succeeded. The ports on an
init container are not aggregated under a service. A pod that is being
initialized is in the `Pending` phase but should has a condition `Initializing`
set to `true`.
If the pod is [restarted](#pod-restart-reasons) all init containers must
execute again.
Changes to the init container spec are limited to the container image field.
Altering a init container image field is equivalent to restarting the pod.
Because init containers can be restarted, retried, or reexecuted, init container
code should be idempotent. In particular, code that writes to files on EmptyDirs
should be prepared for the possibility that an output file already exists.
An init container has all of the fields of an app container. The following
fields are prohibited from being used on init containers by validation:
* `readinessProbe` - init containers must exit for pod startup to continue,
are not included in rotation, and so cannot define readiness distinct from
completion.
Init container authors may use `activeDeadlineSeconds` on the pod and
`livenessProbe` on the container to prevent init containers from failing
forever. The active deadline includes init containers.
The name of each app and init container in a pod must be unique - it is a
validation error for any container to share a name.
### Resources
Given the ordering and execution for init containers, the following rules
for resource usage apply:
* The highest of any particular resource request or limit defined on all init
containers is the **effective init request/limit**
* The pod's **effective request/limit** for a resource is the higher of:
* sum of all app containers request/limit for a resource
* effective init request/limit for a resource
* Scheduling is done based on effective requests/limits, which means
init containers can reserve resources for initialization that are not used
during the life of the pod.
* QoS tier of the pod's **effective QoS tier** is the QoS tier for init containers
and app containers alike.
Quota and limits are applied based on the effective pod request and
limit.
Pod level cGroups are based on the effective pod request and limit, the
same as the scheduler.
## Pod Restart Reasons
A Pod may "restart", causing reexecution of init containers, for the following
reasons:
* An init container image is changed by a user updating the Pod Spec.
* App container image changes only restart the app container.
* The pod infrastructure container is restarted
* This is uncommon and would have to be done by someone with root access to nodes.
* All containers in a pod are terminated, requiring a restart (RestartPolicyAlways) AND the record of init container completion has been lost due to garbage collection.
## Support and compatibilty
A cluster with Kubelet and Apiserver version 1.4.0 or greater supports init
containers with the beta annotations. Support varies for other combinations of
Kubelet and Apiserver version; see the [release notes
](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md) for details.
docs/user-guide/production-pods.md
+3 -1
View File
@@ -204,6 +204,8 @@ The status of the init containers is returned as another annotation - `pod.beta.
Init containers support all of the same features as normal containers, including resource limits, volumes, and security settings. The resource requests and limits for an init container are handled slightly different than normal containers since init containers are run one at a time instead of all at once - any limits or quotas will be applied based on the largest init container resource quantity, rather than as the sum of quantities. Init containers do not support readiness probes since they will run to completion before the pod can be ready.
[Complete Init Container Documentation](/docs/user-guide/pods/init-containers.md)
## Lifecycle hooks and termination notice
@@ -218,7 +220,7 @@ The specification of a pre-stop hook is similar to that of probes, but without t
## Termination message
In order to achieve a reasonably high level of availability, especially for actively developed applications, it's important to debug failures quickly. Kubernetes can speed debugging by surfacing causes of fatal errors in a way that can be display using [`kubectl`](/docs/user-guide/kubectl/kubectl) or the [UI](/docs/user-guide/ui), in addition to general [log collection](/docs/user-guide/logging). It is possible to specify a `terminationMessagePath` where a container will write its 'death rattle'?, such as assertion failure messages, stack traces, exceptions, and so on. The default path is `/dev/termination-log`.
In order to achieve a reasonably high level of availability, especially for actively developed applications, it's important to debug failures quickly. Kubernetes can speed debugging by surfacing causes of fatal errors in a way that can be display using [`kubectl`](/docs/user-guide/kubectl/) or the [UI](/docs/user-guide/ui), in addition to general [log collection](/docs/user-guide/logging). It is possible to specify a `terminationMessagePath` where a container will write its 'death rattle'?, such as assertion failure messages, stack traces, exceptions, and so on. The default path is `/dev/termination-log`.
Here is a toy example:
docs/user-guide/secrets/index.md
+1 -1
View File
@@ -265,7 +265,7 @@ All listed keys must exist in the corresponding secret. Otherwise, the volume is
**Secret files permissions**
You can also specify the permission mode bits files part of a secret will have.
If you don't specify any, `0644` is used by default. You can sepecify a default
If you don't specify any, `0644` is used by default. You can specify a default
mode for the whole secret volume and override per key if needed.
For example, you can specify a default mode like this:
docs/user-guide/service-accounts.md
+2 -2
View File
@@ -31,8 +31,8 @@ you get the raw json or yaml for a pod you have created (e.g. `kubectl get
pods/podname -o yaml`), you can see the `spec.serviceAccount` field has been
[automatically set](/docs/user-guide/working-with-resources/#resources-are-automatically-modified).
You can access the API using a proxy or with a client library, as described in
[Accessing the Cluster](/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod).
With service accounts, you can access the API inside the pod using a proxy or with a client library,
as described in [Accessing the Cluster](/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod).
## Using Multiple Service Accounts.
docs/user-guide/services/index.md
+2 -2
View File
@@ -182,7 +182,7 @@ In Kubernetes v1.0 the proxy was purely in userspace. In Kubernetes v1.1 an
iptables proxy was added, but was not the default operating mode. Since
Kubernetes v1.2, the iptables proxy is the default.
As of Kubernetes v1.0, `Services` are a "layer 3" (TCP/UDP over IP) construct.
As of Kubernetes v1.0, `Services` are a "layer 4" (TCP/UDP over IP) construct.
In Kubernetes v1.1 the `Ingress` API was added (beta) to represent "layer 7"
(HTTP) services.
@@ -345,7 +345,7 @@ can do a DNS SRV query for `"_http._tcp.my-service.my-ns"` to discover the port
number for `"http"`.
The Kubernetes DNS server is the only way to access services of type
`ExternalName`.
`ExternalName`. More information is available in the [DNS Admin Guide](http://kubernetes.io/docs/admin/dns/).
## Headless services
docs/user-guide/sharing-clusters.md
+2 -2
View File
@@ -115,12 +115,12 @@ make all four clusters available on both hosts by running
# on host2, copy host1's default kubeconfig, and merge it from env
$ scp host1:/path/to/home1/.kube/config /path/to/other/.kube/config
$ export $KUBECONFIG=/path/to/other/.kube/config
$ export KUBECONFIG=/path/to/other/.kube/config
# on host1, copy host2's default kubeconfig and merge it from env
$ scp host2:/path/to/home2/.kube/config /path/to/other/.kube/config
$ export $KUBECONFIG=/path/to/other/.kube/config
$ export KUBECONFIG=/path/to/other/.kube/config
```
Detailed examples and explanation of `kubeconfig` loading/merging rules can be found in [kubeconfig-file](/docs/user-guide/kubeconfig-file).
docs/user-guide/thirdpartyresources.md
+1 -1
View File
@@ -22,7 +22,7 @@ Each `ThirdPartyResource` has the following:
* `description` - A free text description of the resource.
* `versions` - A list of the versions of the resource.
The `kind` for a `ThirdPartyResource` takes the form `<kind name>.<domain>`. You are expected to provide a unique kind and domain name in order to avoid conflicts with other `ThirdPartyResource` objects. Kind names will be converted to CamelCase when creating instances of the `ThirdPartyResource`. Hypens in the `kind` are assumed to be word breaks. For instance the kind `camel-case` would be converted to `CamelCase` but `camelcase` would be converted to `Camelcase`.
The `kind` for a `ThirdPartyResource` takes the form `<kind name>.<domain>`. You are expected to provide a unique kind and domain name in order to avoid conflicts with other `ThirdPartyResource` objects. Kind names will be converted to CamelCase when creating instances of the `ThirdPartyResource`. Hyphens in the `kind` are assumed to be word breaks. For instance the kind `camel-case` would be converted to `CamelCase` but `camelcase` would be converted to `Camelcase`.
Other fields on the `ThirdPartyResource` are treated as custom data fields. These fields can hold arbitrary JSON data and have any structure.
docs/user-guide/working-with-resources.md
+1 -1
View File
@@ -46,7 +46,7 @@ The system adds fields in several ways:
- Some fields are added synchronously with creation of the resource and some are set asynchronously.
- For example: `metadata.uid` is set synchronously. (Read more about [metadata](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#metadata)).
- For example, `status.hostIP` is set only after the pod has been scheduled. This often happens fast, but you may notice pods which do not have this set yet. This is called Late Initialization. (Read mode about [status](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) and [late initialization](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#late-initialization) ).
- For example, `status.hostIP` is set only after the pod has been scheduled. This often happens fast, but you may notice pods which do not have this set yet. This is called Late Initialization. (Read more about [status](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#spec-and-status) and [late initialization](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#late-initialization)).
- Some fields are set to default values. Some defaults vary by cluster and some are fixed for the API at a certain version. (Read more about [defaulting](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/docs/devel/api-conventions.md#defaulting)).
- For example, `spec.containers[0].imagePullPolicy` always defaults to `IfNotPresent` in api v1.
- For example, `spec.containers[0].resources.limits.cpu` may be defaulted to `100m` on some clusters, to some other value on others, and not defaulted at all on others.