Convert site to Hugo (#8316)
This commit converts content and layout to use Hugo.
This commit is contained in:
Bjørn Erik Pedersen
committed by
k8s-ci-robot
k8s-ci-robot
parent
7745f0e0c5
commit
7f3b633aa0
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: "Compute, Storage, and Networking Extensions"
|
||||
weight: 30
|
||||
---
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: Installing Addons
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Add-ons extend the functionality of Kubernetes.
|
||||
|
||||
This page lists some of the available add-ons and links to their respective installation instructions.
|
||||
|
||||
Add-ons in each section are sorted alphabetically - the ordering does not imply any preferential status.
|
||||
|
||||
## Networking and Network Policy
|
||||
|
||||
|
||||
* [ACI](https://www.github.com/noironetworks/aci-containers) provides integrated container networking and network security with Cisco ACI.
|
||||
* [Calico](https://docs.projectcalico.org/latest/getting-started/kubernetes/) is a secure L3 networking and network policy provider.
|
||||
* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) unites Flannel and Calico, providing networking and network policy.
|
||||
* [Cilium](https://github.com/cilium/cilium) is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported.
|
||||
* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave.
|
||||
* [Contiv](http://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](http://github.com/contiv). The [installer](http://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options.
|
||||
* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md) is an overlay network provider that can be used with Kubernetes.
|
||||
* [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
|
||||
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and Openshift.
|
||||
* [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring.
|
||||
* [Romana](http://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize).
|
||||
* [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.
|
||||
|
||||
## Service Discovery
|
||||
|
||||
* [CoreDNS](https://coredns.io) is a flexible, extensible DNS server which can be [installed](https://github.com/coredns/deployment/tree/master/kubernetes) as the in-cluster DNS for pods.
|
||||
|
||||
## Visualization & Control
|
||||
|
||||
* [Dashboard](https://github.com/kubernetes/dashboard#kubernetes-dashboard) is a dashboard web interface for Kubernetes.
|
||||
* [Weave Scope](https://www.weave.works/documentation/scope-latest-installing/#k8s) is a tool for graphically visualizing your containers, pods, services etc. Use it in conjunction with a [Weave Cloud account](https://cloud.weave.works/) or host the UI yourself.
|
||||
|
||||
## Legacy Add-ons
|
||||
|
||||
There are several other add-ons documented in the deprecated [cluster/addons](https://git.k8s.io/kubernetes/cluster/addons) directory.
|
||||
|
||||
Well-maintained ones should be linked to here. PRs welcome!
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
title: Certificates
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Creating Certificates
|
||||
|
||||
When using client certificate authentication, you can generate certificates
|
||||
manually through `easyrsa`, `openssl` or `cfssl`.
|
||||
|
||||
### easyrsa
|
||||
|
||||
**easyrsa** can manually generate certificates for your cluster.
|
||||
|
||||
1. Download, unpack, and initialize the patched version of easyrsa3.
|
||||
|
||||
curl -LO https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz
|
||||
tar xzf easy-rsa.tar.gz
|
||||
cd easy-rsa-master/easyrsa3
|
||||
./easyrsa init-pki
|
||||
1. Generate a CA. (`--batch` set automatic mode. `--req-cn` default CN to use.)
|
||||
|
||||
./easyrsa --batch "--req-cn=${MASTER_IP}@`date +%s`" build-ca nopass
|
||||
1. Generate server certificate and key.
|
||||
The argument `--subject-alt-name` sets the possible IPs and DNS names the API server will
|
||||
be accessed with. The `MASTER_CLUSTER_IP` is usually the first IP from the service CIDR
|
||||
that is specified as the `--service-cluster-ip-range` argument for both the API server and
|
||||
the controller manager component. The argument `--days` is used to set the number of days
|
||||
after which the certificate expires.
|
||||
The sample below also assume that you are using `cluster.local` as the default
|
||||
DNS domain name.
|
||||
|
||||
./easyrsa --subject-alt-name="IP:${MASTER_IP},"\
|
||||
"IP:${MASTER_CLUSTER_IP},"\
|
||||
"DNS:kubernetes,"\
|
||||
"DNS:kubernetes.default,"\
|
||||
"DNS:kubernetes.default.svc,"\
|
||||
"DNS:kubernetes.default.svc.cluster,"\
|
||||
"DNS:kubernetes.default.svc.cluster.local" \
|
||||
--days=10000 \
|
||||
build-server-full server nopass
|
||||
1. Copy `pki/ca.crt`, `pki/issued/server.crt`, and `pki/private/server.key` to your directory.
|
||||
1. Fill in and add the following parameters into the API server start parameters:
|
||||
|
||||
--client-ca-file=/yourdirectory/ca.crt
|
||||
--tls-cert-file=/yourdirectory/server.crt
|
||||
--tls-private-key-file=/yourdirectory/server.key
|
||||
|
||||
### openssl
|
||||
|
||||
**openssl** can manually generate certificates for your cluster.
|
||||
|
||||
1. Generate a ca.key with 2048bit:
|
||||
|
||||
openssl genrsa -out ca.key 2048
|
||||
1. According to the ca.key generate a ca.crt (use -days to set the certificate effective time):
|
||||
|
||||
openssl req -x509 -new -nodes -key ca.key -subj "/CN=${MASTER_IP}" -days 10000 -out ca.crt
|
||||
1. Generate a server.key with 2048bit:
|
||||
|
||||
openssl genrsa -out server.key 2048
|
||||
1. Create a config file for generating a Certificate Signing Request (CSR).
|
||||
Be sure to substitute the values marked with angle brackets (e.g. `<MASTER_IP>`)
|
||||
with real values before saving this to a file (e.g. `csr.conf`).
|
||||
Note that the value for `MASTER_CLUSTER_IP` is the service cluster IP for the
|
||||
API server as described in previous subsection.
|
||||
The sample below also assume that you are using `cluster.local` as the default
|
||||
DNS domain name.
|
||||
|
||||
[ req ]
|
||||
default_bits = 2048
|
||||
prompt = no
|
||||
default_md = sha256
|
||||
req_extensions = req_ext
|
||||
distinguished_name = dn
|
||||
|
||||
[ dn ]
|
||||
C = <country>
|
||||
ST = <state>
|
||||
L = <city>
|
||||
O = <organization>
|
||||
OU = <organization unit>
|
||||
CN = <MASTER_IP>
|
||||
|
||||
[ req_ext ]
|
||||
subjectAltName = @alt_names
|
||||
|
||||
[ alt_names ]
|
||||
DNS.1 = kubernetes
|
||||
DNS.2 = kubernetes.default
|
||||
DNS.3 = kubernetes.default.svc
|
||||
DNS.4 = kubernetes.default.svc.cluster
|
||||
DNS.5 = kubernetes.default.svc.cluster.local
|
||||
IP.1 = <MASTER_IP>
|
||||
IP.2 = <MASTER_CLUSTER_IP>
|
||||
|
||||
[ v3_ext ]
|
||||
authorityKeyIdentifier=keyid,issuer:always
|
||||
basicConstraints=CA:FALSE
|
||||
keyUsage=keyEncipherment,dataEncipherment
|
||||
extendedKeyUsage=serverAuth,clientAuth
|
||||
subjectAltName=@alt_names
|
||||
1. Generate the certificate signing request based on the config file:
|
||||
|
||||
openssl req -new -key server.key -out server.csr -config csr.conf
|
||||
1. Generate the server certificate using the ca.key, ca.crt and server.csr:
|
||||
|
||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
|
||||
-CAcreateserial -out server.crt -days 10000 \
|
||||
-extensions v3_ext -extfile csr.conf
|
||||
1. View the certificate:
|
||||
|
||||
openssl x509 -noout -text -in ./server.crt
|
||||
|
||||
Finally, add the same parameters into the API server start parameters.
|
||||
|
||||
### cfssl
|
||||
|
||||
**cfssl** is another tool for certificate generation.
|
||||
|
||||
1. Download, unpack and prepare the command line tools as shown below.
|
||||
Note that you may need to adapt the sample commands based on the hardware
|
||||
architecture and cfssl version you are using.
|
||||
|
||||
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o cfssl
|
||||
chmod +x cfssl
|
||||
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o cfssljson
|
||||
chmod +x cfssljson
|
||||
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o cfssl-certinfo
|
||||
chmod +x cfssl-certinfo
|
||||
1. Create a directory to hold the artifacts and initialize cfssl:
|
||||
|
||||
mkdir cert
|
||||
cd cert
|
||||
../cfssl print-defaults config > config.json
|
||||
../cfssl print-defaults csr > csr.json
|
||||
1. Create a JSON config file for generating the CA file, for example, `ca-config.json`:
|
||||
|
||||
{
|
||||
"signing": {
|
||||
"default": {
|
||||
"expiry": "8760h"
|
||||
},
|
||||
"profiles": {
|
||||
"kubernetes": {
|
||||
"usages": [
|
||||
"signing",
|
||||
"key encipherment",
|
||||
"server auth",
|
||||
"client auth"
|
||||
],
|
||||
"expiry": "8760h"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
1. Create a JSON config file for CA certificate signing request (CSR), for example,
|
||||
`ca-csr.json`. Be sure the replace the values marked with angle brackets with
|
||||
real values you want to use.
|
||||
|
||||
{
|
||||
"CN": "kubernetes",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"names":[{
|
||||
"C": "<country>",
|
||||
"ST": "<state>",
|
||||
"L": "<city>",
|
||||
"O": "<organization>",
|
||||
"OU": "<organization unit>"
|
||||
}]
|
||||
}
|
||||
1. Generate CA key (`ca-key.pem`) and certificate (`ca.pem`):
|
||||
|
||||
../cfssl gencert -initca ca-csr.json | ../cfssljson -bare ca
|
||||
1. Create a JSON config file for generating keys and certificates for the API
|
||||
server as shown below. Be sure to replace the values in angle brackets with
|
||||
real values you want to use. The `MASTER_CLUSTER_IP` is the service cluster
|
||||
IP for the API server as described in previous subsection.
|
||||
The sample below also assume that you are using `cluster.local` as the default
|
||||
DNS domain name.
|
||||
|
||||
{
|
||||
"CN": "kubernetes",
|
||||
"hosts": [
|
||||
"127.0.0.1",
|
||||
"<MASTER_IP>",
|
||||
"<MASTER_CLUSTER_IP>",
|
||||
"kubernetes",
|
||||
"kubernetes.default",
|
||||
"kubernetes.default.svc",
|
||||
"kubernetes.default.svc.cluster",
|
||||
"kubernetes.default.svc.cluster.local"
|
||||
],
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"names": [{
|
||||
"C": "<country>",
|
||||
"ST": "<state>",
|
||||
"L": "<city>",
|
||||
"O": "<organization>",
|
||||
"OU": "<organization unit>"
|
||||
}]
|
||||
}
|
||||
1. Generate the key and certificate for the API server, which are by default
|
||||
saved into file `server-key.pem` and `server.pem` respectively:
|
||||
|
||||
../cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
|
||||
--config=ca-config.json -profile=kubernetes \
|
||||
server-csr.json | ../cfssljson -bare server
|
||||
|
||||
|
||||
## Distributing Self-Signed CA Certificate
|
||||
|
||||
A client node may refuse to recognize a self-signed CA certificate as valid.
|
||||
For a non-production deployment, or for a deployment that runs behind a company
|
||||
firewall, you can distribute a self-signed CA certificate to all clients and
|
||||
refresh the local list for valid certificates.
|
||||
|
||||
On each client, perform the following operations:
|
||||
|
||||
```bash
|
||||
$ sudo cp ca.crt /usr/local/share/ca-certificates/kubernetes.crt
|
||||
$ sudo update-ca-certificates
|
||||
Updating certificates in /etc/ssl/certs...
|
||||
1 added, 0 removed; done.
|
||||
Running hooks in /etc/ca-certificates/update.d....
|
||||
done.
|
||||
```
|
||||
|
||||
## Certificates API
|
||||
|
||||
You can use the `certificates.k8s.io` API to provision
|
||||
x509 certificates to use for authentication as documented
|
||||
[here](/docs/tasks/tls/managing-tls-in-a-cluster).
|
||||
@@ -0,0 +1,269 @@
|
||||
---
|
||||
title: Cloud Providers
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page explains how to manage Kubernetes running on a specific
|
||||
cloud provider.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
# AWS
|
||||
This section describes all the possible configurations which can
|
||||
be used when running Kubernetes on Amazon Web Services.
|
||||
|
||||
## Load Balancers
|
||||
You can setup [external load balancers](/docs/tasks/access-application-cluster/create-external-load-balancer/)
|
||||
to use specific features in AWS by configuring the annotations as shown below.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: example
|
||||
namespace: kube-system
|
||||
labels:
|
||||
run: example
|
||||
annotations:
|
||||
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xx-xxxx-x:xxxxxxxxx:xxxxxxx/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx #replace this value
|
||||
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 5556
|
||||
protocol: TCP
|
||||
selector:
|
||||
app: example
|
||||
```
|
||||
Different settings can be applied to a load balancer service in AWS using _annotations_. The following describes the annotations supported on AWS ELBs:
|
||||
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval`: Used to specify access log emit interval.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-access-log-enabled`: Used on the service to enable or disable access logs.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name`: Used to specify access log s3 bucket name.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix`: Used to specify access log s3 bucket prefix.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags`: Used on the service to specify a comma-separated list of key-value pairs which will be recorded as additional tags in the ELB. For example: `"Key1=Val1,Key2=Val2,KeyNoVal1=,KeyNoVal2"`.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-backend-protocol`: Used on the service to specify the protocol spoken by the backend (pod) behind a listener. If `http` (default) or `https`, an HTTPS listener that terminates the connection and parses headers is created. If set to `ssl` or `tcp`, a "raw" SSL listener is used. If set to `http` and `aws-load-balancer-ssl-cert` is not used then a HTTP listener is used.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-ssl-cert`: Used on the service to request a secure listener. Value is a valid certificate ARN. For more, see [ELB Listener Config](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html) CertARN is an IAM or CM certificate ARN, e.g. `arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012`.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled`: Used on the service to enable or disable connection draining.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout`: Used on the service to specify a connection draining timeout.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout`: Used on the service to specify the idle connection timeout.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled`: Used on the service to enable or disable cross-zone load balancing.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-extra-security-groups`: Used on the service to specify additional security groups to be added to ELB created
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-internal`: Used on the service to indicate that we want an internal ELB.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-proxy-protocol`: Used on the service to enable the proxy protocol on an ELB. Right now we only accept the value `*` which means enable the proxy protocol on all ELB backends. In the future we could adjust this to allow setting the proxy protocol only on certain backends.
|
||||
* `service.beta.kubernetes.io/aws-load-balancer-ssl-ports`: Used on the service to specify a comma-separated list of ports that will use SSL/HTTPS listeners. Defaults to `*` (all)
|
||||
|
||||
The information for the annotations for AWS is taken from the comments on [aws.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go)
|
||||
|
||||
# OpenStack
|
||||
This section describes all the possible configurations which can
|
||||
be used when using OpenStack with Kubernetes. The OpenStack cloud provider
|
||||
implementation for Kubernetes supports the use of these OpenStack services from
|
||||
the underlying cloud, where available:
|
||||
|
||||
| Service | API Version(s) | Required |
|
||||
|--------------------------|----------------|----------|
|
||||
| Block Storage (Cinder) | V1†, V2, V3 | No |
|
||||
| Compute (Nova) | V2 | No |
|
||||
| Identity (Keystone) | V2‡, V3 | Yes |
|
||||
| Load Balancing (Neutron) | V1§, V2 | No |
|
||||
| Load Balancing (Octavia) | V2 | No |
|
||||
|
||||
† Block Storage V1 API support is deprecated, Block Storage V3 API support was
|
||||
added in Kubernetes 1.9.
|
||||
|
||||
‡ Identity V2 API support is deprecated and will be removed from the provider in
|
||||
a future release. As of the "Queens" release, OpenStack will no longer expose the
|
||||
Identity V2 API.
|
||||
|
||||
§ Load Balancing V1 API support was removed in Kubernetes 1.9.
|
||||
|
||||
Service discovery is achieved by listing the service catalog managed by
|
||||
OpenStack Identity (Keystone) using the `auth-url` provided in the provider
|
||||
configuration. The provider will gracefully degrade in functionality when
|
||||
OpenStack services other than Keystone are not available and simply disclaim
|
||||
support for impacted features. Certain features are also enabled or disabled
|
||||
based on the list of extensions published by Neutron in the underlying cloud.
|
||||
|
||||
## cloud.conf
|
||||
Kubernetes knows how to interact with OpenStack via the file cloud.conf. It is
|
||||
the file that will provide Kubernetes with credentials and location for the OpenStack auth endpoint.
|
||||
You can create a cloud.conf file by specifying the following details in it
|
||||
|
||||
### Typical configuration
|
||||
This is an example of a typical configuration that touches the values that most
|
||||
often need to be set. It points the provider at the OpenStack cloud's Keystone
|
||||
endpoint, provides details for how to authenticate with it, and configures the
|
||||
load balancer:
|
||||
|
||||
```yaml
|
||||
[Global]
|
||||
username=user
|
||||
password=pass
|
||||
auth-url=https://<keystone_ip>/identity/v3
|
||||
tenant-id=c869168a828847f39f7f06edd7305637
|
||||
domain-id=2a73b8f597c04551a0fdc8e95544be8a
|
||||
|
||||
[LoadBalancer]
|
||||
subnet-id=6937f8fa-858d-4bc9-a3a5-18d2c957166a
|
||||
```
|
||||
|
||||
#### Global
|
||||
These configuration options for the OpenStack provider pertain to its global
|
||||
configuration and should appear in the `[Global]` section of the `cloud.conf`
|
||||
file:
|
||||
|
||||
* `auth-url` (Required): The URL of the keystone API used to authenticate. On
|
||||
OpenStack control panels, this can be found at Access and Security > API
|
||||
Access > Credentials.
|
||||
* `username` (Required): Refers to the username of a valid user set in keystone.
|
||||
* `password` (Required): Refers to the password of a valid user set in keystone.
|
||||
* `tenant-id` (Required): Used to specify the id of the project where you want
|
||||
to create your resources.
|
||||
* `tenant-name` (Optional): Used to specify the name of the project where you
|
||||
want to create your resources.
|
||||
* `trust-id` (Optional): Used to specify the identifier of the trust to use for
|
||||
authorization. A trust represents a user's (the trustor) authorization to
|
||||
delegate roles to another user (the trustee), and optionally allow the trustee
|
||||
to impersonate the trustor. Available trusts are found under the
|
||||
`/v3/OS-TRUST/trusts` endpoint of the Keystone API.
|
||||
* `domain-id` (Optional): Used to specify the id of the domain your user belongs
|
||||
to.
|
||||
* `domain-name` (Optional): Used to specify the name of the domain your user
|
||||
belongs to.
|
||||
* `region` (Optional): Used to specify the identifier of the region to use when
|
||||
running on a multi-region OpenStack cloud. A region is a general division of
|
||||
an OpenStack deployment. Although a region does not have a strict geographical
|
||||
connotation, a deployment can use a geographical name for a region identifier
|
||||
such as `us-east`. Available regions are found under the `/v3/regions`
|
||||
endpoint of the Keystone API.
|
||||
* `ca-file` (Optional): Used to specify the path to your custom CA file.
|
||||
|
||||
|
||||
When using Keystone V3 - which changes tenant to project - the `tenant-id` value
|
||||
is automatically mapped to the project construct in the API.
|
||||
|
||||
#### Load Balancer
|
||||
These configuration options for the OpenStack provider pertain to the load
|
||||
balancer and should appear in the `[LoadBalancer]` section of the `cloud.conf`
|
||||
file:
|
||||
|
||||
* `lb-version` (Optional): Used to override automatic version detection. Valid
|
||||
values are `v1` or `v2`. Where no value is provided automatic detection will
|
||||
select the highest supported version exposed by the underlying OpenStack
|
||||
cloud.
|
||||
* `use-octavia` (Optional): Used to determine whether to look for and use an
|
||||
Octavia LBaaS V2 service catalog endpoint. Valid values are `true` or `false`.
|
||||
Where `true` is specified and an Octaiva LBaaS V2 entry can not be found, the
|
||||
provider will fall back and attempt to find a Neutron LBaaS V2 endpoint
|
||||
instead. The default value is `false`.
|
||||
* `subnet-id` (Optional): Used to specify the id of the subnet you want to
|
||||
create your loadbalancer on. Can be found at Network > Networks. Click on the
|
||||
respective network to get its subnets.
|
||||
* `floating-network-id` (Optional): If specified, will create a floating IP for
|
||||
the load balancer.
|
||||
* `lb-method` (Optional): Used to specify algorithm by which load will be
|
||||
distributed amongst members of the load balancer pool. The value can be
|
||||
`ROUND_ROBIN`, `LEAST_CONNECTIONS`, or `SOURCE_IP`. The default behavior if
|
||||
none is specified is `ROUND_ROBIN`.
|
||||
* `lb-provider` (Optional): Used to specify the provider of the load balancer.
|
||||
If not specified, the default provider service configured in neutron will be
|
||||
used.
|
||||
* `create-monitor` (Optional): Indicates whether or not to create a health
|
||||
monitor for the Neutron load balancer. Valid values are `true` and `false`.
|
||||
The default is `false`. When `true` is specified then `monitor-delay`,
|
||||
`monitor-timeout`, and `monitor-max-retries` must also be set.
|
||||
* `monitor-delay` (Optional): The time, in seconds, between sending probes to
|
||||
members of the load balancer.
|
||||
* `monitor-timeout` (Optional): Maximum number of seconds for a monitor to wait
|
||||
for a ping reply before it times out. The value must be less than the delay
|
||||
value.
|
||||
* `monitor-max-retries` (Optional): Number of permissible ping failures before
|
||||
changing the load balancer member's status to INACTIVE. Must be a number
|
||||
between 1 and 10.
|
||||
* `manage-security-groups` (Optional): Determines whether or not the load
|
||||
balancer should automatically manage the security group rules. Valid values
|
||||
are `true` and `false`. The default is `false`. When `true` is specified
|
||||
`node-security-group` must also be supplied.
|
||||
* `node-security-group` (Optional): ID of the security group to manage.
|
||||
|
||||
#### Block Storage
|
||||
These configuration options for the OpenStack provider pertain to block storage
|
||||
and should appear in the `[BlockStorage]` section of the `cloud.conf` file:
|
||||
|
||||
* `bs-version` (Optional): Used to override automatic version detection. Valid
|
||||
values are `v1`, `v2`, `v3` and `auto`. When `auto` is specified automatic
|
||||
detection will select the highest supported version exposed by the underlying
|
||||
OpenStack cloud. The default value if none is provided is `auto`.
|
||||
* `trust-device-path` (Optional): In most scenarios the block device names
|
||||
provided by Cinder (e.g. `/dev/vda`) can not be trusted. This boolean toggles
|
||||
this behavior. Setting it to `true` results in trusting the block device names
|
||||
provided by Cinder. The default value of `false` results in the discovery of
|
||||
the device path based on its serial number and `/dev/disk/by-id` mapping and is
|
||||
the recommended approach.
|
||||
* `ignore-volume-az` (Optional): Used to influence availability zone use when
|
||||
attaching Cinder volumes. When Nova and Cinder have different availability
|
||||
zones, this should be set to `true`. This is most commonly the case where
|
||||
there are many Nova availability zones but only one Cinder availability zone.
|
||||
The default value is `false` to preserve the behavior used in earlier
|
||||
releases, but may change in the future.
|
||||
|
||||
If deploying Kubernetes versions <= 1.8 on an OpenStack deployment that uses
|
||||
paths rather than ports to differentiate between endpoints it may be necessary
|
||||
to explicitly set the `bs-version` parameter. A path based endpoint is of the
|
||||
form `http://foo.bar/volume` while a port based endpoint is of the form
|
||||
`http://foo.bar:xxx`.
|
||||
|
||||
In environments that use path based endpoints and Kubernetes is using the older
|
||||
auto-detection logic a `BS API version autodetection failed.` error will be
|
||||
returned on attempting volume detachment. To workaround this issue it is
|
||||
possible to force the use of Cinder API version 2 by adding this to the cloud
|
||||
provider configuration:
|
||||
|
||||
```yaml
|
||||
[BlockStorage]
|
||||
bs-version=v2
|
||||
```
|
||||
|
||||
#### Metadata
|
||||
These configuration options for the OpenStack provider pertain to metadata and
|
||||
should appear in the `[Metadata]` section of the `cloud.conf` file:
|
||||
|
||||
* `search-order` (Optional): This configuration key influences the way that the
|
||||
provider retrieves metadata relating to the instance(s) in which it runs. The
|
||||
default value of `configDrive,metadataService` results in the provider
|
||||
retrieving metadata relating to the instance from the config drive first if
|
||||
available and then the metadata service. Alternative values are:
|
||||
* `configDrive` - Only retrieve instance metadata from the configuration
|
||||
drive.
|
||||
* `metadataService` - Only retrieve instance metadata from the metadata
|
||||
service.
|
||||
* `metadataService,configDrive` - Retrieve instance metadata from the metadata
|
||||
service first if available, then the configuration drive.
|
||||
|
||||
Influencing this behavior may be desirable as the metadata on the
|
||||
configuration drive may grow stale over time, whereas the metadata service
|
||||
always provides the most up to date view. Not all OpenStack clouds provide
|
||||
both configuration drive and metadata service though and only one or the other
|
||||
may be available which is why the default is to check both.
|
||||
|
||||
#### Router
|
||||
|
||||
These configuration options for the OpenStack provider pertain to the [kubenet]
|
||||
Kubernetes network plugin and should appear in the `[Router]` section of the
|
||||
`cloud.conf` file:
|
||||
|
||||
* `router-id` (Optional): If the underlying cloud's Neutron deployment supports
|
||||
the `extraroutes` extension then use `router-id` to specify a router to add
|
||||
routes to. The router chosen must span the private networks containing your
|
||||
cluster nodes (typically there is only one node network, and this value should be
|
||||
the default router for the node network). This value is required to use [kubenet]
|
||||
on OpenStack.
|
||||
|
||||
[kubenet]: https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
reviewers:
|
||||
- davidopp
|
||||
- lavalamp
|
||||
title: Cluster Administration Overview
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
The cluster administration overview is for anyone creating or administering a Kubernetes cluster.
|
||||
It assumes some familiarity with core Kubernetes [concepts](/docs/concepts/).
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
## Planning a cluster
|
||||
|
||||
See the guides in [Picking the Right Solution](/docs/setup/pick-right-solution/) for examples of how to plan, set up, and configure Kubernetes clusters. The solutions listed in this article are called *distros*.
|
||||
|
||||
Before choosing a guide, here are some considerations:
|
||||
|
||||
- Do you just want to try out Kubernetes on your computer, or do you want to build a high-availability, multi-node cluster? Choose distros best suited for your needs.
|
||||
- **If you are designing for high-availability**, learn about configuring [clusters in multiple zones](/docs/concepts/cluster-administration/federation/).
|
||||
- Will you be using **a hosted Kubernetes cluster**, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), or **hosting your own cluster**?
|
||||
- Will your cluster be **on-premises**, or **in the cloud (IaaS)**? Kubernetes does not directly support hybrid clusters. Instead, you can set up multiple clusters.
|
||||
- **If you are configuring Kubernetes on-premises**, consider which [networking model](/docs/concepts/cluster-administration/networking/) fits best.
|
||||
- Will you be running Kubernetes on **"bare metal" hardware** or on **virtual machines (VMs)**?
|
||||
- Do you **just want to run a cluster**, or do you expect to do **active development of Kubernetes project code**? If the
|
||||
latter, choose an actively-developed distro. Some distros only use binary releases, but
|
||||
offer a greater variety of choices.
|
||||
- Familiarize yourself with the [components](/docs/admin/cluster-components/) needed to run a cluster.
|
||||
|
||||
Note: Not all distros are actively maintained. Choose distros which have been tested with a recent version of Kubernetes.
|
||||
|
||||
If you are using a guide involving Salt, see [Configuring Kubernetes with Salt](/docs/admin/salt/).
|
||||
|
||||
## Managing a cluster
|
||||
|
||||
* [Managing a cluster](/docs/tasks/administer-cluster/cluster-management/) describes several topics related to the lifecycle of a cluster: creating a new cluster, upgrading your cluster’s master and worker nodes, performing node maintenance (e.g. kernel upgrades), and upgrading the Kubernetes API version of a running cluster.
|
||||
|
||||
* Learn how to [manage nodes](/docs/concepts/nodes/node/).
|
||||
|
||||
* Learn how to set up and manage the [resource quota](/docs/concepts/policy/resource-quotas/) for shared clusters.
|
||||
|
||||
## Securing a cluster
|
||||
|
||||
* [Certificates](/docs/concepts/cluster-administration/certificates/) describes the steps to generate certificates using different tool chains.
|
||||
|
||||
* [Kubernetes Container Environment](/docs/concepts/containers/container-environment-variables/) describes the environment for Kubelet managed containers on a Kubernetes node.
|
||||
|
||||
* [Controlling Access to the Kubernetes API](/docs/admin/accessing-the-api/) describes how to set up permissions for users and service accounts.
|
||||
|
||||
* [Authenticating](/docs/admin/authentication/) explains authentication in Kubernetes, including the various authentication options.
|
||||
|
||||
* [Authorization](/docs/admin/authorization/) is separate from authentication, and controls how HTTP calls are handled.
|
||||
|
||||
* [Using Admission Controllers](/docs/admin/admission-controllers/) explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization.
|
||||
|
||||
* [Using Sysctls in a Kubernetes Cluster](/docs/concepts/cluster-administration/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters .
|
||||
|
||||
* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs.
|
||||
|
||||
### Securing the kubelet
|
||||
* [Master-Node communication](/docs/concepts/architecture/master-node-communication/)
|
||||
* [TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/)
|
||||
* [Kubelet authentication/authorization](/docs/admin/kubelet-authentication-authorization/)
|
||||
|
||||
## Optional Cluster Services
|
||||
|
||||
* [DNS Integration with SkyDNS](/docs/concepts/services-networking/dns-pod-service/) describes how to resolve a DNS name directly to a Kubernetes service.
|
||||
|
||||
* [Logging and Monitoring Cluster Activity](/docs/concepts/cluster-administration/logging/) explains how logging in Kubernetes works and how to implement it.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Controller manager metrics
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
Controller manager metrics provide important insight into the performance and health of
|
||||
the controller manager.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
## What are controller manager metrics
|
||||
|
||||
Controller manager metrics provide important insight into the performance and health of the controller manager.
|
||||
These metrics include common Go language runtime metrics such as go_routine count and controller specific metrics such as
|
||||
etcd request latencies or Cloudprovider (AWS, GCE, OpenStack) API latencies that can be used
|
||||
to gauge the health of a cluster.
|
||||
|
||||
Starting from Kubernetes 1.7, detailed Cloudprovider metrics are available for storage operations for GCE, AWS, Vsphere and OpenStack.
|
||||
These metrics can be used to monitor health of persistent volume operations.
|
||||
|
||||
For example, for GCE these metrics are called:
|
||||
|
||||
```
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "instance_list"}
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "disk_insert"}
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "disk_delete"}
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "attach_disk"}
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "detach_disk"}
|
||||
cloudprovider_gce_api_request_duration_seconds { request = "list_disk"}
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
|
||||
In a cluster, controller-manager metrics are available from `http://localhost:10252/metrics`
|
||||
from the host where the controller-manager is running.
|
||||
|
||||
The metrics are emitted in [prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/) and are human readable.
|
||||
|
||||
In a production environment you may want to configure prometheus or some other metrics scraper
|
||||
to periodically gather these metrics and make them available in some kind of time series database.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: counter
|
||||
spec:
|
||||
containers:
|
||||
- name: count
|
||||
image: busybox
|
||||
args: [/bin/sh, -c,
|
||||
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
reviewers:
|
||||
title: Device Plugins
|
||||
description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup.
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{< feature-state state="beta" >}}
|
||||
|
||||
{{% capture overview %}}
|
||||
Starting in version 1.8, Kubernetes provides a
|
||||
[device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md)
|
||||
for vendors to advertise their resources to the kubelet without changing Kubernetes core code.
|
||||
Instead of writing custom Kubernetes code, vendors can implement a device plugin that can
|
||||
be deployed manually or as a DaemonSet. The targeted devices include GPUs,
|
||||
High-performance NICs, FPGAs, InfiniBand, and other similar computing resources
|
||||
that may require vendor specific initialization and setup.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Device plugin registration
|
||||
|
||||
The device plugins feature is gated by the `DevicePlugins` feature gate which
|
||||
is disabled by default before 1.10. When the device plugins feature is enabled,
|
||||
the kubelet exports a `Registration` gRPC service:
|
||||
|
||||
```gRPC
|
||||
service Registration {
|
||||
rpc Register(RegisterRequest) returns (Empty) {}
|
||||
}
|
||||
```
|
||||
A device plugin can register itself with the kubelet through this gRPC service.
|
||||
During the registration, the device plugin needs to send:
|
||||
|
||||
* The name of its Unix socket.
|
||||
* The Device Plugin API version against which it was built.
|
||||
* The `ResourceName` it wants to advertise. Here `ResourceName` needs to follow the
|
||||
[extended resource naming scheme](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#extended-resources)
|
||||
as `vendor-domain/resource`.
|
||||
For example, an Nvidia GPU is advertised as `nvidia.com/gpu`.
|
||||
|
||||
Following a successful registration, the device plugin sends the kubelet the
|
||||
list of devices it manages, and the kubelet is then in charge of advertising those
|
||||
resources to the API server as part of the kubelet node status update.
|
||||
For example, after a device plugin registers `vendor-domain/foo` with the kubelet
|
||||
and reports two healthy devices on a node, the node status is updated
|
||||
to advertise 2 `vendor-domain/foo`.
|
||||
|
||||
Then, users can request devices in a
|
||||
[Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core)
|
||||
specification as they request other types of resources, with the following limitations:
|
||||
* Extended resources are only supported as integer resources and cannot be overcommitted.
|
||||
* Devices cannot be shared among Containers.
|
||||
|
||||
Suppose a Kubernetes cluster is running a device plugin that advertises resource `vendor-domain/resource`
|
||||
on certain nodes, here is an example user pod requesting this resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: demo-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: demo-container-1
|
||||
image: k8s.gcr.io/pause:2.0
|
||||
resources:
|
||||
limits:
|
||||
vendor-domain/resource: 2 # requesting 2 vendor-domain/resource
|
||||
```
|
||||
|
||||
## Device plugin implementation
|
||||
|
||||
The general workflow of a device plugin includes the following steps:
|
||||
|
||||
* Initialization. During this phase, the device plugin performs vendor specific
|
||||
initialization and setup to make sure the devices are in a ready state.
|
||||
|
||||
* The plugin starts a gRPC service, with a Unix socket under host path
|
||||
`/var/lib/kubelet/device-plugins/`, that implements the following interfaces:
|
||||
|
||||
```gRPC
|
||||
service DevicePlugin {
|
||||
// ListAndWatch returns a stream of List of Devices
|
||||
// Whenever a Device state change or a Device disappears, ListAndWatch
|
||||
// returns the new list
|
||||
rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
|
||||
|
||||
// Allocate is called during container creation so that the Device
|
||||
// Plugin can run device specific operations and instruct Kubelet
|
||||
// of the steps to make the Device available in the container
|
||||
rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
|
||||
}
|
||||
```
|
||||
|
||||
* The plugin registers itself with the kubelet through the Unix socket at host
|
||||
path `/var/lib/kubelet/device-plugins/kubelet.sock`.
|
||||
|
||||
* After successfully registering itself, the device plugin runs in serving mode, during which it keeps
|
||||
monitoring device health and reports back to the kubelet upon any device state changes.
|
||||
It is also responsible for serving `Allocate` gRPC requests. During `Allocate`, the device plugin may
|
||||
do device-specific preparation; for example, GPU cleanup or QRNG initialization.
|
||||
If the operations succeed, the device plugin returns an `AllocateResponse` that contains container
|
||||
runtime configurations for accessing the allocated devices. The kubelet passes this information
|
||||
to the container runtime.
|
||||
|
||||
A device plugin is expected to detect kubelet restarts and re-register itself with the new
|
||||
kubelet instance. In the current implementation, a new kubelet instance deletes all the existing Unix sockets
|
||||
under `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion
|
||||
of its Unix socket and re-register itself upon such an event.
|
||||
|
||||
## Device plugin deployment
|
||||
|
||||
A device plugin can be deployed manually or as a DaemonSet. Being deployed as a DaemonSet has
|
||||
the benefit that Kubernetes can restart the device plugin if it fails.
|
||||
Otherwise, an extra mechanism is needed to recover from device plugin failures.
|
||||
The canonical directory `/var/lib/kubelet/device-plugins` requires privileged access,
|
||||
so a device plugin must run in a privileged security context.
|
||||
If a device plugin is running as a DaemonSet, `/var/lib/kubelet/device-plugins`
|
||||
must be mounted as a
|
||||
[Volume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core)
|
||||
in the plugin's
|
||||
[PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core).
|
||||
|
||||
Kubernetes device plugin support is still in alpha. As development continues, its API version can
|
||||
change in incompatible ways. We recommend that device plugin developers do the following:
|
||||
* Watch for changes in future releases.
|
||||
* Support multiple versions of the device plugin API for backward/forward compatibility.
|
||||
|
||||
If you enable the DevicePlugins feature and run device plugins on nodes that need to be upgraded to
|
||||
a Kubernetes release with a newer device plugin API version, upgrade your device plugins
|
||||
to support both versions before upgrading these nodes to
|
||||
ensure the continuous functioning of the device allocations during the upgrade.
|
||||
|
||||
## Examples
|
||||
|
||||
For examples of device plugin implementations, see:
|
||||
* The official [NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin)
|
||||
* it requires using [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) which allows you to run GPU enabled docker containers
|
||||
* The [NVIDIA GPU device plugin for COS base OS](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu).
|
||||
* The [RDMA device plugin](https://github.com/hustcat/k8s-rdma-device-plugin)
|
||||
* The [Solarflare device plugin](https://github.com/vikaschoudhary16/sfc-device-plugin)
|
||||
* The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin)
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
---
|
||||
title: Federation
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page explains why and how to manage multiple Kubernetes clusters using
|
||||
federation.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
## Why federation
|
||||
|
||||
Federation makes it easy to manage multiple clusters. It does so by providing 2
|
||||
major building blocks:
|
||||
|
||||
* Sync resources across clusters: Federation provides the ability to keep
|
||||
resources in multiple clusters in sync. For example, you can ensure that the same deployment exists in multiple clusters.
|
||||
* Cross cluster discovery: Federation provides the ability to auto-configure DNS servers and load balancers with backends from all clusters. For example, you can ensure that a global VIP or DNS record can be used to access backends from multiple clusters.
|
||||
|
||||
Some other use cases that federation enables are:
|
||||
|
||||
* High Availability: By spreading load across clusters and auto configuring DNS
|
||||
servers and load balancers, federation minimises the impact of cluster
|
||||
failure.
|
||||
* Avoiding provider lock-in: By making it easier to migrate applications across
|
||||
clusters, federation prevents cluster provider lock-in.
|
||||
|
||||
|
||||
Federation is not helpful unless you have multiple clusters. Some of the reasons
|
||||
why you might want multiple clusters are:
|
||||
|
||||
* Low latency: Having clusters in multiple regions minimises latency by serving
|
||||
users from the cluster that is closest to them.
|
||||
* Fault isolation: It might be better to have multiple small clusters rather
|
||||
than a single large cluster for fault isolation (for example: multiple
|
||||
clusters in different availability zones of a cloud provider).
|
||||
* Scalability: There are scalability limits to a single kubernetes cluster (this
|
||||
should not be the case for most users. For more details:
|
||||
[Kubernetes Scaling and Performance Goals](https://git.k8s.io/community/sig-scalability/goals.md)).
|
||||
* [Hybrid cloud](#hybrid-cloud-capabilities): You can have multiple clusters on different cloud providers or
|
||||
on-premises data centers.
|
||||
|
||||
### Caveats
|
||||
|
||||
While there are a lot of attractive use cases for federation, there are also
|
||||
some caveats:
|
||||
|
||||
* Increased network bandwidth and cost: The federation control plane watches all
|
||||
clusters to ensure that the current state is as expected. This can lead to
|
||||
significant network cost if the clusters are running in different regions on
|
||||
a cloud provider or on different cloud providers.
|
||||
* Reduced cross cluster isolation: A bug in the federation control plane can
|
||||
impact all clusters. This is mitigated by keeping the logic in federation
|
||||
control plane to a minimum. It mostly delegates to the control plane in
|
||||
kubernetes clusters whenever it can. The design and implementation also errs
|
||||
on the side of safety and avoiding multi-cluster outage.
|
||||
* Maturity: The federation project is relatively new and is not very mature.
|
||||
Not all resources are available and many are still alpha. [Issue
|
||||
88](https://github.com/kubernetes/federation/issues/88) enumerates
|
||||
known issues with the system that the team is busy solving.
|
||||
|
||||
### Hybrid cloud capabilities
|
||||
|
||||
Federations of Kubernetes Clusters can include clusters running in
|
||||
different cloud providers (e.g. Google Cloud, AWS), and on-premises
|
||||
(e.g. on OpenStack). [Kubefed](https://kubernetes.io/docs/tasks/federation/set-up-cluster-federation-kubefed/) is the recommended way to deploy federated clusters.
|
||||
|
||||
Thereafter, your [API resources](#api-resources) can span different clusters
|
||||
and cloud providers.
|
||||
|
||||
## Setting up federation
|
||||
|
||||
To be able to federate multiple clusters, you first need to set up a federation
|
||||
control plane.
|
||||
Follow the [setup guide](/docs/tutorials/federation/set-up-cluster-federation-kubefed/) to set up the
|
||||
federation control plane.
|
||||
|
||||
## API resources
|
||||
|
||||
Once you have the control plane set up, you can start creating federation API
|
||||
resources.
|
||||
The following guides explain some of the resources in detail:
|
||||
|
||||
* [Cluster](/docs/tasks/administer-federation/cluster/)
|
||||
* [ConfigMap](/docs/tasks/administer-federation/configmap/)
|
||||
* [DaemonSets](/docs/tasks/administer-federation/daemonset/)
|
||||
* [Deployment](/docs/tasks/administer-federation/deployment/)
|
||||
* [Events](/docs/tasks/administer-federation/events/)
|
||||
* [Hpa](/docs/tasks/administer-federation/hpa/)
|
||||
* [Ingress](/docs/tasks/administer-federation/ingress/)
|
||||
* [Jobs](/docs/tasks/administer-federation/job/)
|
||||
* [Namespaces](/docs/tasks/administer-federation/namespaces/)
|
||||
* [ReplicaSets](/docs/tasks/administer-federation/replicaset/)
|
||||
* [Secrets](/docs/tasks/administer-federation/secret/)
|
||||
* [Services](/docs/concepts/cluster-administration/federation-service-discovery/)
|
||||
|
||||
|
||||
The [API reference docs](/docs/reference/generated/federation/) list all the
|
||||
resources supported by federation apiserver.
|
||||
|
||||
## Cascading deletion
|
||||
|
||||
Kubernetes version 1.6 includes support for cascading deletion of federated
|
||||
resources. With cascading deletion, when you delete a resource from the
|
||||
federation control plane, you also delete the corresponding resources in all underlying clusters.
|
||||
|
||||
Cascading deletion is not enabled by default when using the REST API. To enable
|
||||
it, set the option `DeleteOptions.orphanDependents=false` when you delete a
|
||||
resource from the federation control plane using the REST API. Using `kubectl
|
||||
delete`
|
||||
enables cascading deletion by default. You can disable it by running `kubectl
|
||||
delete --cascade=false`
|
||||
|
||||
Note: Kubernetes version 1.5 included cascading deletion support for a subset of
|
||||
federation resources.
|
||||
|
||||
## Scope of a single cluster
|
||||
|
||||
On IaaS providers such as Google Compute Engine or Amazon Web Services, a VM exists in a
|
||||
[zone](https://cloud.google.com/compute/docs/zones) or [availability
|
||||
zone](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).
|
||||
We suggest that all the VMs in a Kubernetes cluster should be in the same availability zone, because:
|
||||
|
||||
- compared to having a single global Kubernetes cluster, there are fewer single-points of failure.
|
||||
- compared to a cluster that spans availability zones, it is easier to reason about the availability properties of a
|
||||
single-zone cluster.
|
||||
- when the Kubernetes developers are designing the system (e.g. making assumptions about latency, bandwidth, or
|
||||
correlated failures) they are assuming all the machines are in a single data center, or otherwise closely connected.
|
||||
|
||||
It is recommended to run fewer clusters with more VMs per availability zone; but it is possible to run multiple clusters per availability zones.
|
||||
|
||||
Reasons to prefer fewer clusters per availability zone are:
|
||||
|
||||
- improved bin packing of Pods in some cases with more nodes in one cluster (less resource fragmentation).
|
||||
- reduced operational overhead (though the advantage is diminished as ops tooling and processes mature).
|
||||
- reduced costs for per-cluster fixed resource costs, e.g. apiserver VMs (but small as a percentage
|
||||
of overall cluster cost for medium to large clusters).
|
||||
|
||||
Reasons to have multiple clusters include:
|
||||
|
||||
- strict security policies requiring isolation of one class of work from another (but, see Partitioning Clusters
|
||||
below).
|
||||
- test clusters to canary new Kubernetes releases or other cluster software.
|
||||
|
||||
## Selecting the right number of clusters
|
||||
|
||||
The selection of the number of Kubernetes clusters may be a relatively static choice, only revisited occasionally.
|
||||
By contrast, the number of nodes in a cluster and the number of pods in a service may change frequently according to
|
||||
load and growth.
|
||||
|
||||
To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run
|
||||
on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not
|
||||
be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions.
|
||||
Call the number of regions to be in `R`.
|
||||
|
||||
Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call
|
||||
the number that can be unavailable `U`. If you are not sure, then 1 is a fine choice.
|
||||
|
||||
If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then
|
||||
you need at least the larger of `R` or `U + 1` clusters. If it is not (e.g. you want to ensure low latency for all
|
||||
users in the event of a cluster failure), then you need to have `R * (U + 1)` clusters
|
||||
(`U + 1` in each of `R` regions). In any case, try to put each cluster in a different zone.
|
||||
|
||||
Finally, if any of your clusters would need more than the maximum recommended number of nodes for a Kubernetes cluster, then
|
||||
you may need even more clusters. Kubernetes v1.3 supports clusters up to 1000 nodes in size. Kubernetes v1.8 supports
|
||||
clusters up to 5000 nodes. See [Building Large Clusters](/docs/admin/cluster-large/) for more guidance.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
* Learn more about the [Federation
|
||||
proposal](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/design-proposals/multicluster/federation.md).
|
||||
* See this [setup guide](/docs/tutorials/federation/set-up-cluster-federation-kubefed/) for cluster federation.
|
||||
* See this [Kubecon2016 talk on federation](https://www.youtube.com/watch?v=pq9lbkmxpS8)
|
||||
* See this [Kubecon2017 Europe update on federation](https://www.youtube.com/watch?v=kwOvOLnFYck)
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
apiVersion: v1
|
||||
data:
|
||||
fluentd.conf: |
|
||||
<source>
|
||||
type tail
|
||||
format none
|
||||
path /var/log/1.log
|
||||
pos_file /var/log/1.log.pos
|
||||
tag count.format1
|
||||
</source>
|
||||
|
||||
<source>
|
||||
type tail
|
||||
format none
|
||||
path /var/log/2.log
|
||||
pos_file /var/log/2.log.pos
|
||||
tag count.format2
|
||||
</source>
|
||||
|
||||
<match **>
|
||||
type google_cloud
|
||||
</match>
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: fluentd-config
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
reviewers:
|
||||
- mikedanese
|
||||
title: Configuring kubelet Garbage Collection
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
Garbage collection is a helpful function of kubelet that will clean up unused images and unused containers. Kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.
|
||||
|
||||
External garbage collection tools are not recommended as these tools can potentially break the behavior of kubelet by removing containers expected to exist.
|
||||
|
||||
### Image Collection
|
||||
|
||||
Kubernetes manages lifecycle of all images through imageManager, with the cooperation
|
||||
of cadvisor.
|
||||
|
||||
The policy for garbage collecting images takes two factors into consideration:
|
||||
`HighThresholdPercent` and `LowThresholdPercent`. Disk usage above the high threshold
|
||||
will trigger garbage collection. The garbage collection will delete least recently used images until the low
|
||||
threshold has been met.
|
||||
|
||||
### Container Collection
|
||||
|
||||
The policy for garbage collecting containers considers three user-defined variables. `MinAge` is the minimum age at which a container can be garbage collected. `MaxPerPodContainer` is the maximum number of dead containers every single
|
||||
pod (UID, container name) pair is allowed to have. `MaxContainers` is the maximum number of total dead containers. These variables can be individually disabled by setting `MinAge` to zero and setting `MaxPerPodContainer` and `MaxContainers` respectively to less than zero.
|
||||
|
||||
Kubelet will act on containers that are unidentified, deleted, or outside of the boundaries set by the previously mentioned flags. The oldest containers will generally be removed first. `MaxPerPodContainer` and `MaxContainer` may potentially conflict with each other in situations where retaining the maximum number of containers per pod (`MaxPerPodContainer`) would go outside the allowable range of global dead containers (`MaxContainers`). `MaxPerPodContainer` would be adjusted in this situation: A worst case scenario would be to downgrade `MaxPerPodContainer` to 1 and evict the oldest containers. Additionally, containers owned by pods that have been deleted are removed once they are older than `MinAge`.
|
||||
|
||||
Containers that are not managed by kubelet are not subject to container garbage collection.
|
||||
|
||||
### User Configuration
|
||||
|
||||
Users can adjust the following thresholds to tune image garbage collection with the following kubelet flags :
|
||||
|
||||
1. `image-gc-high-threshold`, the percent of disk usage which triggers image garbage collection.
|
||||
Default is 90%.
|
||||
2. `image-gc-low-threshold`, the percent of disk usage to which image garbage collection attempts
|
||||
to free. Default is 80%.
|
||||
|
||||
We also allow users to customize garbage collection policy through the following kubelet flags:
|
||||
|
||||
1. `minimum-container-ttl-duration`, minimum age for a finished container before it is
|
||||
garbage collected. Default is 0 minute, which means every finished container will be garbage collected.
|
||||
2. `maximum-dead-containers-per-container`, maximum number of old instances to be retained
|
||||
per container. Default is 1.
|
||||
3. `maximum-dead-containers`, maximum number of old instances of containers to retain globally.
|
||||
Default is -1, which means there is no global limit.
|
||||
|
||||
Containers can potentially be garbage collected before their usefulness has expired. These containers
|
||||
can contain logs and other data that can be useful for troubleshooting. A sufficiently large value for
|
||||
`maximum-dead-containers-per-container` is highly recommended to allow at least 1 dead container to be
|
||||
retained per expected container. A larger value for `maximum-dead-containers` is also recommended for a
|
||||
similar reason.
|
||||
See [this issue](https://github.com/kubernetes/kubernetes/issues/13287) for more details.
|
||||
|
||||
|
||||
### Deprecation
|
||||
|
||||
Some kubelet Garbage Collection features in this doc will be replaced by kubelet eviction in the future.
|
||||
|
||||
Including:
|
||||
|
||||
| Existing Flag | New Flag | Rationale |
|
||||
| ------------- | -------- | --------- |
|
||||
| `--image-gc-high-threshold` | `--eviction-hard` or `--eviction-soft` | existing eviction signals can trigger image garbage collection |
|
||||
| `--image-gc-low-threshold` | `--eviction-minimum-reclaim` | eviction reclaims achieve the same behavior |
|
||||
| `--maximum-dead-containers` | | deprecated once old logs are stored outside of container's context |
|
||||
| `--maximum-dead-containers-per-container` | | deprecated once old logs are stored outside of container's context |
|
||||
| `--minimum-container-ttl-duration` | | deprecated once old logs are stored outside of container's context |
|
||||
| `--low-diskspace-threshold-mb` | `--eviction-hard` or `eviction-soft` | eviction generalizes disk thresholds to other resources |
|
||||
| `--outofdisk-transition-frequency` | `--eviction-pressure-transition-period` | eviction generalizes disk pressure transition to other resources |
|
||||
|
||||
See [Configuring Out Of Resource Handling](/docs/tasks/administer-cluster/out-of-resource/) for more details.
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
reviewers:
|
||||
- piosz
|
||||
- x13n
|
||||
title: Logging Architecture
|
||||
---
|
||||
|
||||
Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism; as such, most container engines are likewise designed to support some kind of logging. The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams.
|
||||
|
||||
However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution. For example, if a container crashes, a pod is evicted, or a node dies, you'll usually still want to access your application's logs. As such, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level-logging_. Cluster-level logging requires a separate backend to store, analyze, and query logs. Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster.
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
Cluster-level logging architectures are described in assumption that
|
||||
a logging backend is present inside or outside of your cluster. If you're
|
||||
not interested in having cluster-level logging, you might still find
|
||||
the description of how logs are stored and handled on the node to be useful.
|
||||
|
||||
## Basic logging in Kubernetes
|
||||
|
||||
In this section, you can see an example of basic logging in Kubernetes that
|
||||
outputs data to the standard output stream. This demonstration uses
|
||||
a [pod specification](/docs/concepts/cluster-administration/counter-pod.yaml) with
|
||||
a container that writes some text to standard output once per second.
|
||||
|
||||
{{< code file="counter-pod.yaml" >}}
|
||||
|
||||
To run this pod, use the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/counter-pod.yaml
|
||||
pod "counter" created
|
||||
```
|
||||
|
||||
To fetch the logs, use the `kubectl logs` command, as follows:
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter
|
||||
0: Mon Jan 1 00:00:00 UTC 2001
|
||||
1: Mon Jan 1 00:00:01 UTC 2001
|
||||
2: Mon Jan 1 00:00:02 UTC 2001
|
||||
...
|
||||
```
|
||||
|
||||
You can use `kubectl logs` to retrieve logs from a previous instantiation of a container with `--previous` flag, in case the container has crashed. If your pod has multiple containers, you should specify which container's logs you want to access by appending a container name to the command. See the [`kubectl logs` documentation](/docs/reference/generated/kubectl/kubectl-commands#logs) for more details.
|
||||
|
||||
## Logging at the node level
|
||||
|
||||
|
||||
|
||||
Everything a containerized application writes to `stdout` and `stderr` is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to [a logging driver](https://docs.docker.com/engine/admin/logging/overview), which is configured in Kubernetes to write to a file in json format.
|
||||
|
||||
**Note:** The Docker json logging driver treats each line as a separate message. When using the Docker logging driver, there is no direct support for multi-line messages. You need to handle multi-line messages at the logging agent level or higher.
|
||||
|
||||
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs.
|
||||
|
||||
An important consideration in node-level logging is implementing log rotation,
|
||||
so that logs don't consume all available storage on the node. Kubernetes
|
||||
currently is not responsible for rotating logs, but rather a deployment tool
|
||||
should set up a solution to address that.
|
||||
For example, in Kubernetes clusters, deployed by the `kube-up.sh` script,
|
||||
there is a [`logrotate`](https://linux.die.net/man/8/logrotate)
|
||||
tool configured to run each hour. You can also set up a container runtime to
|
||||
rotate application's logs automatically, e.g. by using Docker's `log-opt`.
|
||||
In the `kube-up.sh` script, the latter approach is used for COS image on GCP,
|
||||
and the former approach is used in any other environment. In both cases, by
|
||||
default rotation is configured to take place when log file exceeds 10MB.
|
||||
|
||||
As an example, you can find detailed information about how `kube-up.sh` sets
|
||||
up logging for COS image on GCP in the corresponding [script]
|
||||
[cosConfigureHelper].
|
||||
|
||||
When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
|
||||
the basic logging example, the kubelet on the node handles the request and
|
||||
reads directly from the log file, returning the contents in the response.
|
||||
|
||||
**Note:** currently, if some external system has performed the rotation,
|
||||
only the contents of the latest log file will be available through
|
||||
`kubectl logs`. E.g. if there's a 10MB file, `logrotate` performs
|
||||
the rotation and there are two files, one 10MB in size and one empty,
|
||||
`kubectl logs` will return an empty response.
|
||||
|
||||
[cosConfigureHelper]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh
|
||||
|
||||
### System component logs
|
||||
|
||||
There are two types of system components: those that run in a container and those
|
||||
that do not run in a container. For example:
|
||||
|
||||
* The Kubernetes scheduler and kube-proxy run in a container.
|
||||
* The kubelet and container runtime, for example Docker, do not run in containers.
|
||||
|
||||
On machines with systemd, the kubelet and container runtime write to journald. If
|
||||
systemd is not present, they write to `.log` files in the `/var/log` directory.
|
||||
System components inside containers always write to the `/var/log` directory,
|
||||
bypassing the default logging mechanism. They use the [glog][glog]
|
||||
logging library. You can find the conventions for logging severity for those
|
||||
components in the [development docs on logging](https://git.k8s.io/community/contributors/devel/logging.md).
|
||||
|
||||
Similarly to the container logs, system component logs in the `/var/log`
|
||||
directory should be rotated. In Kubernetes clusters brought up by
|
||||
the `kube-up.sh` script, those logs are configured to be rotated by
|
||||
the `logrotate` tool daily or once the size exceeds 100MB.
|
||||
|
||||
[glog]: https://godoc.org/github.com/golang/glog
|
||||
|
||||
## Cluster-level logging architectures
|
||||
|
||||
While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you can consider. Here are some options:
|
||||
|
||||
* Use a node-level logging agent that runs on every node.
|
||||
* Include a dedicated sidecar container for logging in an application pod.
|
||||
* Push logs directly to a backend from within an application.
|
||||
|
||||
### Using a node logging agent
|
||||
|
||||
|
||||
|
||||
You can implement cluster-level logging by including a _node-level logging agent_ on each node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node.
|
||||
|
||||
Because the logging agent must run on every node, it's common to implement it as either a DaemonSet replica, a manifest pod, or a dedicated native process on the node. However the latter two approaches are deprecated and highly discouraged.
|
||||
|
||||
Using a node-level logging agent is the most common and encouraged approach for a Kubernetes cluster, because it creates only one agent per node, and it doesn't require any changes to the applications running on the node. However, node-level logging _only works for applications' standard output and standard error_.
|
||||
|
||||
Kubernetes doesn't specify a logging agent, but two optional logging agents are packaged with the Kubernetes release: [Stackdriver Logging](/docs/user-guide/logging/stackdriver) for use with Google Cloud Platform, and [Elasticsearch](/docs/user-guide/logging/elasticsearch). You can find more information and instructions in the dedicated documents. Both use [fluentd](http://www.fluentd.org/) with custom configuration as an agent on the node.
|
||||
|
||||
### Using a sidecar container with the logging agent
|
||||
|
||||
You can use a sidecar container in one of the following ways:
|
||||
|
||||
* The sidecar container streams application logs to its own `stdout`.
|
||||
* The sidecar container runs a logging agent, which is configured to pick up logs from an application container.
|
||||
|
||||
#### Streaming sidecar container
|
||||
|
||||
|
||||
|
||||
By having your sidecar containers stream to their own `stdout` and `stderr`
|
||||
streams, you can take advantage of the kubelet and the logging agent that
|
||||
already run on each node. The sidecar containers read logs from a file, a socket,
|
||||
or the journald. Each individual sidecar container prints log to its own `stdout`
|
||||
or `stderr` stream.
|
||||
|
||||
This approach allows you to separate several log streams from different
|
||||
parts of your application, some of which can lack support
|
||||
for writing to `stdout` or `stderr`. The logic behind redirecting logs
|
||||
is minimal, so it's hardly a significant overhead. Additionally, because
|
||||
`stdout` and `stderr` are handled by the kubelet, you can use built-in tools
|
||||
like `kubectl logs`.
|
||||
|
||||
Consider the following example. A pod runs a single container, and the container
|
||||
writes to two different log files, using two different formats. Here's a
|
||||
configuration file for the Pod:
|
||||
|
||||
{{< code file="two-files-counter-pod.yaml" >}}
|
||||
|
||||
It would be a mess to have log entries of different formats in the same log
|
||||
stream, even if you managed to redirect both components to the `stdout` stream of
|
||||
the container. Instead, you could introduce two sidecar containers. Each sidecar
|
||||
container could tail a particular log file from a shared volume and then redirect
|
||||
the logs to its own `stdout` stream.
|
||||
|
||||
Here's a configuration file for a pod that has two sidecar containers:
|
||||
|
||||
{{< code file="two-files-counter-pod-streaming-sidecar.yaml" >}}
|
||||
|
||||
Now when you run this pod, you can access each log stream separately by
|
||||
running the following commands:
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter count-log-1
|
||||
0: Mon Jan 1 00:00:00 UTC 2001
|
||||
1: Mon Jan 1 00:00:01 UTC 2001
|
||||
2: Mon Jan 1 00:00:02 UTC 2001
|
||||
...
|
||||
```
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter count-log-2
|
||||
Mon Jan 1 00:00:00 UTC 2001 INFO 0
|
||||
Mon Jan 1 00:00:01 UTC 2001 INFO 1
|
||||
Mon Jan 1 00:00:02 UTC 2001 INFO 2
|
||||
...
|
||||
```
|
||||
|
||||
The node-level agent installed in your cluster picks up those log streams
|
||||
automatically without any further configuration. If you like, you can configure
|
||||
the agent to parse log lines depending on the source container.
|
||||
|
||||
Note, that despite low CPU and memory usage (order of couple of millicores
|
||||
for cpu and order of several megabytes for memory), writing logs to a file and
|
||||
then streaming them to `stdout` can double disk usage. If you have
|
||||
an application that writes to a single file, it's generally better to set
|
||||
`/dev/stdout` as destination rather than implementing the streaming sidecar
|
||||
container approach.
|
||||
|
||||
Sidecar containers can also be used to rotate log files that cannot be
|
||||
rotated by the application itself. [An example](https://github.com/samsung-cnct/logrotate)
|
||||
of this approach is a small container running logrotate periodically.
|
||||
However, it's recommended to use `stdout` and `stderr` directly and leave rotation
|
||||
and retention policies to the kubelet.
|
||||
|
||||
#### Sidecar container with a logging agent
|
||||
|
||||
|
||||
|
||||
If the node-level logging agent is not flexible enough for your situation, you
|
||||
can create a sidecar container with a separate logging agent that you have
|
||||
configured specifically to run with your application.
|
||||
|
||||
**Note**: Using a logging agent in a sidecar container can lead
|
||||
to significant resource consumption. Moreover, you won't be able to access
|
||||
those logs using `kubectl logs` command, because they are not controlled
|
||||
by the kubelet.
|
||||
|
||||
As an example, you could use [Stackdriver](/docs/tasks/debug-application-cluster/logging-stackdriver/),
|
||||
which uses fluentd as a logging agent. Here are two configuration files that
|
||||
you can use to implement this approach. The first file contains
|
||||
a [ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/) to configure fluentd.
|
||||
|
||||
{{< code file="fluentd-sidecar-config.yaml" >}}
|
||||
|
||||
**Note**: The configuration of fluentd is beyond the scope of this article. For
|
||||
information about configuring fluentd, see the
|
||||
[official fluentd documentation](http://docs.fluentd.org/).
|
||||
|
||||
The second file describes a pod that has a sidecar container running fluentd.
|
||||
The pod mounts a volume where fluentd can pick up its configuration data.
|
||||
|
||||
{{< code file="two-files-counter-pod-agent-sidecar.yaml" >}}
|
||||
|
||||
After some time you can find log messages in the Stackdriver interface.
|
||||
|
||||
Remember, that this is just an example and you can actually replace fluentd
|
||||
with any logging agent, reading from any source inside an application
|
||||
container.
|
||||
|
||||
### Exposing logs directly from the application
|
||||
|
||||
|
||||
|
||||
You can implement cluster-level logging by exposing or pushing logs directly from
|
||||
every application; however, the implementation for such a logging mechanism
|
||||
is outside the scope of Kubernetes.
|
||||
@@ -0,0 +1,396 @@
|
||||
---
|
||||
reviewers:
|
||||
- bgrant0607
|
||||
- janetkuo
|
||||
- mikedanese
|
||||
title: Managing Resources
|
||||
---
|
||||
|
||||
You've deployed your application and exposed it via a service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating. Among the features that we will discuss in more depth are [configuration files](/docs/concepts/configuration/overview/) and [labels](/docs/concepts/overview/working-with-objects/labels/).
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
## Organizing resource configurations
|
||||
|
||||
Many applications require multiple resources to be created, such as a Deployment and a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML). For example:
|
||||
|
||||
{{< code file="nginx-app.yaml" >}}
|
||||
|
||||
Multiple resources can be created the same way as a single resource:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/docs/concepts/cluster-administration/nginx-app.yaml
|
||||
service "my-nginx-svc" created
|
||||
deployment "my-nginx" created
|
||||
```
|
||||
|
||||
The resources will be created in the order they appear in the file. Therefore, it's best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment.
|
||||
|
||||
`kubectl create` also accepts multiple `-f` arguments:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/docs/concepts/cluster-administration/nginx/nginx-svc.yaml -f https://k8s.io/docs/concepts/cluster-administration/nginx/nginx-deployment.yaml
|
||||
```
|
||||
|
||||
And a directory can be specified rather than or in addition to individual files:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/docs/concepts/cluster-administration/nginx/
|
||||
```
|
||||
|
||||
`kubectl` will read any files with suffixes `.yaml`, `.yml`, or `.json`.
|
||||
|
||||
It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, then you can then simply deploy all of the components of your stack en masse.
|
||||
|
||||
A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into github:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://raw.githubusercontent.com/kubernetes/website/master/docs/concepts/cluster-administration/nginx-deployment.yaml
|
||||
deployment "nginx-deployment" created
|
||||
```
|
||||
|
||||
## Bulk operations in kubectl
|
||||
|
||||
Resource creation isn't the only operation that `kubectl` can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created:
|
||||
|
||||
```shell
|
||||
$ kubectl delete -f https://k8s.io/docs/concepts/cluster-administration/nginx-app.yaml
|
||||
deployment "my-nginx" deleted
|
||||
service "my-nginx-svc" deleted
|
||||
```
|
||||
|
||||
In the case of just two resources, it's also easy to specify both on the command line using the resource/name syntax:
|
||||
|
||||
```shell
|
||||
$ kubectl delete deployments/my-nginx services/my-nginx-svc
|
||||
```
|
||||
|
||||
For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using `-l` or `--selector`, to filter resources by their labels:
|
||||
|
||||
```shell
|
||||
$ kubectl delete deployment,services -l app=nginx
|
||||
deployment "my-nginx" deleted
|
||||
service "my-nginx-svc" deleted
|
||||
```
|
||||
|
||||
Because `kubectl` outputs resource names in the same syntax it accepts, it's easy to chain operations using `$()` or `xargs`:
|
||||
|
||||
```shell
|
||||
$ kubectl get $(kubectl create -f docs/concepts/cluster-administration/nginx/ -o name | grep service)
|
||||
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
my-nginx-svc 10.0.0.208 <pending> 80/TCP 0s
|
||||
```
|
||||
|
||||
With the above commands, we first create resources under `docs/concepts/cluster-administration/nginx/` and print the resources created with `-o name` output format
|
||||
(print each resource as resource/name). Then we `grep` only the "service", and then print it with `kubectl get`.
|
||||
|
||||
If you happen to organize your resources across several subdirectories within a particular directory, you can recursively perform the operations on the subdirectories also, by specifying `--recursive` or `-R` alongside the `--filename,-f` flag.
|
||||
|
||||
For instance, assume there is a directory `project/k8s/development` that holds all of the manifests needed for the development environment, organized by resource type:
|
||||
|
||||
```
|
||||
project/k8s/development
|
||||
├── configmap
|
||||
│ └── my-configmap.yaml
|
||||
├── deployment
|
||||
│ └── my-deployment.yaml
|
||||
└── pvc
|
||||
└── my-pvc.yaml
|
||||
```
|
||||
|
||||
By default, performing a bulk operation on `project/k8s/development` will stop at the first level of the directory, not processing any subdirectories. If we had tried to create the resources in this directory using the following command, we would have encountered an error:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/development
|
||||
error: you must provide one or more resources by argument or filename (.json|.yaml|.yml|stdin)
|
||||
```
|
||||
|
||||
Instead, specify the `--recursive` or `-R` flag with the `--filename,-f` flag as such:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/development --recursive
|
||||
configmap "my-config" created
|
||||
deployment "my-deployment" created
|
||||
persistentvolumeclaim "my-pvc" created
|
||||
```
|
||||
|
||||
The `--recursive` flag works with any operation that accepts the `--filename,-f` flag such as: `kubectl {create,get,delete,describe,rollout} etc.`
|
||||
|
||||
The `--recursive` flag also works when multiple `-f` arguments are provided:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive
|
||||
namespace "development" created
|
||||
namespace "staging" created
|
||||
configmap "my-config" created
|
||||
deployment "my-deployment" created
|
||||
persistentvolumeclaim "my-pvc" created
|
||||
```
|
||||
|
||||
If you're interested in learning more about `kubectl`, go ahead and read [kubectl Overview](/docs/reference/kubectl/overview/).
|
||||
|
||||
## Using labels effectively
|
||||
|
||||
The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
|
||||
|
||||
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: frontend
|
||||
```
|
||||
|
||||
while the Redis master and slave would have different `tier` labels, and perhaps even an additional `role` label:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: backend
|
||||
role: master
|
||||
```
|
||||
|
||||
and
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: backend
|
||||
role: slave
|
||||
```
|
||||
|
||||
The labels allow us to slice and dice our resources along any dimension specified by a label:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml
|
||||
$ kubectl get pods -Lapp -Ltier -Lrole
|
||||
NAME READY STATUS RESTARTS AGE APP TIER ROLE
|
||||
guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend <none>
|
||||
guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend <none>
|
||||
guestbook-fe-jpy62 1/1 Running 0 1m guestbook frontend <none>
|
||||
guestbook-redis-master-5pg3b 1/1 Running 0 1m guestbook backend master
|
||||
guestbook-redis-slave-2q2yf 1/1 Running 0 1m guestbook backend slave
|
||||
guestbook-redis-slave-qgazl 1/1 Running 0 1m guestbook backend slave
|
||||
my-nginx-divi2 1/1 Running 0 29m nginx <none> <none>
|
||||
my-nginx-o0ef1 1/1 Running 0 29m nginx <none> <none>
|
||||
$ kubectl get pods -lapp=guestbook,role=slave
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
guestbook-redis-slave-2q2yf 1/1 Running 0 3m
|
||||
guestbook-redis-slave-qgazl 1/1 Running 0 3m
|
||||
```
|
||||
|
||||
## Canary deployments
|
||||
|
||||
Another scenario where multiple labels are needed is to distinguish deployments of different releases or configurations of the same component. It is common practice to deploy a *canary* of a new application release (specified via image tag in the pod template) side by side with the previous release so that the new release can receive live production traffic before fully rolling it out.
|
||||
|
||||
For instance, you can use a `track` label to differentiate different releases.
|
||||
|
||||
The primary, stable release would have a `track` label with value as `stable`:
|
||||
|
||||
```yaml
|
||||
name: frontend
|
||||
replicas: 3
|
||||
...
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: frontend
|
||||
track: stable
|
||||
...
|
||||
image: gb-frontend:v3
|
||||
```
|
||||
|
||||
and then you can create a new release of the guestbook frontend that carries the `track` label with different value (i.e. `canary`), so that two sets of pods would not overlap:
|
||||
|
||||
```yaml
|
||||
name: frontend-canary
|
||||
replicas: 1
|
||||
...
|
||||
labels:
|
||||
app: guestbook
|
||||
tier: frontend
|
||||
track: canary
|
||||
...
|
||||
image: gb-frontend:v4
|
||||
```
|
||||
|
||||
|
||||
The frontend service would span both sets of replicas by selecting the common subset of their labels (i.e. omitting the `track` label), so that the traffic will be redirected to both applications:
|
||||
|
||||
```yaml
|
||||
selector:
|
||||
app: guestbook
|
||||
tier: frontend
|
||||
```
|
||||
|
||||
You can tweak the number of replicas of the stable and canary releases to determine the ratio of each release that will receive live production traffic (in this case, 3:1).
|
||||
Once you're confident, you can update the stable track to the new application release and remove the canary one.
|
||||
|
||||
For a more concrete example, check the [tutorial of deploying Ghost](https://github.com/kelseyhightower/talks/tree/master/kubecon-eu-2016/demo#deploy-a-canary).
|
||||
|
||||
## Updating labels
|
||||
|
||||
Sometimes existing pods and other resources need to be relabeled before creating new resources. This can be done with `kubectl label`.
|
||||
For example, if you want to label all your nginx pods as frontend tier, simply run:
|
||||
|
||||
```shell
|
||||
$ kubectl label pods -l app=nginx tier=fe
|
||||
pod "my-nginx-2035384211-j5fhi" labeled
|
||||
pod "my-nginx-2035384211-u2c7e" labeled
|
||||
pod "my-nginx-2035384211-u3t6x" labeled
|
||||
```
|
||||
|
||||
This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe".
|
||||
To see the pods you just labeled, run:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l app=nginx -L tier
|
||||
NAME READY STATUS RESTARTS AGE TIER
|
||||
my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe
|
||||
my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe
|
||||
my-nginx-2035384211-u3t6x 1/1 Running 0 23m fe
|
||||
```
|
||||
|
||||
This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with `-L` or `--label-columns`).
|
||||
|
||||
For more information, please see [labels](/docs/concepts/overview/working-with-objects/labels/) and [kubectl label](/docs/reference/generated/kubectl/kubectl-commands/#label).
|
||||
|
||||
## Updating annotations
|
||||
|
||||
Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with `kubectl annotate`. For example:
|
||||
|
||||
```shell
|
||||
$ kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx'
|
||||
$ kubectl get pods my-nginx-v4-9gw19 -o yaml
|
||||
apiversion: v1
|
||||
kind: pod
|
||||
metadata:
|
||||
annotations:
|
||||
description: my frontend running nginx
|
||||
...
|
||||
```
|
||||
|
||||
For more information, please see [annotations](/docs/concepts/overview/working-with-objects/annotations/) and [kubectl annotate](/docs/reference/generated/kubectl/kubectl-commands/#annotate) document.
|
||||
|
||||
## Scaling your application
|
||||
|
||||
When load on your application grows or shrinks, it's easy to scale with `kubectl`. For instance, to decrease the number of nginx replicas from 3 to 1, do:
|
||||
|
||||
```shell
|
||||
$ kubectl scale deployment/my-nginx --replicas=1
|
||||
deployment "my-nginx" scaled
|
||||
```
|
||||
|
||||
Now you only have one pod managed by the deployment.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l app=nginx
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
my-nginx-2035384211-j5fhi 1/1 Running 0 30m
|
||||
```
|
||||
|
||||
To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do:
|
||||
|
||||
```shell
|
||||
$ kubectl autoscale deployment/my-nginx --min=1 --max=3
|
||||
deployment "my-nginx" autoscaled
|
||||
```
|
||||
|
||||
Now your nginx replicas will be scaled up and down as needed, automatically.
|
||||
|
||||
For more information, please see [kubectl scale](/docs/reference/generated/kubectl/kubectl-commands/#scale), [kubectl autoscale](/docs/reference/generated/kubectl/kubectl-commands/#autoscale) and [horizontal pod autoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) document.
|
||||
|
||||
|
||||
## In-place updates of resources
|
||||
|
||||
Sometimes it's necessary to make narrow, non-disruptive updates to resources you've created.
|
||||
|
||||
### kubectl apply
|
||||
|
||||
It is suggested to maintain a set of configuration files in source control (see [configuration as code](http://martinfowler.com/bliki/InfrastructureAsCode.html)),
|
||||
so that they can be maintained and versioned along with the code for the resources they configure.
|
||||
Then, you can use [`kubectl apply`](/docs/reference/generated/kubectl/kubectl-commands/#apply) to push your configuration changes to the cluster.
|
||||
|
||||
This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified.
|
||||
|
||||
```shell
|
||||
$ kubectl apply -f docs/concepts/cluster-administration/nginx/nginx-deployment.yaml
|
||||
deployment "my-nginx" configured
|
||||
```
|
||||
|
||||
Note that `kubectl apply` attaches an annotation to the resource in order to determine the changes to the configuration since the previous invocation. When it's invoked, `kubectl apply` does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource.
|
||||
|
||||
Currently, resources are created without this annotation, so the first invocation of `kubectl apply` will fall back to a two-way diff between the provided input and the current configuration of the resource. During this first invocation, it cannot detect the deletion of properties set when the resource was created. For this reason, it will not remove them.
|
||||
|
||||
All subsequent calls to `kubectl apply`, and other commands that modify the configuration, such as `kubectl replace` and `kubectl edit`, will update the annotation, allowing subsequent calls to `kubectl apply` to detect and perform deletions using a three-way diff.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** To use apply, always create resource initially with either `kubectl apply` or `kubectl create --save-config`.
|
||||
{{< /note >}}
|
||||
|
||||
### kubectl edit
|
||||
|
||||
Alternatively, you may also update resources with `kubectl edit`:
|
||||
|
||||
```shell
|
||||
$ kubectl edit deployment/my-nginx
|
||||
```
|
||||
|
||||
This is equivalent to first `get` the resource, edit it in text editor, and then `apply` the resource with the updated version:
|
||||
|
||||
```shell
|
||||
$ kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml
|
||||
$ vi /tmp/nginx.yaml
|
||||
# do some edit, and then save the file
|
||||
$ kubectl apply -f /tmp/nginx.yaml
|
||||
deployment "my-nginx" configured
|
||||
$ rm /tmp/nginx.yaml
|
||||
```
|
||||
|
||||
This allows you to do more significant changes more easily. Note that you can specify the editor with your `EDITOR` or `KUBE_EDITOR` environment variables.
|
||||
|
||||
For more information, please see [kubectl edit](/docs/reference/generated/kubectl/kubectl-commands/#edit) document.
|
||||
|
||||
### kubectl patch
|
||||
|
||||
You can use `kubectl patch` to update API objects in place. This command supports JSON patch,
|
||||
JSON merge patch, and strategic merge patch. See
|
||||
[Update API Objects in Place Using kubectl patch](/docs/tasks/run-application/update-api-object-kubectl-patch/)
|
||||
and
|
||||
[kubectl patch](/docs/reference/generated/kubectl/kubectl-commands/#patch).
|
||||
|
||||
## Disruptive updates
|
||||
|
||||
In some cases, you may need to update resource fields that cannot be updated once initialized, or you may just want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use `replace --force`, which deletes and re-creates the resource. In this case, you can simply modify your original configuration file:
|
||||
|
||||
```shell
|
||||
$ kubectl replace -f docs/concepts/cluster-administration/nginx/nginx-deployment.yaml --force
|
||||
deployment "my-nginx" deleted
|
||||
deployment "my-nginx" replaced
|
||||
```
|
||||
|
||||
## Updating your application without a service outage
|
||||
|
||||
At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios.
|
||||
|
||||
We'll guide you through how to create and update applications with Deployments. If your deployed application is managed by Replication Controllers,
|
||||
you should read [how to use `kubectl rolling-update`](/docs/tasks/run-application/rolling-update-replication-controller/) instead.
|
||||
|
||||
Let's say you were running version 1.7.9 of nginx:
|
||||
|
||||
```shell
|
||||
$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3
|
||||
deployment "my-nginx" created
|
||||
```
|
||||
|
||||
To update to version 1.9.1, simply change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`, with the kubectl commands we learned above.
|
||||
|
||||
```shell
|
||||
$ kubectl edit deployment/my-nginx
|
||||
```
|
||||
|
||||
That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit [Deployment page](/docs/concepts/workloads/controllers/deployment/).
|
||||
|
||||
## What's next?
|
||||
|
||||
- [Learn about how to use `kubectl` for application introspection and debugging.](/docs/tasks/debug-application-cluster/debug-application-introspection/)
|
||||
- [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/)
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
reviewers:
|
||||
- dcbw
|
||||
- freehan
|
||||
- thockin
|
||||
title: Network Plugins
|
||||
---
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
__Disclaimer__: Network plugins are in alpha. Its contents will change rapidly.
|
||||
|
||||
Network plugins in Kubernetes come in a few flavors:
|
||||
|
||||
* CNI plugins: adhere to the appc/CNI specification, designed for interoperability.
|
||||
* Kubenet plugin: implements basic `cbr0` using the `bridge` and `host-local` CNI plugins
|
||||
|
||||
## Installation
|
||||
|
||||
The kubelet has a single default network plugin, and a default network common to the entire cluster. It probes for plugins when it starts up, remembers what it found, and executes the selected plugin at appropriate times in the pod lifecycle (this is only true for Docker, as rkt manages its own CNI plugins). There are two Kubelet command line parameters to keep in mind when using plugins:
|
||||
|
||||
* `cni-bin-dir`: Kubelet probes this directory for plugins on startup
|
||||
* `network-plugin`: The network plugin to use from `cni-bin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is simply "cni".
|
||||
|
||||
## Network Plugin Requirements
|
||||
|
||||
Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy.
|
||||
|
||||
By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy.
|
||||
|
||||
### CNI
|
||||
|
||||
The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`).
|
||||
|
||||
If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used.
|
||||
|
||||
In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0
|
||||
|
||||
Limitation: Due to [#31307](https://github.com/kubernetes/kubernetes/issues/31307), `HostPort` won't work with CNI networking plugin at the moment. That means all `hostPort` attribute in pod would be simply ignored.
|
||||
|
||||
### kubenet
|
||||
|
||||
Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.
|
||||
|
||||
Kubenet creates a Linux bridge named `cbr0` and creates a veth pair for each pod with the host end of each pair connected to `cbr0`. The pod end of the pair is assigned an IP address allocated from a range assigned to the node either through configuration or by the controller-manager. `cbr0` is assigned an MTU matching the smallest MTU of an enabled normal interface on the host.
|
||||
|
||||
The plugin requires a few things:
|
||||
|
||||
* The standard CNI `bridge`, `lo` and `host-local` plugins are required, at minimum version 0.2.0. Kubenet will first search for them in `/opt/cni/bin`. Specify `cni-bin-dir` to supply additional search path. The first found match will take effect.
|
||||
* Kubelet must be run with the `--network-plugin=kubenet` argument to enable the plugin
|
||||
* Kubelet should also be run with the `--non-masquerade-cidr=<clusterCidr>` argument to ensure traffic to IPs outside this range will use IP masquerade.
|
||||
* The node must be assigned an IP subnet through either the `--pod-cidr` kubelet command-line option or the `--allocate-node-cidrs=true --cluster-cidr=<cidr>` controller-manager command-line options.
|
||||
|
||||
### Customizing the MTU (with kubenet)
|
||||
|
||||
The MTU should always be configured correctly to get the best networking performance. Network plugins will usually try
|
||||
to infer a sensible MTU, but sometimes the logic will not result in an optimal MTU. For example, if the
|
||||
Docker bridge or another interface has a small MTU, kubenet will currently select that MTU. Or if you are
|
||||
using IPSEC encapsulation, the MTU must be reduced, and this calculation is out-of-scope for
|
||||
most network plugins.
|
||||
|
||||
Where needed, you can specify the MTU explicitly with the `network-plugin-mtu` kubelet option. For example,
|
||||
on AWS the `eth0` MTU is typically 9001, so you might specify `--network-plugin-mtu=9001`. If you're using IPSEC you
|
||||
might reduce it to allow for encapsulation overhead e.g. `--network-plugin-mtu=8873`.
|
||||
|
||||
This option is provided to the network-plugin; currently **only kubenet supports `network-plugin-mtu`**.
|
||||
|
||||
## Usage Summary
|
||||
|
||||
* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`).
|
||||
* `--network-plugin=kubenet` specifies that we use the `kubenet` network plugin with CNI `bridge` and `host-local` plugins placed in `/opt/cni/bin` or `cni-bin-dir`.
|
||||
* `--network-plugin-mtu=9001` specifies the MTU to use, currently only used by the `kubenet` network plugin.
|
||||
@@ -0,0 +1,260 @@
|
||||
---
|
||||
reviewers:
|
||||
- thockin
|
||||
title: Cluster Networking
|
||||
---
|
||||
|
||||
Kubernetes approaches networking somewhat differently than Docker does by
|
||||
default. There are 4 distinct networking problems to solve:
|
||||
|
||||
1. Highly-coupled container-to-container communications: this is solved by
|
||||
[pods](/docs/concepts/workloads/pods/pod/) and `localhost` communications.
|
||||
2. Pod-to-Pod communications: this is the primary focus of this document.
|
||||
3. Pod-to-Service communications: this is covered by [services](/docs/concepts/services-networking/service/).
|
||||
4. External-to-Service communications: this is covered by [services](/docs/concepts/services-networking/service/).
|
||||
|
||||
{{< toc >}}
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Kubernetes assumes that pods can communicate with other pods, regardless of
|
||||
which host they land on. Every pod gets its own IP address so you do not
|
||||
need to explicitly create links between pods and you almost never need to deal
|
||||
with mapping container ports to host ports. This creates a clean,
|
||||
backwards-compatible model where pods can be treated much like VMs or physical
|
||||
hosts from the perspectives of port allocation, naming, service discovery, load
|
||||
balancing, application configuration, and migration.
|
||||
|
||||
There are requirements imposed on how you set up your cluster networking to
|
||||
achieve this.
|
||||
|
||||
## Docker model
|
||||
|
||||
Before discussing the Kubernetes approach to networking, it is worthwhile to
|
||||
review the "normal" way that networking works with Docker. By default, Docker
|
||||
uses host-private networking. It creates a virtual bridge, called `docker0` by
|
||||
default, and allocates a subnet from one of the private address blocks defined
|
||||
in [RFC1918](https://tools.ietf.org/html/rfc1918) for that bridge. For each
|
||||
container that Docker creates, it allocates a virtual Ethernet device (called
|
||||
`veth`) which is attached to the bridge. The veth is mapped to appear as `eth0`
|
||||
in the container, using Linux namespaces. The in-container `eth0` interface is
|
||||
given an IP address from the bridge's address range.
|
||||
|
||||
The result is that Docker containers can talk to other containers only if they
|
||||
are on the same machine (and thus the same virtual bridge). Containers on
|
||||
different machines can not reach each other - in fact they may end up with the
|
||||
exact same network ranges and IP addresses.
|
||||
|
||||
In order for Docker containers to communicate across nodes, there must
|
||||
be allocated ports on the machine’s own IP address, which are then
|
||||
forwarded or proxied to the containers. This obviously means that
|
||||
containers must either coordinate which ports they use very carefully
|
||||
or ports must be allocated dynamically.
|
||||
|
||||
## Kubernetes model
|
||||
|
||||
Coordinating ports across multiple developers is very difficult to do at
|
||||
scale and exposes users to cluster-level issues outside of their control.
|
||||
Dynamic port allocation brings a lot of complications to the system - every
|
||||
application has to take ports as flags, the API servers have to know how to
|
||||
insert dynamic port numbers into configuration blocks, services have to know
|
||||
how to find each other, etc. Rather than deal with this, Kubernetes takes a
|
||||
different approach.
|
||||
|
||||
Kubernetes imposes the following fundamental requirements on any networking
|
||||
implementation (barring any intentional network segmentation policies):
|
||||
|
||||
* all containers can communicate with all other containers without NAT
|
||||
* all nodes can communicate with all containers (and vice-versa) without NAT
|
||||
* the IP that a container sees itself as is the same IP that others see it as
|
||||
|
||||
What this means in practice is that you can not just take two computers
|
||||
running Docker and expect Kubernetes to work. You must ensure that the
|
||||
fundamental requirements are met.
|
||||
|
||||
This model is not only less complex overall, but it is principally compatible
|
||||
with the desire for Kubernetes to enable low-friction porting of apps from VMs
|
||||
to containers. If your job previously ran in a VM, your VM had an IP and could
|
||||
talk to other VMs in your project. This is the same basic model.
|
||||
|
||||
Until now this document has talked about containers. In reality, Kubernetes
|
||||
applies IP addresses at the `Pod` scope - containers within a `Pod` share their
|
||||
network namespaces - including their IP address. This means that containers
|
||||
within a `Pod` can all reach each other's ports on `localhost`. This does imply
|
||||
that containers within a `Pod` must coordinate port usage, but this is no
|
||||
different than processes in a VM. This is called the "IP-per-pod" model. This
|
||||
is implemented, using Docker, as a "pod container" which holds the network namespace
|
||||
open while "app containers" (the things the user specified) join that namespace
|
||||
with Docker's `--net=container:<id>` function.
|
||||
|
||||
As with Docker, it is possible to request host ports, but this is reduced to a
|
||||
very niche operation. In this case a port will be allocated on the host `Node`
|
||||
and traffic will be forwarded to the `Pod`. The `Pod` itself is blind to the
|
||||
existence or non-existence of host ports.
|
||||
|
||||
## How to achieve this
|
||||
|
||||
There are a number of ways that this network model can be implemented. This
|
||||
document is not an exhaustive study of the various methods, but hopefully serves
|
||||
as an introduction to various technologies and serves as a jumping-off point.
|
||||
|
||||
The following networking options are sorted alphabetically - the order does not
|
||||
imply any preferential status.
|
||||
|
||||
### ACI
|
||||
|
||||
[Cisco Application Centric Infrastructure](https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html) offers an integrated overlay and underlay SDN solution that supports containers, virtual machines, and bare metal servers. [ACI](https://www.github.com/noironetworks/aci-containers) provides container networking integration for ACI. An overview of the integration is provided [here](https://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/solution-overview-c22-739493.pdf).
|
||||
|
||||
### Big Cloud Fabric from Big Switch Networks
|
||||
|
||||
[Big Cloud Fabric](https://www.bigswitch.com/container-network-automation) is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premise environments. Using unified physical & virtual SDN, Big Cloud Fabric tackles inherent container networking problems such as load balancing, visibility, troubleshooting, security policies & container traffic monitoring.
|
||||
|
||||
With the help of the Big Cloud Fabric's virtual pod multi-tenant architecture, container orchestration systems such as Kubernetes, RedHat Openshift, Mesosphere DC/OS & Docker Swarm will be natively integrated along side with VM orchestration systems such as VMware, OpenStack & Nutanix. Customers will be able to securely inter-connect any number of these clusters and enable inter-tenant communication between them if needed.
|
||||
|
||||
BCF was recognized by Gartner as a visionary in the latest [Magic Quadrant](http://go.bigswitch.com/17GatedDocuments-MagicQuadrantforDataCenterNetworking_Reg.html). One of the BCF Kubernetes on premise deployments (which includes Kubernetes, DC/OS & VMware running on multiple DCs across different geographic regions) is also referenced [here](https://portworx.com/architects-corner-kubernetes-satya-komala-nio/).
|
||||
|
||||
### Cilium
|
||||
|
||||
[Cilium](https://github.com/cilium/cilium) is open source software for
|
||||
providing and transparently securing network connectivity between application
|
||||
containers. Cilium is L7/HTTP aware and can enforce network policies on L3-L7
|
||||
using an identity based security model that is decoupled from network
|
||||
addressing.
|
||||
|
||||
### Contiv
|
||||
|
||||
[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced.
|
||||
|
||||
### Contrail
|
||||
|
||||
[Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [OpenContrail](http://www.opencontrail.org), is a truly open, multi-cloud network virtualization and policy management platform. Contrail / OpenContrail is integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provides different isolation modes for virtual machines, containers/pods and bare metal workloads.
|
||||
|
||||
### Flannel
|
||||
|
||||
[Flannel](https://github.com/coreos/flannel#flannel) is a very simple overlay
|
||||
network that satisfies the Kubernetes requirements. Many
|
||||
people have reported success with Flannel and Kubernetes.
|
||||
|
||||
### Google Compute Engine (GCE)
|
||||
|
||||
For the Google Compute Engine cluster configuration scripts, [advanced
|
||||
routing](https://cloud.google.com/vpc/docs/routes) is used to
|
||||
assign each VM a subnet (default is `/24` - 254 IPs). Any traffic bound for that
|
||||
subnet will be routed directly to the VM by the GCE network fabric. This is in
|
||||
addition to the "main" IP address assigned to the VM, which is NAT'ed for
|
||||
outbound internet access. A linux bridge (called `cbr0`) is configured to exist
|
||||
on that subnet, and is passed to docker's `--bridge` flag.
|
||||
|
||||
Docker is started with:
|
||||
|
||||
```shell
|
||||
DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false"
|
||||
```
|
||||
|
||||
This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet`
|
||||
flag) according to the `Node`'s `spec.podCIDR`.
|
||||
|
||||
Docker will now allocate IPs from the `cbr-cidr` block. Containers can reach
|
||||
each other and `Nodes` over the `cbr0` bridge. Those IPs are all routable
|
||||
within the GCE project network.
|
||||
|
||||
GCE itself does not know anything about these IPs, though, so it will not NAT
|
||||
them for outbound internet traffic. To achieve that an iptables rule is used
|
||||
to masquerade (aka SNAT - to make it seem as if packets came from the `Node`
|
||||
itself) traffic that is bound for IPs outside the GCE project network
|
||||
(10.0.0.0/8).
|
||||
|
||||
```shell
|
||||
iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE
|
||||
```
|
||||
|
||||
Lastly IP forwarding is enabled in the kernel (so the kernel will process
|
||||
packets for bridged containers):
|
||||
|
||||
```shell
|
||||
sysctl net.ipv4.ip_forward=1
|
||||
```
|
||||
|
||||
The result of all this is that all `Pods` can reach each other and can egress
|
||||
traffic to the internet.
|
||||
|
||||
### Kube-router
|
||||
|
||||
[Kube-router](https://github.com/cloudnativelabs/kube-router) is a purpose-built networking solution for Kubernetes that aims to provide high performance and operational simplicity. Kube-router provides a Linux [LVS/IPVS](http://www.linuxvirtualserver.org/software/ipvs.html)-based service proxy, a Linux kernel forwarding-based pod-to-pod networking solution with no overlays, and iptables/ipset-based network policy enforcer.
|
||||
|
||||
### L2 networks and linux bridging
|
||||
|
||||
If you have a "dumb" L2 network, such as a simple switch in a "bare-metal"
|
||||
environment, you should be able to do something similar to the above GCE setup.
|
||||
Note that these instructions have only been tried very casually - it seems to
|
||||
work, but has not been thoroughly tested. If you use this technique and
|
||||
perfect the process, please let us know.
|
||||
|
||||
Follow the "With Linux Bridge devices" section of [this very nice
|
||||
tutorial](http://blog.oddbit.com/2014/08/11/four-ways-to-connect-a-docker/) from
|
||||
Lars Kellogg-Stedman.
|
||||
|
||||
### Multus (a Multi Network plugin)
|
||||
|
||||
[Multus](https://github.com/Intel-Corp/multus-cni) is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.
|
||||
|
||||
Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.
|
||||
|
||||
### NSX-T
|
||||
|
||||
[VMware NSX-T](https://docs.vmware.com/en/VMware-NSX-T/index.html) is a network virtualization and security platform. NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks. In addition to vSphere hypervisors, these environments include other hypervisors such as KVM, containers, and bare metal.
|
||||
|
||||
[NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and Openshift.
|
||||
|
||||
### Nuage Networks VCS (Virtualized Cloud Services)
|
||||
|
||||
[Nuage](http://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.
|
||||
|
||||
The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications.
|
||||
|
||||
### OpenVSwitch
|
||||
|
||||
[OpenVSwitch](https://www.openvswitch.org/) is a somewhat more mature but also
|
||||
complicated way to build an overlay network. This is endorsed by several of the
|
||||
"Big Shops" for networking.
|
||||
|
||||
### OVN (Open Virtual Networking)
|
||||
|
||||
OVN is an opensource network virtualization solution developed by the
|
||||
Open vSwitch community. It lets one create logical switches, logical routers,
|
||||
stateful ACLs, load-balancers etc to build different virtual networking
|
||||
topologies. The project has a specific Kubernetes plugin and documentation
|
||||
at [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes).
|
||||
|
||||
### Project Calico
|
||||
|
||||
[Project Calico](http://docs.projectcalico.org/) is an open source container networking provider and network policy engine.
|
||||
|
||||
Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall.
|
||||
|
||||
Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka [canal](https://github.com/tigera/canal), or native GCE networking.
|
||||
|
||||
### Romana
|
||||
|
||||
[Romana](http://romana.io) is an open source network and security automation solution that lets you deploy Kubernetes without an overlay network. Romana supports Kubernetes [Network Policy](/docs/concepts/services-networking/network-policies/) to provide isolation across network namespaces.
|
||||
|
||||
### Weave Net from Weaveworks
|
||||
|
||||
[Weave Net](https://www.weave.works/products/weave-net/) is a
|
||||
resilient and simple to use network for Kubernetes and its hosted applications.
|
||||
Weave Net runs as a [CNI plug-in](https://www.weave.works/docs/net/latest/cni-plugin/)
|
||||
or stand-alone. In either version, it doesn't require any configuration or extra code
|
||||
to run, and in both cases, the network provides one IP address per pod - as is standard for Kubernetes.
|
||||
|
||||
### CNI-Genie from Huawei
|
||||
|
||||
[CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) is a CNI plugin that enables Kubernetes to [simultaneously have access to different implementations](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-cni-plugins/README.md#what-cni-genie-feature-1-multiple-cni-plugins-enables) of the [Kubernetes network model](https://git.k8s.io/website/docs/concepts/cluster-administration/networking.md#kubernetes-model) in runtime. This includes any implementation that runs as a [CNI plugin](https://github.com/containernetworking/cni#3rd-party-plugins), such as [Flannel](https://github.com/coreos/flannel#flannel), [Calico](http://docs.projectcalico.org/), [Romana](http://romana.io), [Weave-net](https://www.weave.works/products/weave-net/).
|
||||
|
||||
CNI-Genie also supports [assigning multiple IP addresses to a pod](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-ips/README.md#feature-2-extension-cni-genie-multiple-ip-addresses-per-pod), each from a different CNI plugin.
|
||||
|
||||
## Other reading
|
||||
|
||||
The early design of the networking model and its rationale, and some future
|
||||
plans are described in more detail in the [networking design
|
||||
document](https://git.k8s.io/community/contributors/design-proposals/network/networking.md).
|
||||
@@ -0,0 +1,34 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: my-nginx-svc
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
ports:
|
||||
- port: 80
|
||||
selector:
|
||||
app: nginx
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: my-nginx
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:1.7.9
|
||||
ports:
|
||||
- containerPort: 80
|
||||
@@ -0,0 +1,19 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nginx-deployment
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx
|
||||
replicas: 3
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:1.7.9
|
||||
ports:
|
||||
- containerPort: 80
|
||||
@@ -0,0 +1,19 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: my-nginx
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nginx
|
||||
replicas: 3
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:1.7.9
|
||||
ports:
|
||||
- containerPort: 80
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: my-nginx-svc
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
ports:
|
||||
- port: 80
|
||||
selector:
|
||||
app: nginx
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: Proxies in Kubernetes
|
||||
content_template: templates/concept
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
This page explains proxies used with Kubernetes.
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Proxies
|
||||
|
||||
There are several different proxies you may encounter when using Kubernetes:
|
||||
|
||||
1. The [kubectl proxy](/docs/tasks/access-application-cluster/access-cluster/#directly-accessing-the-rest-api):
|
||||
|
||||
- runs on a user's desktop or in a pod
|
||||
- proxies from a localhost address to the Kubernetes apiserver
|
||||
- client to proxy uses HTTP
|
||||
- proxy to apiserver uses HTTPS
|
||||
- locates apiserver
|
||||
- adds authentication headers
|
||||
|
||||
1. The [apiserver proxy](/docs/tasks/access-application-cluster/access-cluster/#discovering-builtin-services):
|
||||
|
||||
- is a bastion built into the apiserver
|
||||
- connects a user outside of the cluster to cluster IPs which otherwise might not be reachable
|
||||
- runs in the apiserver processes
|
||||
- client to proxy uses HTTPS (or http if apiserver so configured)
|
||||
- proxy to target may use HTTP or HTTPS as chosen by proxy using available information
|
||||
- can be used to reach a Node, Pod, or Service
|
||||
- does load balancing when used to reach a Service
|
||||
|
||||
1. The [kube proxy](/docs/concepts/services-networking/service/#ips-and-vips):
|
||||
|
||||
- runs on each node
|
||||
- proxies UDP and TCP
|
||||
- does not understand HTTP
|
||||
- provides load balancing
|
||||
- is just used to reach services
|
||||
|
||||
1. A Proxy/Load-balancer in front of apiserver(s):
|
||||
|
||||
- existence and implementation varies from cluster to cluster (e.g. nginx)
|
||||
- sits between all clients and one or more apiservers
|
||||
- acts as load balancer if there are several apiservers.
|
||||
|
||||
1. Cloud Load Balancers on external services:
|
||||
|
||||
- are provided by some cloud providers (e.g. AWS ELB, Google Cloud Load Balancer)
|
||||
- are created automatically when the Kubernetes service has type `LoadBalancer`
|
||||
- use UDP/TCP only
|
||||
- implementation varies by cloud provider.
|
||||
|
||||
Kubernetes users will typically not need to worry about anything other than the first two types. The cluster admin
|
||||
will typically ensure that the latter types are setup correctly.
|
||||
|
||||
## Requesting redirects
|
||||
|
||||
Proxies have replaced redirect capabilities. Redirects have been deprecated.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: counter
|
||||
spec:
|
||||
containers:
|
||||
- name: count
|
||||
image: busybox
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
i=0;
|
||||
while true;
|
||||
do
|
||||
echo "$i: $(date)" >> /var/log/1.log;
|
||||
echo "$(date) INFO $i" >> /var/log/2.log;
|
||||
i=$((i+1));
|
||||
sleep 1;
|
||||
done
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
- name: count-agent
|
||||
image: k8s.gcr.io/fluentd-gcp:1.30
|
||||
env:
|
||||
- name: FLUENTD_ARGS
|
||||
value: -c /etc/fluentd-config/fluentd.conf
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
- name: config-volume
|
||||
mountPath: /etc/fluentd-config
|
||||
volumes:
|
||||
- name: varlog
|
||||
emptyDir: {}
|
||||
- name: config-volume
|
||||
configMap:
|
||||
name: fluentd-config
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: counter
|
||||
spec:
|
||||
containers:
|
||||
- name: count
|
||||
image: busybox
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
i=0;
|
||||
while true;
|
||||
do
|
||||
echo "$i: $(date)" >> /var/log/1.log;
|
||||
echo "$(date) INFO $i" >> /var/log/2.log;
|
||||
i=$((i+1));
|
||||
sleep 1;
|
||||
done
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
- name: count-log-1
|
||||
image: busybox
|
||||
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
- name: count-log-2
|
||||
image: busybox
|
||||
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log']
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
volumes:
|
||||
- name: varlog
|
||||
emptyDir: {}
|
||||
@@ -0,0 +1,26 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: counter
|
||||
spec:
|
||||
containers:
|
||||
- name: count
|
||||
image: busybox
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- >
|
||||
i=0;
|
||||
while true;
|
||||
do
|
||||
echo "$i: $(date)" >> /var/log/1.log;
|
||||
echo "$(date) INFO $i" >> /var/log/2.log;
|
||||
i=$((i+1));
|
||||
sleep 1;
|
||||
done
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
volumes:
|
||||
- name: varlog
|
||||
emptyDir: {}
|
||||
Reference in New Issue
Block a user