Merge branch 'master' into patch-1
This commit is contained in:
gavinB-orange
@@ -1,12 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: counter
|
||||
spec:
|
||||
containers:
|
||||
- name: count
|
||||
image: ubuntu:14.04
|
||||
args: [bash, -c,
|
||||
'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done']
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ wget -q -O - https://get.k8s.io | bash
|
||||
|
||||
Once this command completes, you will have a master VM and four worker VMs, running as a Kubernetes cluster.
|
||||
|
||||
By default, some containers will already be running on your cluster. Containers like `fluentd` provide [logging](/docs/getting-started-guides/logging), while `heapster` provides [monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) services.
|
||||
By default, some containers will already be running on your cluster. Containers like `fluentd` provide [logging](/docs/user-guide/logging/overview), while `heapster` provides [monitoring](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/README.md) services.
|
||||
|
||||
The script run by the commands above creates a cluster with the name/prefix "kubernetes". It defines one specific cluster config, so you can't run it more than once.
|
||||
|
||||
|
||||
@@ -319,8 +319,9 @@ edit the `kubeadm` dropin for the `kubelet` service (`/etc/systemd/system/kubele
|
||||
If your cloud provider requires any extra packages installed on host, for example for volume mounting/unmounting, install those packages.
|
||||
|
||||
Specify the `--cloud-provider` flag to kubelet and set it to the cloud of your choice. If your cloudprovider requires a configuration
|
||||
file, create the file `/etc/kubernetes/cloud-config` on every node and set the values your cloud requires. Also append
|
||||
`--cloud-config=/etc/kubernetes/cloud-config` to the kubelet arguments.
|
||||
file, create the file `/etc/kubernetes/cloud-config` on every node. The exact format and content of that file depends on the requirements imposed by your cloud provider.
|
||||
If you use the `/etc/kubernetes/cloud-config` file, you must append it to the `kubelet` arguments as follows:
|
||||
`--cloud-config=/etc/kubernetes/cloud-config`
|
||||
|
||||
Lastly, run `kubeadm init --cloud-provider=xxx` to bootstrap your cluster with cloud provider features.
|
||||
|
||||
|
||||
@@ -1,103 +0,0 @@
|
||||
---
|
||||
assignees:
|
||||
- lavalamp
|
||||
- satnam6502
|
||||
title: Logging with Elasticsearch and Kibana
|
||||
---
|
||||
|
||||
On the Google Compute Engine (GCE) platform, the default logging support targets
|
||||
[Google Cloud Logging](https://cloud.google.com/logging/) as described in the
|
||||
[Logging](/docs/getting-started-guides/logging) getting-started guide. Here we
|
||||
describe how to set up a cluster to ingest logs into
|
||||
[Elasticsearch](https://github.com/elastic/elasticsearch) and view
|
||||
them using [Kibana](https://github.com/elastic/kibana) as an alternative to
|
||||
Google Cloud Logging when running on GCE (note that this will not work as
|
||||
written for Google Container Engine).
|
||||
|
||||
To use Elasticsearch and Kibana for cluster logging, you should set the
|
||||
following environment variable as shown below when creating your cluster with
|
||||
kube-up.sh:
|
||||
|
||||
```shell
|
||||
KUBE_LOGGING_DESTINATION=elasticsearch
|
||||
```
|
||||
|
||||
You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform).
|
||||
|
||||
Now, when you create a cluster, a message will indicate that the Fluentd log
|
||||
collection daemons that run on each node will target Elasticsearch:
|
||||
|
||||
```shell
|
||||
$ cluster/kube-up.sh
|
||||
...
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
... calling kube-up
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel
|
||||
+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d)
|
||||
+++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0)
|
||||
Looking for already existing resources
|
||||
Starting master and configuring firewalls
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd].
|
||||
NAME ZONE SIZE_GB TYPE STATUS
|
||||
kubernetes-master-pd us-central1-b 20 pd-ssd READY
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip].
|
||||
+++ Logging using Fluentd to elasticsearch
|
||||
```
|
||||
|
||||
The per-node Fluentd pods, the Elasticsearch pods, and the Kibana pods should
|
||||
all be running in the kube-system namespace soon after the cluster comes to
|
||||
life.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods --namespace=kube-system
|
||||
NAME READY REASON RESTARTS AGE
|
||||
elasticsearch-logging-v1-78nog 1/1 Running 0 2h
|
||||
elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-5oq0 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-6896 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-l1ds 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-lz9j 1/1 Running 0 2h
|
||||
kibana-logging-v1-bhpo8 1/1 Running 0 2h
|
||||
kube-dns-v3-7r1l9 3/3 Running 0 2h
|
||||
monitoring-heapster-v4-yl332 1/1 Running 1 2h
|
||||
monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h
|
||||
```
|
||||
|
||||
The `fluentd-elasticsearch` pods gather logs from each node and send them to
|
||||
the `elasticsearch-logging` pods, which are part of a
|
||||
[service](/docs/user-guide/services/) named `elasticsearch-logging`. These
|
||||
Elasticsearch pods store the logs and expose them via a REST API.
|
||||
The `kibana-logging` pod provides a web UI for reading the logs stored in
|
||||
Elasticsearch, and is part of a service named `kibana-logging`.
|
||||
|
||||
The Elasticsearch and Kibana services are both in the `kube-system` namespace
|
||||
and are not directly exposed via a publicly reachable IP address. To reach them,
|
||||
follow the instructions for [Accessing services running in a cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster).
|
||||
|
||||
If you try accessing the `elasticsearch-logging` service in your browser, you'll
|
||||
see a status page that looks something like this:
|
||||
|
||||
|
||||
|
||||
You can now type Elasticsearch queries directly into the browser, if you'd
|
||||
like. See [Elasticsearch's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html)
|
||||
for more details on how to do so.
|
||||
|
||||
Alternatively, you can view your cluster's logs using Kibana (again using the
|
||||
[instructions for accessing a service running in the cluster](/docs/user-guide/accessing-the-cluster/#accessing-services-running-on-the-cluster)).
|
||||
The first time you visit the Kibana URL you will be presented with a page that
|
||||
asks you to configure your view of the ingested logs. Select the option for
|
||||
timeseries values and select `@timestamp`. On the following page select the
|
||||
`Discover` tab and then you should be able to see the ingested logs.
|
||||
You can set the refresh interval to 5 seconds to have the logs
|
||||
regularly refreshed.
|
||||
|
||||
Here is a typical view of ingested logs from the Kibana viewer:
|
||||
|
||||
|
||||
|
||||
Kibana opens up all sorts of powerful options for exploring your logs! For some
|
||||
ideas on how to dig into it, check out [Kibana's documentation](https://www.elastic.co/guide/en/kibana/current/discover.html).
|
||||
@@ -1,170 +0,0 @@
|
||||
---
|
||||
assignees:
|
||||
- lavalamp
|
||||
- satnam6502
|
||||
title: Logging
|
||||
---
|
||||
|
||||
A Kubernetes cluster will typically be humming along running many system and application pods. How does the system administrator collect, manage and query the logs of the system pods? How does a user query the logs of their application which is composed of many pods which may be restarted or automatically generated by the Kubernetes system? These questions are addressed by the Kubernetes **cluster level logging** services.
|
||||
|
||||
Cluster level logging for Kubernetes allows us to collect logs which persist beyond the lifetime of the pod's container images or the lifetime of the pod or even cluster. In this article we assume that a Kubernetes cluster has been created with cluster level logging support for sending logs to Google Cloud Logging. After a cluster has been created you will have a collection of system pods running in the `kube-system` namespace that support monitoring,
|
||||
logging and DNS resolution for names of Kubernetes services:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods --namespace=kube-system
|
||||
NAME READY REASON RESTARTS AGE
|
||||
fluentd-cloud-logging-kubernetes-node-0f64 1/1 Running 0 32m
|
||||
fluentd-cloud-logging-kubernetes-node-27gf 1/1 Running 0 32m
|
||||
fluentd-cloud-logging-kubernetes-node-pk22 1/1 Running 0 31m
|
||||
fluentd-cloud-logging-kubernetes-node-20ej 1/1 Running 0 31m
|
||||
kube-dns-v3-pk22 3/3 Running 0 32m
|
||||
monitoring-heapster-v1-20ej 0/1 Running 9 32m
|
||||
```
|
||||
|
||||
Here is the same information in a picture which shows how the pods might be placed on specific nodes.
|
||||
|
||||
|
||||
|
||||
This diagram shows four nodes created on a Google Compute Engine cluster with the name of each VM node on a purple background. The internal and public IPs of each node are shown on gray boxes and the pods running in each node are shown in green boxes. Each pod box shows the name of the pod and the namespace it runs in, the IP address of the pod and the images which are run as part of the pod's execution. Here we see that every node is running a fluentd-cloud-logging pod which is collecting the log output of the containers running on the same node and sending them to Google Cloud Logging. A pod which provides the
|
||||
[cluster DNS service](/docs/admin/dns) runs on one of the nodes and a pod which provides monitoring support runs on another node.
|
||||
|
||||
To help explain how cluster level logging works let's start off with a synthetic log generator pod specification [counter-pod.yaml](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/examples/blog-logging/counter-pod.yaml):
|
||||
|
||||
{% include code.html language="yaml" file="counter-pod.yaml" k8slink="/examples/blog-logging/counter-pod.yaml" %}
|
||||
|
||||
This pod specification has one container which runs a bash script when the container is born. This script simply writes out the value of a counter and the date once per second and runs indefinitely. Let's create the pod in the default
|
||||
namespace.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/blog-logging/counter-pod.yaml
|
||||
pods/counter
|
||||
```
|
||||
|
||||
We can observe the running pod:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
counter 1/1 Running 0 5m
|
||||
```
|
||||
|
||||
This step may take a few minutes to download the ubuntu:14.04 image during which the pod status will be shown as `Pending`.
|
||||
|
||||
One of the nodes is now running the counter pod:
|
||||
|
||||
|
||||
|
||||
When the pod status changes to `Running` we can use the kubectl logs command to view the output of this counter pod.
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter
|
||||
0: Tue Jun 2 21:37:31 UTC 2015
|
||||
1: Tue Jun 2 21:37:32 UTC 2015
|
||||
2: Tue Jun 2 21:37:33 UTC 2015
|
||||
3: Tue Jun 2 21:37:34 UTC 2015
|
||||
4: Tue Jun 2 21:37:35 UTC 2015
|
||||
5: Tue Jun 2 21:37:36 UTC 2015
|
||||
...
|
||||
```
|
||||
|
||||
This command fetches the log text from the Docker log file for the image that is running in this container. We can connect to the running container and observe the running counter bash script.
|
||||
|
||||
```shell
|
||||
$ kubectl exec -i counter bash
|
||||
ps aux
|
||||
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
|
||||
root 1 0.0 0.0 17976 2888 ? Ss 00:02 0:00 bash -c for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 1; done
|
||||
root 468 0.0 0.0 17968 2904 ? Ss 00:05 0:00 bash
|
||||
root 479 0.0 0.0 4348 812 ? S 00:05 0:00 sleep 1
|
||||
root 480 0.0 0.0 15572 2212 ? R 00:05 0:00 ps aux
|
||||
```
|
||||
|
||||
What happens if for any reason the image in this pod is killed off and then restarted by Kubernetes? Will we still see the log lines from the previous invocation of the container followed by the log lines for the started container? Or will we lose the log lines from the original container's execution and only see the log lines for the new container? Let's find out. First let's delete the currently running counter.
|
||||
|
||||
```shell
|
||||
$ kubectl delete pod counter
|
||||
pods/counter
|
||||
```
|
||||
|
||||
Now let's restart the counter.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/blog-logging/counter-pod.yaml
|
||||
pods/counter
|
||||
```
|
||||
|
||||
Let's wait for the container to restart and get the log lines again.
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter
|
||||
0: Tue Jun 2 21:51:40 UTC 2015
|
||||
1: Tue Jun 2 21:51:41 UTC 2015
|
||||
2: Tue Jun 2 21:51:42 UTC 2015
|
||||
3: Tue Jun 2 21:51:43 UTC 2015
|
||||
4: Tue Jun 2 21:51:44 UTC 2015
|
||||
5: Tue Jun 2 21:51:45 UTC 2015
|
||||
6: Tue Jun 2 21:51:46 UTC 2015
|
||||
7: Tue Jun 2 21:51:47 UTC 2015
|
||||
8: Tue Jun 2 21:51:48 UTC 2015
|
||||
```
|
||||
|
||||
We've lost the log lines from the first invocation of the container in this pod! Ideally, we want to preserve all the log lines from each invocation of each container in the pod. Furthermore, even if the pod is restarted we would still like to preserve all the log lines that were ever emitted by the containers in the pod. But don't fear, this is the functionality provided by cluster level logging in Kubernetes. When a cluster is created, the standard output and standard error output of each container can be ingested using a [Fluentd](http://www.fluentd.org/) agent running on each node into either [Google Cloud Logging](https://cloud.google.com/logging/docs/) or into Elasticsearch and viewed with Kibana.
|
||||
|
||||
When a Kubernetes cluster is created with logging to Google Cloud Logging enabled, the system creates a pod called `fluentd-cloud-logging` on each node of the cluster to collect Docker container logs. These pods were shown at the start of this blog article in the response to the first get pods command.
|
||||
|
||||
This log collection pod has a specification which looks something like this:
|
||||
|
||||
{% include code.html language="yaml" file="fluentd-gcp.yaml" k8slink="/cluster/saltbase/salt/fluentd-gcp/fluentd-gcp.yaml" %}
|
||||
|
||||
This pod specification maps the directory on the host containing the Docker log files, `/var/lib/docker/containers`, to a directory inside the container which has the same path. The pod runs one image, `gcr.io/google_containers/fluentd-gcp:1.6`, which is configured to collect the Docker log files from the logs directory and ingest them into Google Cloud Logging. One instance of this pod runs on each node of the cluster. Kubernetes will notice if this pod fails and automatically restart it.
|
||||
|
||||
We can click on the Logs item under the Monitoring section of the Google Developer Console and select the logs for the counter container, which will be called kubernetes.counter_default_count. This identifies the name of the pod (counter), the namespace (default) and the name of the container (count) for which the log collection occurred. Using this name we can select just the logs for our counter container from the drop down menu:
|
||||
|
||||
|
||||
|
||||
When we view the logs in the Developer Console we observe the logs for both invocations of the container.
|
||||
|
||||
|
||||
|
||||
Note the first container counted to 108 and then it was terminated. When the next container image restarted the counting process resumed from 0. Similarly if we deleted the pod and restarted it we would capture the logs for all instances of the containers in the pod whenever the pod was running.
|
||||
|
||||
Logs ingested into Google Cloud Logging may be exported to various other destinations including [Google Cloud Storage](https://cloud.google.com/storage/) buckets and [BigQuery](https://cloud.google.com/bigquery/). Use the Exports tab in the Cloud Logging console to specify where logs should be streamed to. You can also follow this link to the
|
||||
[settings tab](https://pantheon.corp.google.com/project/_/logs/settings).
|
||||
|
||||
We could query the ingested logs from BigQuery using the SQL query which reports the counter log lines showing the newest lines first:
|
||||
|
||||
```shell
|
||||
SELECT metadata.timestamp, structPayload.log
|
||||
FROM [mylogs.kubernetes_counter_default_count_20150611]
|
||||
ORDER BY metadata.timestamp DESC
|
||||
```
|
||||
|
||||
Here is some sample output:
|
||||
|
||||
|
||||
|
||||
We could also fetch the logs from Google Cloud Storage buckets to our desktop or laptop and then search them locally. The following command fetches logs for the counter pod running in a cluster which is itself in a Compute Engine project called `myproject`. Only logs for the date 2015-06-11 are fetched.
|
||||
|
||||
|
||||
```shell
|
||||
$ gsutil -m cp -r gs://myproject/kubernetes.counter_default_count/2015/06/11 .
|
||||
```
|
||||
|
||||
Now we can run queries over the ingested logs. The example below uses the [jq](http://stedolan.github.io/jq/) program to extract just the log lines.
|
||||
|
||||
```shell
|
||||
$ cat 21\:00\:00_21\:59\:59_S0.json | jq '.structPayload.log'
|
||||
"0: Thu Jun 11 21:39:38 UTC 2015\n"
|
||||
"1: Thu Jun 11 21:39:39 UTC 2015\n"
|
||||
"2: Thu Jun 11 21:39:40 UTC 2015\n"
|
||||
"3: Thu Jun 11 21:39:41 UTC 2015\n"
|
||||
"4: Thu Jun 11 21:39:42 UTC 2015\n"
|
||||
"5: Thu Jun 11 21:39:43 UTC 2015\n"
|
||||
"6: Thu Jun 11 21:39:44 UTC 2015\n"
|
||||
"7: Thu Jun 11 21:39:45 UTC 2015\n"
|
||||
...
|
||||
```
|
||||
|
||||
This page has touched briefly on the underlying mechanisms that support gathering cluster level logs on a Kubernetes deployment. The approach here only works for gathering the standard output and standard error output of the processes running in the pod's containers. To gather other logs that are stored in files one can use a sidecar container to gather the required files as described at the page [Collecting log files within containers with Fluentd](https://github.com/kubernetes/contrib/blob/master/logging/fluentd-sidecar-gcp/README.md) and sending them to the Google Cloud Logging service.
|
||||
|
||||
Some of the material in this section also appears in the blog article [Cluster Level Logging with Kubernetes](http://blog.kubernetes.io/2015/06/cluster-level-logging-with-kubernetes.html)
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
assignees:
|
||||
- bboreham
|
||||
|
||||
title: Weave Net Addon
|
||||
---
|
||||
|
||||
The [Weave Net Addon](https://www.weave.works/docs/net/latest/kube-addon/) for Kubernetes comes with a Network Policy Controller.
|
||||
|
||||
@@ -826,11 +826,9 @@ Notes for setting up each cluster service are given below:
|
||||
* [Setup instructions](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/dns/)
|
||||
* [Admin Guide](/docs/admin/dns/)
|
||||
* Cluster-level Logging
|
||||
* Multiple implementations with different storage backends and UIs.
|
||||
* [Elasticsearch Backend Setup Instructions](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/fluentd-elasticsearch/)
|
||||
* [Google Cloud Logging Backend Setup Instructions](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/fluentd-gcp/).
|
||||
* Both require running fluentd on each node.
|
||||
* [User Guide](/docs/user-guide/logging/)
|
||||
* [Cluster-level Logging Overview](/docs/user-guide/logging/overview)
|
||||
* [Cluster-level Logging with Elasticsearch](/docs/user-guide/logging/elasticsearch)
|
||||
* [Cluster-level Logging with Stackdriver Logging](/docs/user-guide/logging/stackdriver)
|
||||
* Container Resource Monitoring
|
||||
* [Setup instructions](http://releases.k8s.io/{{page.githubbranch}}/cluster/addons/cluster-monitoring/)
|
||||
* GUI
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
assignees:
|
||||
- caesarxuchao
|
||||
- erictune
|
||||
|
||||
title: Setting up Kubernetes with Juju
|
||||
---
|
||||
|
||||
Ubuntu 16.04 introduced the [Canonical Distribution of Kubernetes](https://jujucharms.com/canonical-kubernetes/), a pure upstream distribution of Kubernetes designed for production usage. Out of the box it comes with the following components on 12 machines:
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
|
||||
title: Deploying Kubernetes with Calico Networking on Ubuntu
|
||||
---
|
||||
|
||||
This document describes how to deploy Kubernetes with Calico networking from scratch on _bare metal_ Ubuntu. For more information on Project Calico, visit [projectcalico.org](http://projectcalico.org) and the [calico-containers repository](https://github.com/projectcalico/calico-containers).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
assignees:
|
||||
- thockin
|
||||
|
||||
title: Manually Deploying Kubernetes on Ubuntu Nodes
|
||||
---
|
||||
|
||||
This document describes how to deploy Kubernetes on ubuntu nodes, 1 master and 3 nodes involved
|
||||
|
||||
Reference in New Issue
Block a user