Merge pull request #24769 from kbhawkey/kb-task-monitoring-stackdriver
remove stackdriver, elastic task pages
This commit is contained in:
Kubernetes Prow Robot
@@ -1,94 +0,0 @@
|
||||
---
|
||||
reviewers:
|
||||
- piosz
|
||||
- x13n
|
||||
content_type: concept
|
||||
title: Events in Stackdriver
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Kubernetes events are objects that provide insight into what is happening
|
||||
inside a cluster, such as what decisions were made by scheduler or why some
|
||||
pods were evicted from the node. You can read more about using events
|
||||
for debugging your application in the [Application Introspection and Debugging
|
||||
](/docs/tasks/debug-application-cluster/debug-application-introspection/)
|
||||
section.
|
||||
|
||||
Since events are API objects, they are stored in the apiserver on master. To
|
||||
avoid filling up master's disk, a retention policy is enforced: events are
|
||||
removed one hour after the last occurrence. To provide longer history
|
||||
and aggregation capabilities, a third party solution should be installed
|
||||
to capture events.
|
||||
|
||||
This article describes a solution that exports Kubernetes events to
|
||||
Stackdriver Logging, where they can be processed and analyzed.
|
||||
|
||||
{{< note >}}
|
||||
It is not guaranteed that all events happening in a cluster will be
|
||||
exported to Stackdriver. One possible scenario when events will not be
|
||||
exported is when event exporter is not running (e.g. during restart or
|
||||
upgrade). In most cases it's fine to use events for purposes like setting up
|
||||
[metrics](https://cloud.google.com/logging/docs/logs-based-metrics/) and [alerts](https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts), but you should be aware
|
||||
of the potential inaccuracy.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Deployment
|
||||
|
||||
### Google Kubernetes Engine
|
||||
|
||||
In Google Kubernetes Engine, if cloud logging is enabled, event exporter
|
||||
is deployed by default to the clusters with master running version 1.7 and
|
||||
higher. To prevent disturbing your workloads, event exporter does not have
|
||||
resources set and is in the best effort QOS class, which means that it will
|
||||
be the first to be killed in the case of resource starvation. If you want
|
||||
your events to be exported, make sure you have enough resources to facilitate
|
||||
the event exporter pod. This may vary depending on the workload, but on
|
||||
average, approximately 100Mb RAM and 100m CPU is needed.
|
||||
|
||||
### Deploying to the Existing Cluster
|
||||
|
||||
Deploy event exporter to your cluster using the following command:
|
||||
|
||||
```shell
|
||||
kubectl apply -f https://k8s.io/examples/debug/event-exporter.yaml
|
||||
```
|
||||
|
||||
Since event exporter accesses the Kubernetes API, it requires permissions to
|
||||
do so. The following deployment is configured to work with RBAC
|
||||
authorization. It sets up a service account and a cluster role binding
|
||||
to allow event exporter to read events. To make sure that event exporter
|
||||
pod will not be evicted from the node, you can additionally set up resource
|
||||
requests. As mentioned earlier, 100Mb RAM and 100m CPU should be enough.
|
||||
|
||||
{{< codenew file="debug/event-exporter.yaml" >}}
|
||||
|
||||
## User Guide
|
||||
|
||||
Events are exported to the `GKE Cluster` resource in Stackdriver Logging.
|
||||
You can find them by selecting an appropriate option from a drop-down menu
|
||||
of available resources:
|
||||
|
||||
<img src="/images/docs/stackdriver-event-exporter-resource.png" alt="Events location in the Stackdriver Logging interface" width="500">
|
||||
|
||||
You can filter based on the event object fields using Stackdriver Logging
|
||||
[filtering mechanism](https://cloud.google.com/logging/docs/view/advanced_filters).
|
||||
For example, the following query will show events from the scheduler
|
||||
about pods from deployment `nginx-deployment`:
|
||||
|
||||
```
|
||||
resource.type="gke_cluster"
|
||||
jsonPayload.kind="Event"
|
||||
jsonPayload.source.component="default-scheduler"
|
||||
jsonPayload.involvedObject.name:"nginx-deployment"
|
||||
```
|
||||
|
||||
{{< figure src="/images/docs/stackdriver-event-exporter-filter.png" alt="Filtered events in the Stackdriver Logging interface" width="500" >}}
|
||||
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
---
|
||||
reviewers:
|
||||
- piosz
|
||||
- x13n
|
||||
content_type: concept
|
||||
title: Logging Using Elasticsearch and Kibana
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
On the Google Compute Engine (GCE) platform, the default logging support targets
|
||||
[Stackdriver Logging](https://cloud.google.com/logging/), which is described in detail
|
||||
in the [Logging With Stackdriver Logging](/docs/tasks/debug-application-cluster/logging-stackdriver).
|
||||
|
||||
This article describes how to set up a cluster to ingest logs into
|
||||
[Elasticsearch](https://www.elastic.co/products/elasticsearch) and view
|
||||
them using [Kibana](https://www.elastic.co/products/kibana), as an alternative to
|
||||
Stackdriver Logging when running on GCE.
|
||||
|
||||
{{< note >}}
|
||||
You cannot automatically deploy Elasticsearch and Kibana in the Kubernetes cluster hosted on Google Kubernetes Engine. You have to deploy them manually.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
To use Elasticsearch and Kibana for cluster logging, you should set the
|
||||
following environment variable as shown below when creating your cluster with
|
||||
kube-up.sh:
|
||||
|
||||
```shell
|
||||
KUBE_LOGGING_DESTINATION=elasticsearch
|
||||
```
|
||||
|
||||
You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform).
|
||||
|
||||
Now, when you create a cluster, a message will indicate that the Fluentd log
|
||||
collection daemons that run on each node will target Elasticsearch:
|
||||
|
||||
```shell
|
||||
cluster/kube-up.sh
|
||||
```
|
||||
```
|
||||
...
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
... calling kube-up
|
||||
Project: kubernetes-satnam
|
||||
Zone: us-central1-b
|
||||
+++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel
|
||||
+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d)
|
||||
+++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0)
|
||||
Looking for already existing resources
|
||||
Starting master and configuring firewalls
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd].
|
||||
NAME ZONE SIZE_GB TYPE STATUS
|
||||
kubernetes-master-pd us-central1-b 20 pd-ssd READY
|
||||
Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip].
|
||||
+++ Logging using Fluentd to elasticsearch
|
||||
```
|
||||
|
||||
The per-node Fluentd pods, the Elasticsearch pods, and the Kibana pods should
|
||||
all be running in the kube-system namespace soon after the cluster comes to
|
||||
life.
|
||||
|
||||
```shell
|
||||
kubectl get pods --namespace=kube-system
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
elasticsearch-logging-v1-78nog 1/1 Running 0 2h
|
||||
elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-5oq0 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-6896 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-l1ds 1/1 Running 0 2h
|
||||
fluentd-elasticsearch-kubernetes-node-lz9j 1/1 Running 0 2h
|
||||
kibana-logging-v1-bhpo8 1/1 Running 0 2h
|
||||
kube-dns-v3-7r1l9 3/3 Running 0 2h
|
||||
monitoring-heapster-v4-yl332 1/1 Running 1 2h
|
||||
monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h
|
||||
```
|
||||
|
||||
The `fluentd-elasticsearch` pods gather logs from each node and send them to
|
||||
the `elasticsearch-logging` pods, which are part of a
|
||||
[service](/docs/concepts/services-networking/service/) named `elasticsearch-logging`. These
|
||||
Elasticsearch pods store the logs and expose them via a REST API.
|
||||
The `kibana-logging` pod provides a web UI for reading the logs stored in
|
||||
Elasticsearch, and is part of a service named `kibana-logging`.
|
||||
|
||||
The Elasticsearch and Kibana services are both in the `kube-system` namespace
|
||||
and are not directly exposed via a publicly reachable IP address. To reach them,
|
||||
follow the instructions for
|
||||
[Accessing services running in a cluster](/docs/tasks/access-application-cluster/access-cluster/#accessing-services-running-on-the-cluster).
|
||||
|
||||
If you try accessing the `elasticsearch-logging` service in your browser, you'll
|
||||
see a status page that looks something like this:
|
||||
|
||||
|
||||
|
||||
You can now type Elasticsearch queries directly into the browser, if you'd
|
||||
like. See [Elasticsearch's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html)
|
||||
for more details on how to do so.
|
||||
|
||||
Alternatively, you can view your cluster's logs using Kibana (again using the
|
||||
[instructions for accessing a service running in the cluster](/docs/tasks/access-application-cluster/access-cluster/#accessing-services-running-on-the-cluster)).
|
||||
The first time you visit the Kibana URL you will be presented with a page that
|
||||
asks you to configure your view of the ingested logs. Select the option for
|
||||
timeseries values and select `@timestamp`. On the following page select the
|
||||
`Discover` tab and then you should be able to see the ingested logs.
|
||||
You can set the refresh interval to 5 seconds to have the logs
|
||||
regularly refreshed.
|
||||
|
||||
Here is a typical view of ingested logs from the Kibana viewer:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## {{% heading "whatsnext" %}}
|
||||
|
||||
|
||||
Kibana opens up all sorts of powerful options for exploring your logs! For some
|
||||
ideas on how to dig into it, check out [Kibana's documentation](https://www.elastic.co/guide/en/kibana/current/discover.html).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user