whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into merged-master-dev-1.16

Browse Source
This commit is contained in:
simplytunde
2019-08-23 12:43:41 -05:00
parent 0125d9477c 92e439e124
commit 6c66ccd456
94 changed files with 3831 additions and 292 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/debug-application-cluster/audit.md
+7 -12
View File
@@ -262,7 +262,7 @@ and can optionally include a custom CA bundle to use to verify the TLS connectio
The `host` should not refer to a service running in the cluster; use
a service reference by specifying the `service` field instead.
The host might be resolved via external DNS in some apiservers
(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would
(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would
be a layering violation). `host` may also be an IP address.
Please note that using `localhost` or `127.0.0.1` as a `host` is
@@ -489,16 +489,11 @@ Note that in addition to file output plugin, logstash has a variety of outputs t
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
[kube-apiserver]: /docs/admin/kube-apiserver
[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md
[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh#L735
[kubeconfig]: /docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd
[fluentd_plugin_management_doc]: https://docs.fluentd.org/v1.0/articles/plugin-management
[logstash]: https://www.elastic.co/products/logstash
[logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
[kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation
{{% /capture %}}
{{% capture whatsnext %}}
Visit [Auditing with Falco](/docs/tasks/debug-application-cluster/falco)
{{% /capture %}}
content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md
+1 -1
View File
@@ -63,7 +63,7 @@ case you can try several things:
information:
```shell
kubectl get nodes -o yaml | egrep '\sname:\|cpu:\|memory:'
kubectl get nodes -o yaml | egrep '\sname:|cpu:|memory:'
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, cap: .status.capacity}'
```
content/en/docs/tasks/debug-application-cluster/falco.md
+121
View File
@@ -0,0 +1,121 @@
---
reviewers:
- soltysh
- sttts
- ericchiang
content_template: templates/concept
title: Auditing with Falco
---
{{% capture overview %}}
### Use Falco to collect audit events
[Falco](https://falco.org/) is an open source project for intrusion and abnormality detection for Cloud Native platforms.
This section describes how to set up Falco, how to send audit events to the Kubernetes Audit endpoint exposed by Falco, and how Falco applies a set of rules to automatically detect suspicious behavior.
{{% /capture %}}
{{% capture body %}}
#### Install Falco
Install Falco by using one of the following methods:
- [Standalone Falco][falco_installation]
- [Kubernetes DaemonSet][falco_installation]
- [Falco Helm Chart][falco_helm_chart]
Once Falco is installed make sure it is configured to expose the Audit webhook. To do so, use the following configuration:
```yaml
webserver:
enabled: true
listen_port: 8765
k8s_audit_endpoint: /k8s_audit
ssl_enabled: false
ssl_certificate: /etc/falco/falco.pem
```
This configuration is typically found in the `/etc/falco/falco.yaml` file. If Falco is installed as a Kubernetes DaemonSet, edit the `falco-config` ConfigMap and add this configuration.
#### Configure Kubernetes Audit
1. Create a [kubeconfig file](/docs/concepts/configuration/organize-cluster-access-kubeconfig/) for the [kube-apiserver][kube-apiserver] webhook audit backend.
cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
server: http://<ip_of_falco>:8765/k8s_audit
name: falco
contexts:
- context:
cluster: falco
user: ""
name: default-context
current-context: default-context
preferences: {}
users: []
EOF
1. Start [kube-apiserver][kube-apiserver] with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```
#### Audit Rules
Rules devoted to Kubernetes Audit Events can be found in [k8s_audit_rules.yaml][falco_k8s_audit_rules]. If Audit Rules is installed as a native package or using the official Docker images, Falco copies the rules file to `/etc/falco/`, so they are available for use.
There are three classes of rules.
The first class of rules looks for suspicious or exceptional activities, such as:
- Any activity by an unauthorized or anonymous user.
- Creating a pod with an unknown or disallowed image.
- Creating a privileged pod, a pod mounting a sensitive filesystem from the host, or a pod using host networking.
- Creating a NodePort service.
- Creating a ConfigMap containing private credentials, such as passwords and cloud provider secrets.
- Attaching to or executing a command on a running pod.
- Creating a namespace external to a set of allowed namespaces.
- Creating a pod or service account in the kube-system or kube-public namespaces.
- Trying to modify or delete a system ClusterRole.
- Creating a ClusterRoleBinding to the cluster-admin role.
- Creating a ClusterRole with wildcarded verbs or resources. For example, overly permissive.
- Creating a ClusterRole with write permissions or a ClusterRole that can execute commands on pods.
A second class of rules tracks resources being created or destroyed, including:
- Deployments
- Services
- ConfigMaps
- Namespaces
- Service accounts
- Role/ClusterRoles
- Role/ClusterRoleBindings
The final class of rules simply displays any Audit Event received by Falco. This rule is disabled by default, as it can be quite noisy.
For further details, see [Kubernetes Audit Events][falco_ka_docs] in the Falco documentation.
[kube-apiserver]: /docs/admin/kube-apiserver
[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md
[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh#L735
[kubeconfig]: /docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd
[fluentd_plugin_management_doc]: https://docs.fluentd.org/v1.0/articles/plugin-management
[logstash]: https://www.elastic.co/products/logstash
[logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
[kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation
[falco_website]: https://www.falco.org
[falco_k8s_audit_rules]: https://github.com/falcosecurity/falco/blob/master/rules/k8s_audit_rules.yaml
[falco_ka_docs]: https://falco.org/docs/event-sources/kubernetes-audit
[falco_installation]: https://falco.org/docs/installation
[falco_helm_chart]: https://github.com/helm/charts/tree/master/stable/falco
{{% /capture %}}
content/en/docs/tasks/debug-application-cluster/resource-usage-monitoring.md
+27 -87
View File
@@ -20,99 +20,39 @@ where bottlenecks can be removed to improve overall performance.
{{% capture body %}}
In Kubernetes, application monitoring does not depend on a single monitoring
solution. On new clusters, you can use two separate pipelines to collect
monitoring statistics by default:
- The [**resource metrics pipeline**](#resource-metrics-pipeline) provides a limited set of metrics related
to cluster components such as the HorizontalPodAutoscaler controller, as well
as the `kubectl top` utility. These metrics are collected by
[metrics-server](https://github.com/kubernetes-incubator/metrics-server)
and are exposed via the `metrics.k8s.io` API. `metrics-server` discovers
all nodes on the cluster and queries each node's [Kubelet](/docs/admin/kubelet)
for CPU and memory usage. The Kubelet fetches the data from
[cAdvisor](https://github.com/google/cadvisor). `metrics-server` is a
lightweight short-term in-memory store.
- A [**full metrics pipeline**](#full-metrics-pipelines), such as Prometheus, gives you access to richer
metrics. In addition, Kubernetes can respond to these metrics by automatically
scaling or adapting the cluster based on its current state, using mechanisms
such as the Horizontal Pod Autoscaler. The monitoring pipeline fetches
metrics from the Kubelet, and then exposes them to Kubernetes via an adapter
by implementing either the `custom.metrics.k8s.io` or
`external.metrics.k8s.io` API.
In Kubernetes, application monitoring does not depend on a single monitoring solution. On new clusters, you can use [resource metrics](#resource-metrics-pipeline) or [full metrics](#full-metrics-pipeline) pipelines to collect monitoring statistics.
## Resource metrics pipeline
### Kubelet
The resource metrics pipeline provides a limited set of metrics related to
cluster components such as the [Horizontal Pod Autoscaler](/docs/tasks/run-application/horizontal-pod-autoscale) controller, as well as the `kubectl top` utility.
These metrics are collected by the lightweight, short-term, in-memory
[metrics-server](https://github.com/kubernetes-incubator/metrics-server) and
are exposed via the `metrics.k8s.io` API.
The Kubelet acts as a bridge between the Kubernetes master and the nodes. It manages the pods and containers running on a machine. Kubelet translates each pod into its constituent containers and fetches individual container usage statistics from the container runtime, through the container runtime interface. For the legacy docker integration, it fetches this information from cAdvisor. It then exposes the aggregated pod resource usage statistics through the kubelet resource metrics api. This api is served at `/metrics/resource/v1alpha1` on the kubelet's authenticated and read-only ports.
metrics-server discovers all nodes on the cluster and
queries each node's
[kubelet](/docs/reference/command-line-tools-reference/kubelet) for CPU and
memory usage. The kubelet acts as a bridge between the Kubernetes master and
the nodes, managing the pods and containers running on a machine. The kubelet
translates each pod into its constituent containers and fetches individual
container usage statistics from the container runtime through the container
runtime interface. The kubelet fetches this information from the integrated
cAdvisor for the legacy Docker integration. It then exposes the aggregated pod
resource usage statistics through the metrics-server Resource Metrics API.
This API is served at `/metrics/resource/v1beta1` on the kubelet's authenticated and
read-only ports.
### cAdvisor
## Full metrics pipeline
cAdvisor is an open source container resource usage and performance analysis agent. It is purpose-built for containers and supports Docker containers natively. In Kubernetes, cAdvisor is integrated into the Kubelet binary. cAdvisor auto-discovers all containers in the machine and collects CPU, memory, filesystem, and network usage statistics. cAdvisor also provides the overall machine usage by analyzing the 'root' container on the machine.
A full metrics pipeline gives you access to richer metrics. Kubernetes can
respond to these metrics by automatically scaling or adapting the cluster
based on its current state, using mechanisms such as the Horizontal Pod
Autoscaler. The monitoring pipeline fetches metrics from the kubelet and
then exposes them to Kubernetes via an adapter by implementing either the
`custom.metrics.k8s.io` or `external.metrics.k8s.io` API.
Kubelet exposes a simple cAdvisor UI for containers on a machine, via the default port 4194.
The picture below is an example showing the overall machine usage. However, this feature has been marked
deprecated in v1.10 and completely removed in v1.12.
![cAdvisor](/images/docs/cadvisor.png)
Starting from v1.13, you can [deploy cAdvisor as a DaemonSet](https://github.com/google/cadvisor/tree/master/deploy/kubernetes) for an access to the cAdvisor UI.
## Full metrics pipelines
Many full metrics solutions exist for Kubernetes.
### Prometheus
[Prometheus](https://prometheus.io) can natively monitor kubernetes, nodes, and prometheus itself.
The [Prometheus Operator](https://coreos.com/operators/prometheus/docs/latest/)
simplifies Prometheus setup on Kubernetes, and allows you to serve the
custom metrics API using the
[Prometheus adapter](https://github.com/directxman12/k8s-prometheus-adapter).
Prometheus provides a robust query language and a built-in dashboard for
querying and visualizing your data. Prometheus is also a supported
data source for [Grafana](https://prometheus.io/docs/visualization/grafana/).
### Sysdig
[Sysdig](http://sysdig.com) provides full spectrum container and platform intelligence, and is a
true container native solution. Sysdig pulls together data from system calls, Kubernetes events,
Prometheus metrics, statsD, JMX, and more into a single pane that gives you a comprehensive picture
of your environment. Sysdig also provides an API to query for providing robust and customizable
solutions. Sysdig is built on Open Source. [Sysdig and Sysdig Inspect](https://sysdig.com/opensource/inspect/) give you the
ability to freely perform troubleshooting, performance analyis and forensics.
### Google Cloud Monitoring
Google Cloud Monitoring is a hosted monitoring service you can use to
visualize and alert on important metrics in your application. You can collect
metrics from Kubernetes, and you can access them
using the [Cloud Monitoring Console](https://app.google.stackdriver.com/).
You can create and customize dashboards to visualize the data gathered
from your Kubernetes cluster.
This video shows how to configure and run a Google Cloud Monitoring backed Heapster:
[![how to setup and run a Google Cloud Monitoring backed Heapster](https://img.youtube.com/vi/xSMNR2fcoLs/0.jpg)](https://www.youtube.com/watch?v=xSMNR2fcoLs)
{{< figure src="/images/docs/gcm.png" alt="Google Cloud Monitoring dashboard example" title="Google Cloud Monitoring dashboard example" caption="This dashboard shows cluster-wide resource usage." >}}
## CronJob monitoring
### Kubernetes Job Monitor
With the [Kubernetes Job Monitor](https://github.com/pietervogelaar/kubernetes-job-monitor) dashboard a Cluster Administrator can see which jobs are running and view the status of completed jobs.
### New Relic Kubernetes monitoring integration
[New Relic Kubernetes](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration) integration provides increased visibility into the performance of your Kubernetes environment. New Relic's Kubernetes integration instruments the container orchestration layer by reporting metrics from Kubernetes objects. The integration gives you insight into your Kubernetes nodes, namespaces, deployments, replica sets, pods, and containers.
Marquee capabilities:
View your data in pre-built dashboards for immediate insight into your Kubernetes environment.
Create your own custom queries and charts in Insights from automatically reported data.
Create alert conditions on Kubernetes data.
Learn more on this [page](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration).
[Prometheus](https://prometheus.io), a CNCF project, can natively monitor Kubernetes, nodes, and Prometheus itself.
Full metrics pipeline projects that are not part of the CNCF are outside the scope of Kubernetes documentation.
{{% /capture %}}