Merge remote-tracking branch 'upstream/master' into merged-master-dev-1.16

This commit is contained in:
simplytunde
2019-08-23 12:43:41 -05:00
94 changed files with 3831 additions and 292 deletions
@@ -9,6 +9,10 @@ weight: 80
This page shows how to create an External Load Balancer.
{{< note >}}
This feature is only available for cloud providers or environments which support external load balancers.
{{< /note >}}
When creating a service, you have the option of automatically creating a
cloud network load balancer. This provides an externally-accessible IP address
that sends traffic to the correct port on your cluster nodes
@@ -35,30 +39,24 @@ documentation.
To create an external load balancer, add the following line to your
[service configuration file](/docs/concepts/services-networking/service/#loadbalancer):
```json
"type": "LoadBalancer"
```yaml
type: LoadBalancer
```
Your configuration file might look like:
```json
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "example-service"
},
"spec": {
"ports": [{
"port": 8765,
"targetPort": 9376
}],
"selector": {
"app": "example"
},
"type": "LoadBalancer"
}
}
```yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
selector:
app: example
ports:
- port: 8765
targetPort: 9376
type: LoadBalancer
```
## Using kubectl
@@ -118,82 +116,42 @@ minikube service example-service --url
## Preserving the client source IP
Due to the implementation of this feature, the source IP seen in the target
container will *not be the original source IP* of the client. To enable
container is *not the original source IP* of the client. To enable
preservation of the client IP, the following fields can be configured in the
service spec (supported in GCE/Google Kubernetes Engine environments):
* `service.spec.externalTrafficPolicy` - denotes if this Service desires to route
external traffic to node-local or cluster-wide endpoints. There are two available
options: "Cluster" (default) and "Local". "Cluster" obscures the client source
options: Cluster (default) and Local. Cluster obscures the client source
IP and may cause a second hop to another node, but should have good overall
load-spreading. "Local" preserves the client source IP and avoids a second hop
load-spreading. Local preserves the client source IP and avoids a second hop
for LoadBalancer and NodePort type services, but risks potentially imbalanced
traffic spreading.
* `service.spec.healthCheckNodePort` - specifies the healthcheck nodePort
(numeric port number) for the service. If not specified, healthCheckNodePort is
created by the service API backend with the allocated nodePort. It will use the
user-specified nodePort value if specified by the client. It only has an
effect when type is set to "LoadBalancer" and externalTrafficPolicy is set
to "Local".
* `service.spec.healthCheckNodePort` - specifies the health check nodePort
(numeric port number) for the service. If not specified, `healthCheckNodePort` is
created by the service API backend with the allocated `nodePort`. It will use the
user-specified `nodePort` value if specified by the client. It only has an
effect when `type` is set to LoadBalancer and `externalTrafficPolicy` is set
to Local.
This feature can be activated by setting `externalTrafficPolicy` to "Local" in the
Service Configuration file.
Setting `externalTrafficPolicy` to Local in the Service configuration file
activates this feature.
```json
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "example-service"
},
"spec": {
"ports": [{
"port": 8765,
"targetPort": 9376
}],
"selector": {
"app": "example"
},
"type": "LoadBalancer",
"externalTrafficPolicy": "Local"
}
}
```yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
selector:
app: example
ports:
- port: 8765
targetPort: 9376
externalTrafficPolicy: Local
type: LoadBalancer
```
### Feature availability
| K8s version | Feature support |
| :---------: |:-----------:|
| 1.7+ | Supports the full API fields |
| 1.5 - 1.6 | Supports Beta Annotations |
| <1.5 | Unsupported |
Below you could find the deprecated Beta annotations used to enable this feature
prior to its stable version. Newer Kubernetes versions may stop supporting these
after v1.7. Please update existing applications to use the fields directly.
* `service.beta.kubernetes.io/external-traffic` annotation <-> `service.spec.externalTrafficPolicy` field
* `service.beta.kubernetes.io/healthcheck-nodeport` annotation <-> `service.spec.healthCheckNodePort` field
`service.beta.kubernetes.io/external-traffic` annotation has a different set of values
compared to the `service.spec.externalTrafficPolicy` field. The values match as follows:
* "OnlyLocal" for annotation <-> "Local" for field
* "Global" for annotation <-> "Cluster" for field
{{< note >}}
This feature is not currently implemented for all cloudproviders/environments.
{{< /note >}}
Known issues:
* AWS: [kubernetes/kubernetes#35758](https://github.com/kubernetes/kubernetes/issues/35758)
* Weave-Net: [weaveworks/weave/#2924](https://github.com/weaveworks/weave/issues/2924)
{{% /capture %}}
{{% capture discussion %}}
## Garbage Collecting Load Balancers
In usual case, the correlating load balancer resources in cloud provider should
@@ -203,7 +161,7 @@ associated Service is deleted. Finalizer Protection for Service LoadBalancers wa
introduced to prevent this from happening. By using finalizers, a Service resource
will never be deleted until the correlating load balancer resources are also deleted.
Specifically, if a Service has Type=LoadBalancer, the service controller will attach
Specifically, if a Service has `type` LoadBalancer, the service controller will attach
a finalizer named `service.kubernetes.io/load-balancer-cleanup`.
The finalizer will only be removed after the load balancer resource is cleaned up.
This prevents dangling load balancer resources even in corner cases such as the
@@ -217,14 +175,18 @@ enabling the [feature gate](/docs/reference/command-line-tools-reference/feature
It is important to note that the datapath for this functionality is provided by a load balancer external to the Kubernetes cluster.
When the service type is set to `LoadBalancer`, Kubernetes provides functionality equivalent to `type=<ClusterIP>` to pods within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the Kubernetes pods. The Kubernetes service controller automates the creation of the external load balancer, health checks (if needed), firewall rules (if needed) and retrieves the external IP allocated by the cloud provider and populates it in the service object.
When the Service `type` is set to LoadBalancer, Kubernetes provides functionality equivalent to `type` equals ClusterIP to pods
within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the Kubernetes
pods. The Kubernetes service controller automates the creation of the external load balancer, health checks (if needed),
firewall rules (if needed) and retrieves the external IP allocated by the cloud provider and populates it in the service
object.
## Caveats and Limitations when preserving source IPs
GCE/AWS load balancers do not provide weights for their target pools. This was not an issue with the old LB
kube-proxy rules which would correctly balance across all endpoints.
With the new functionality, the external traffic will not be equally load balanced across pods, but rather
With the new functionality, the external traffic is not equally load balanced across pods, but rather
equally balanced at the node level (because GCE/AWS and other external LB implementations do not have the ability
for specifying the weight per node, they balance equally across all target nodes, disregarding the number of
pods on each node).
@@ -238,5 +200,3 @@ Once the external load balancers provide weights, this functionality can be adde
Internal pod to pod traffic should behave similar to ClusterIP services, with equal probability across all pods.
{{% /capture %}}
@@ -140,13 +140,13 @@ The following file is an Ingress resource that sends traffic to your Service via
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /*
- path: /(.+)
backend:
serviceName: web
servicePort: 8080
@@ -113,7 +113,7 @@ of `Always`.
kubectl exec -it redis -- /bin/bash
```
1. In your shell, goto `/data/redis`, and verify that `test-file` is still there.
1. In your shell, go to `/data/redis`, and verify that `test-file` is still there.
```shell
root@redis:/data/redis# cd /data/redis/
root@redis:/data/redis# ls
@@ -262,7 +262,7 @@ and can optionally include a custom CA bundle to use to verify the TLS connectio
The `host` should not refer to a service running in the cluster; use
a service reference by specifying the `service` field instead.
The host might be resolved via external DNS in some apiservers
(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would
(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would
be a layering violation). `host` may also be an IP address.
Please note that using `localhost` or `127.0.0.1` as a `host` is
@@ -489,16 +489,11 @@ Note that in addition to file output plugin, logstash has a variety of outputs t
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
[kube-apiserver]: /docs/admin/kube-apiserver
[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md
[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh#L735
[kubeconfig]: /docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd
[fluentd_plugin_management_doc]: https://docs.fluentd.org/v1.0/articles/plugin-management
[logstash]: https://www.elastic.co/products/logstash
[logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
[kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation
{{% /capture %}}
{{% capture whatsnext %}}
Visit [Auditing with Falco](/docs/tasks/debug-application-cluster/falco)
{{% /capture %}}
@@ -63,7 +63,7 @@ case you can try several things:
information:
```shell
kubectl get nodes -o yaml | egrep '\sname:\|cpu:\|memory:'
kubectl get nodes -o yaml | egrep '\sname:|cpu:|memory:'
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, cap: .status.capacity}'
```
@@ -0,0 +1,121 @@
---
reviewers:
- soltysh
- sttts
- ericchiang
content_template: templates/concept
title: Auditing with Falco
---
{{% capture overview %}}
### Use Falco to collect audit events
[Falco](https://falco.org/) is an open source project for intrusion and abnormality detection for Cloud Native platforms.
This section describes how to set up Falco, how to send audit events to the Kubernetes Audit endpoint exposed by Falco, and how Falco applies a set of rules to automatically detect suspicious behavior.
{{% /capture %}}
{{% capture body %}}
#### Install Falco
Install Falco by using one of the following methods:
- [Standalone Falco][falco_installation]
- [Kubernetes DaemonSet][falco_installation]
- [Falco Helm Chart][falco_helm_chart]
Once Falco is installed make sure it is configured to expose the Audit webhook. To do so, use the following configuration:
```yaml
webserver:
enabled: true
listen_port: 8765
k8s_audit_endpoint: /k8s_audit
ssl_enabled: false
ssl_certificate: /etc/falco/falco.pem
```
This configuration is typically found in the `/etc/falco/falco.yaml` file. If Falco is installed as a Kubernetes DaemonSet, edit the `falco-config` ConfigMap and add this configuration.
#### Configure Kubernetes Audit
1. Create a [kubeconfig file](/docs/concepts/configuration/organize-cluster-access-kubeconfig/) for the [kube-apiserver][kube-apiserver] webhook audit backend.
cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
server: http://<ip_of_falco>:8765/k8s_audit
name: falco
contexts:
- context:
cluster: falco
user: ""
name: default-context
current-context: default-context
preferences: {}
users: []
EOF
1. Start [kube-apiserver][kube-apiserver] with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```
#### Audit Rules
Rules devoted to Kubernetes Audit Events can be found in [k8s_audit_rules.yaml][falco_k8s_audit_rules]. If Audit Rules is installed as a native package or using the official Docker images, Falco copies the rules file to `/etc/falco/`, so they are available for use.
There are three classes of rules.
The first class of rules looks for suspicious or exceptional activities, such as:
- Any activity by an unauthorized or anonymous user.
- Creating a pod with an unknown or disallowed image.
- Creating a privileged pod, a pod mounting a sensitive filesystem from the host, or a pod using host networking.
- Creating a NodePort service.
- Creating a ConfigMap containing private credentials, such as passwords and cloud provider secrets.
- Attaching to or executing a command on a running pod.
- Creating a namespace external to a set of allowed namespaces.
- Creating a pod or service account in the kube-system or kube-public namespaces.
- Trying to modify or delete a system ClusterRole.
- Creating a ClusterRoleBinding to the cluster-admin role.
- Creating a ClusterRole with wildcarded verbs or resources. For example, overly permissive.
- Creating a ClusterRole with write permissions or a ClusterRole that can execute commands on pods.
A second class of rules tracks resources being created or destroyed, including:
- Deployments
- Services
- ConfigMaps
- Namespaces
- Service accounts
- Role/ClusterRoles
- Role/ClusterRoleBindings
The final class of rules simply displays any Audit Event received by Falco. This rule is disabled by default, as it can be quite noisy.
For further details, see [Kubernetes Audit Events][falco_ka_docs] in the Falco documentation.
[kube-apiserver]: /docs/admin/kube-apiserver
[auditing-proposal]: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/auditing.md
[auditing-api]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go
[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh#L735
[kubeconfig]: /docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[fluentd]: http://www.fluentd.org/
[fluentd_install_doc]: https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd
[fluentd_plugin_management_doc]: https://docs.fluentd.org/v1.0/articles/plugin-management
[logstash]: https://www.elastic.co/products/logstash
[logstash_install_doc]: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
[kube-aggregator]: /docs/concepts/api-extension/apiserver-aggregation
[falco_website]: https://www.falco.org
[falco_k8s_audit_rules]: https://github.com/falcosecurity/falco/blob/master/rules/k8s_audit_rules.yaml
[falco_ka_docs]: https://falco.org/docs/event-sources/kubernetes-audit
[falco_installation]: https://falco.org/docs/installation
[falco_helm_chart]: https://github.com/helm/charts/tree/master/stable/falco
{{% /capture %}}
@@ -20,99 +20,39 @@ where bottlenecks can be removed to improve overall performance.
{{% capture body %}}
In Kubernetes, application monitoring does not depend on a single monitoring
solution. On new clusters, you can use two separate pipelines to collect
monitoring statistics by default:
- The [**resource metrics pipeline**](#resource-metrics-pipeline) provides a limited set of metrics related
to cluster components such as the HorizontalPodAutoscaler controller, as well
as the `kubectl top` utility. These metrics are collected by
[metrics-server](https://github.com/kubernetes-incubator/metrics-server)
and are exposed via the `metrics.k8s.io` API. `metrics-server` discovers
all nodes on the cluster and queries each node's [Kubelet](/docs/admin/kubelet)
for CPU and memory usage. The Kubelet fetches the data from
[cAdvisor](https://github.com/google/cadvisor). `metrics-server` is a
lightweight short-term in-memory store.
- A [**full metrics pipeline**](#full-metrics-pipelines), such as Prometheus, gives you access to richer
metrics. In addition, Kubernetes can respond to these metrics by automatically
scaling or adapting the cluster based on its current state, using mechanisms
such as the Horizontal Pod Autoscaler. The monitoring pipeline fetches
metrics from the Kubelet, and then exposes them to Kubernetes via an adapter
by implementing either the `custom.metrics.k8s.io` or
`external.metrics.k8s.io` API.
In Kubernetes, application monitoring does not depend on a single monitoring solution. On new clusters, you can use [resource metrics](#resource-metrics-pipeline) or [full metrics](#full-metrics-pipeline) pipelines to collect monitoring statistics.
## Resource metrics pipeline
### Kubelet
The resource metrics pipeline provides a limited set of metrics related to
cluster components such as the [Horizontal Pod Autoscaler](/docs/tasks/run-application/horizontal-pod-autoscale) controller, as well as the `kubectl top` utility.
These metrics are collected by the lightweight, short-term, in-memory
[metrics-server](https://github.com/kubernetes-incubator/metrics-server) and
are exposed via the `metrics.k8s.io` API.
The Kubelet acts as a bridge between the Kubernetes master and the nodes. It manages the pods and containers running on a machine. Kubelet translates each pod into its constituent containers and fetches individual container usage statistics from the container runtime, through the container runtime interface. For the legacy docker integration, it fetches this information from cAdvisor. It then exposes the aggregated pod resource usage statistics through the kubelet resource metrics api. This api is served at `/metrics/resource/v1alpha1` on the kubelet's authenticated and read-only ports.
metrics-server discovers all nodes on the cluster and
queries each node's
[kubelet](/docs/reference/command-line-tools-reference/kubelet) for CPU and
memory usage. The kubelet acts as a bridge between the Kubernetes master and
the nodes, managing the pods and containers running on a machine. The kubelet
translates each pod into its constituent containers and fetches individual
container usage statistics from the container runtime through the container
runtime interface. The kubelet fetches this information from the integrated
cAdvisor for the legacy Docker integration. It then exposes the aggregated pod
resource usage statistics through the metrics-server Resource Metrics API.
This API is served at `/metrics/resource/v1beta1` on the kubelet's authenticated and
read-only ports.
### cAdvisor
## Full metrics pipeline
cAdvisor is an open source container resource usage and performance analysis agent. It is purpose-built for containers and supports Docker containers natively. In Kubernetes, cAdvisor is integrated into the Kubelet binary. cAdvisor auto-discovers all containers in the machine and collects CPU, memory, filesystem, and network usage statistics. cAdvisor also provides the overall machine usage by analyzing the 'root' container on the machine.
A full metrics pipeline gives you access to richer metrics. Kubernetes can
respond to these metrics by automatically scaling or adapting the cluster
based on its current state, using mechanisms such as the Horizontal Pod
Autoscaler. The monitoring pipeline fetches metrics from the kubelet and
then exposes them to Kubernetes via an adapter by implementing either the
`custom.metrics.k8s.io` or `external.metrics.k8s.io` API.
Kubelet exposes a simple cAdvisor UI for containers on a machine, via the default port 4194.
The picture below is an example showing the overall machine usage. However, this feature has been marked
deprecated in v1.10 and completely removed in v1.12.
![cAdvisor](/images/docs/cadvisor.png)
Starting from v1.13, you can [deploy cAdvisor as a DaemonSet](https://github.com/google/cadvisor/tree/master/deploy/kubernetes) for an access to the cAdvisor UI.
## Full metrics pipelines
Many full metrics solutions exist for Kubernetes.
### Prometheus
[Prometheus](https://prometheus.io) can natively monitor kubernetes, nodes, and prometheus itself.
The [Prometheus Operator](https://coreos.com/operators/prometheus/docs/latest/)
simplifies Prometheus setup on Kubernetes, and allows you to serve the
custom metrics API using the
[Prometheus adapter](https://github.com/directxman12/k8s-prometheus-adapter).
Prometheus provides a robust query language and a built-in dashboard for
querying and visualizing your data. Prometheus is also a supported
data source for [Grafana](https://prometheus.io/docs/visualization/grafana/).
### Sysdig
[Sysdig](http://sysdig.com) provides full spectrum container and platform intelligence, and is a
true container native solution. Sysdig pulls together data from system calls, Kubernetes events,
Prometheus metrics, statsD, JMX, and more into a single pane that gives you a comprehensive picture
of your environment. Sysdig also provides an API to query for providing robust and customizable
solutions. Sysdig is built on Open Source. [Sysdig and Sysdig Inspect](https://sysdig.com/opensource/inspect/) give you the
ability to freely perform troubleshooting, performance analyis and forensics.
### Google Cloud Monitoring
Google Cloud Monitoring is a hosted monitoring service you can use to
visualize and alert on important metrics in your application. You can collect
metrics from Kubernetes, and you can access them
using the [Cloud Monitoring Console](https://app.google.stackdriver.com/).
You can create and customize dashboards to visualize the data gathered
from your Kubernetes cluster.
This video shows how to configure and run a Google Cloud Monitoring backed Heapster:
[![how to setup and run a Google Cloud Monitoring backed Heapster](https://img.youtube.com/vi/xSMNR2fcoLs/0.jpg)](https://www.youtube.com/watch?v=xSMNR2fcoLs)
{{< figure src="/images/docs/gcm.png" alt="Google Cloud Monitoring dashboard example" title="Google Cloud Monitoring dashboard example" caption="This dashboard shows cluster-wide resource usage." >}}
## CronJob monitoring
### Kubernetes Job Monitor
With the [Kubernetes Job Monitor](https://github.com/pietervogelaar/kubernetes-job-monitor) dashboard a Cluster Administrator can see which jobs are running and view the status of completed jobs.
### New Relic Kubernetes monitoring integration
[New Relic Kubernetes](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration) integration provides increased visibility into the performance of your Kubernetes environment. New Relic's Kubernetes integration instruments the container orchestration layer by reporting metrics from Kubernetes objects. The integration gives you insight into your Kubernetes nodes, namespaces, deployments, replica sets, pods, and containers.
Marquee capabilities:
View your data in pre-built dashboards for immediate insight into your Kubernetes environment.
Create your own custom queries and charts in Insights from automatically reported data.
Create alert conditions on Kubernetes data.
Learn more on this [page](https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration).
[Prometheus](https://prometheus.io), a CNCF project, can natively monitor Kubernetes, nodes, and Prometheus itself.
Full metrics pipeline projects that are not part of the CNCF are outside the scope of Kubernetes documentation.
{{% /capture %}}
@@ -150,7 +150,7 @@ Example:
```bash
# create a plugin
echo '#!/bin/bash\n\necho "My first command-line argument was $1"' > kubectl-foo-bar-baz
echo -e '#!/bin/bash\n\necho "My first command-line argument was $1"' > kubectl-foo-bar-baz
sudo chmod +x ./kubectl-foo-bar-baz
# "install" our plugin by placing it on our PATH
@@ -185,7 +185,7 @@ Example:
```bash
# create a plugin containing an underscore in its filename
echo '#!/bin/bash\n\necho "I am a plugin with a dash in my name"' > ./kubectl-foo_bar
echo -e '#!/bin/bash\n\necho "I am a plugin with a dash in my name"' > ./kubectl-foo_bar
sudo chmod +x ./kubectl-foo_bar
# move the plugin into your PATH
@@ -287,7 +287,7 @@ For example, if you had your monitoring system collecting metrics about network
you could update the definition above using `kubectl edit` to look like this:
```yaml
apiVersion: autoscaling/v2beta1
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
@@ -311,7 +311,7 @@ The following commands are equivalent:
```shell
kubectl patch deployment patch-demo --patch "$(cat patch-file.yaml)"
kubectl patch deployment patch-demo --patch "$(cat patch-file.json)"
kubectl patch deployment patch-demo --patch 'spec:\n template:\n spec:\n containers:\n - name: patch-demo-ctr-2\n image: redis'
kubectl patch deployment patch-demo --patch "$(cat patch-file.json)"
@@ -44,7 +44,7 @@ chain and adding the parsed certificates to the `RootCAs` field in the
[`tls.Config`](https://godoc.org/crypto/tls#Config) struct.
You can distribute the CA certificate as a
[ConfigMap](/docs/tasks/configure-pod-container/configure-pod-config) that your
[ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap) that your
pods have access to use.
## Requesting a Certificate