whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into release-1.9

Browse Source
This commit is contained in:
zacharysarah
2017-12-15 13:11:00 -06:00
parent 989acbf6f8 0e628e7895
commit c4ea942a40
13 changed files with 273 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tasks/debug-application-cluster/audit.md
+154 -96
View File
@@ -37,11 +37,48 @@ of rules in order. The first matching rule sets the [audit level][auditing-level
of the event. The audit policy object structure is defined in the
[`audit.k8s.io` API group][auditing-api].
You can pass a file with the policy to [kube-apiserver][kube-apiserver]
using the `--audit-policy-file` flag. If the flag is omitted, no events are logged.
__Note:__ `kind` and `apiVersion` fields along with `rules` __must__ be provided
in the audit policy file. A policy with no (0) rules, or a policy that doesn't
provide valid `apiVersion` and `kind` values is treated as illegal.
`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded,
and backends, which persist records. Backend implementations include logs files and
webhooks.
The structure of audit events changes when enabling the `AdvancedAuditing` feature
flag. This includes some cleanups, such as the `method` reflecting the verb evaluated
by the [authorization layer](/docs/admin/authorization/) instead of the [HTTP verb](/docs/admin/authorization/#determine-the-request-verb).
Also, instead of always generating two events per request, events are recorded with an associated "stage".
The known stages are:
- `RequestReceived` - The stage for events generated as soon as the audit handler receives the request.
- `ResponseStarted` - Once the response headers are sent, but before the response body is sent. This stage is only generated for long-running requests (e.g. watch).
- `ResponseComplete` - Once the response body has been completed.
- `Panic` - Events generated when a panic occurred.
### Audit Policy
Audit policy is a document defining rules about what events should be recorded.
The policy is passed to the [kube-apiserver][kube-apiserver] using the
`--audit-policy-file` flag.
```
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
```
If `AdvancedAuditing` is enabled and this flag is omitted, no events are logged.
The policy file holds rules that determine the level of an event. Known audit levels are:
- `None` - don't log events that match this rule.
- `Metadata` - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
- `Request` - log event metadata and request body but not response body.
- `RequestResponse` - log event metadata, request and response bodies.
When an event is processed, it's compared against the list of rules in order.
The first matching rule sets the audit level of the event. The audit policy is
defined by the [`audit.k8s.io` API group][audit-api].
Some new fields are supported in beta version, like `resourceNames` and `omitStages`.
In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be provided in
the audit policy file. A policy file with 0 rules, or a policy file that doesn't provide
a valid `apiVersion` and `kind` value will be treated as illegal.
Some example audit policy files:
@@ -109,57 +146,66 @@ In this example, we will use fluentd to split audit events by different namespac
1. install [fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter][fluentd_install_doc] in the kube-apiserver node
1. create a config file for fluentd
$ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
rewriterule1 namespace ^(.+) ${tag}.$1
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
```shell
$ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
rewriterule1 namespace ^(.+) ${tag}.$1
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
```
1. start fluentd
$ fluentd -c /etc/fluentd/config -vv
```shell
$ fluentd -c /etc/fluentd/config -vv
```
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
```
1. check audits for different namespaces in /var/log/audit-*.log
### Use logstash to collect and distribute audit events from webhook backend
@@ -171,56 +217,68 @@ different users into different files.
1. install [logstash][logstash_install_doc]
1. create config file for logstash
$ cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
```shell
$ cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
```
1. start logstash
$ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
```shell
$ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
```
1. create a [kubeconfig file](/docs/tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/) for kube-apiserver webhook audit backend
$ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF
```shell
$ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF
```
1. start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```
1. check audits in logstash node's directories /var/log/kube-audit-*/audit
Note that in addition to file output plugin, logstash has a variety of outputs that