whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into dev-1.20 to keep in sync - 11-25-2020

Browse Source
This commit is contained in:
reylejano-rxm
2020-11-25 07:03:22 -08:00
parent 6fa3e91eb8 b4df624a5e
commit d8ae37587e
139 changed files with 6562 additions and 5229 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/administer-cluster/cluster-management.md
-223
View File
@@ -1,223 +0,0 @@
---
reviewers:
- lavalamp
- thockin
title: Cluster Management
content_type: concept
---
<!-- overview -->
This document describes several topics related to the lifecycle of a cluster: creating a new cluster,
upgrading your cluster's
master and worker nodes, performing node maintenance (e.g. kernel upgrades), and upgrading the Kubernetes API version of a
running cluster.
<!-- body -->
## Creating and configuring a Cluster
To install Kubernetes on a set of machines, consult one of the existing [Getting Started guides](/docs/setup/) depending on your environment.
## Upgrading a cluster
The current state of cluster upgrades is provider dependent, and some releases may require special care when upgrading. It is recommended that administrators consult both the [release notes](https://git.k8s.io/kubernetes/CHANGELOG/README.md), as well as the version specific upgrade notes prior to upgrading their clusters.
### Upgrading an Azure Kubernetes Service (AKS) cluster
Azure Kubernetes Service enables easy self-service upgrades of the control plane and nodes in your cluster. The process is
currently user-initiated and is described in the [Azure AKS documentation](https://docs.microsoft.com/en-us/azure/aks/upgrade-cluster).
### Upgrading Google Compute Engine clusters
Google Compute Engine Open Source (GCE-OSS) support master upgrades by deleting and
recreating the master, while maintaining the same Persistent Disk (PD) to ensure that data is retained across the
upgrade.
Node upgrades for GCE use a [Managed Instance Group](https://cloud.google.com/compute/docs/instance-groups/), each node
is sequentially destroyed and then recreated with new software. Any Pods that are running on that node need to be
controlled by a Replication Controller, or manually re-created after the roll out.
Upgrades on open source Google Compute Engine (GCE) clusters are controlled by the `cluster/gce/upgrade.sh` script.
Get its usage by running `cluster/gce/upgrade.sh -h`.
For example, to upgrade just your master to a specific version (v1.0.2):
```shell
cluster/gce/upgrade.sh -M v1.0.2
```
Alternatively, to upgrade your entire cluster to the latest stable release:
```shell
cluster/gce/upgrade.sh release/stable
```
### Upgrading Google Kubernetes Engine clusters
Google Kubernetes Engine automatically updates master components (e.g. `kube-apiserver`, `kube-scheduler`) to the latest version. It also handles upgrading the operating system and other components that the master runs on.
The node upgrade process is user-initiated and is described in the [Google Kubernetes Engine documentation](https://cloud.google.com/kubernetes-engine/docs/clusters/upgrade).
### Upgrading an Amazon EKS Cluster
Amazon EKS cluster's master components can be upgraded by using eksctl, AWS Management Console, or AWS CLI. The process is user-initiated and is described in the [Amazon EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html).
### Upgrading an Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) cluster
Oracle creates and manages a set of master nodes in the Oracle control plane on your behalf (and associated Kubernetes infrastructure such as etcd nodes) to ensure you have a highly available managed Kubernetes control plane. You can also seamlessly upgrade these master nodes to new versions of Kubernetes with zero downtime. These actions are described in the [OKE documentation](https://docs.cloud.oracle.com/iaas/Content/ContEng/Tasks/contengupgradingk8smasternode.htm).
### Upgrading clusters on other platforms
Different providers, and tools, will manage upgrades differently. It is recommended that you consult their main documentation regarding upgrades.
* [kops](https://github.com/kubernetes/kops)
* [kubespray](https://github.com/kubernetes-sigs/kubespray)
* [CoreOS Tectonic](https://coreos.com/tectonic/docs/latest/admin/upgrade.html)
* [Digital Rebar](https://provision.readthedocs.io/en/tip/doc/content-packages/krib.html)
* ...
To upgrade a cluster on a platform not mentioned in the above list, check the order of component upgrade on the
[Skewed versions](/docs/setup/release/version-skew-policy/#supported-component-upgrade-order) page.
## Resizing a cluster
If your cluster runs short on resources you can easily add more machines to it if your cluster
is running in [Node self-registration mode](/docs/concepts/architecture/nodes/#self-registration-of-nodes).
If you're using GCE or Google Kubernetes Engine it's done by resizing the Instance Group managing your Nodes.
It can be accomplished by modifying number of instances on
`Compute > Compute Engine > Instance groups > your group > Edit group`
[Google Cloud Console page](https://console.developers.google.com) or using gcloud CLI:
```shell
gcloud compute instance-groups managed resize kubernetes-node-pool --size=42 --zone=$ZONE
```
The Instance Group will take care of putting appropriate image on new machines and starting them,
while the Kubelet will register its Node with the API server to make it available for scheduling.
If you scale the instance group down, system will randomly choose Nodes to kill.
In other environments you may need to configure the machine yourself and tell the Kubelet on which machine API server is running.
### Resizing an Azure Kubernetes Service (AKS) cluster
Azure Kubernetes Service enables user-initiated resizing of the cluster from either the CLI or
the Azure Portal and is described in the
[Azure AKS documentation](https://docs.microsoft.com/en-us/azure/aks/scale-cluster).
### Cluster autoscaling
If you are using GCE or Google Kubernetes Engine, you can configure your cluster so that it is automatically rescaled based on
pod needs.
As described in [Compute Resource](/docs/concepts/configuration/manage-resources-containers/),
users can reserve how much CPU and memory is allocated to pods.
This information is used by the Kubernetes scheduler to find a place to run the pod. If there is
no node that has enough free capacity (or doesn't match other pod requirements) then the pod has
to wait until some pods are terminated or a new node is added.
Cluster autoscaler looks for the pods that cannot be scheduled and checks if adding a new node, similar
to the other in the cluster, would help. If yes, then it resizes the cluster to accommodate the waiting pods.
Cluster autoscaler also scales down the cluster if it notices that one or more nodes are not needed anymore for
an extended period of time (10min but it may change in the future).
Cluster autoscaler is configured per instance group (GCE) or node pool (Google Kubernetes Engine).
If you are using GCE then you can either enable it while creating a cluster with kube-up.sh script.
To configure cluster autoscaler you have to set three environment variables:
* `KUBE_ENABLE_CLUSTER_AUTOSCALER` - it enables cluster autoscaler if set to true.
* `KUBE_AUTOSCALER_MIN_NODES` - minimum number of nodes in the cluster.
* `KUBE_AUTOSCALER_MAX_NODES` - maximum number of nodes in the cluster.
Example:
```shell
KUBE_ENABLE_CLUSTER_AUTOSCALER=true KUBE_AUTOSCALER_MIN_NODES=3 KUBE_AUTOSCALER_MAX_NODES=10 NUM_NODES=5 ./cluster/kube-up.sh
```
On Google Kubernetes Engine you configure cluster autoscaler either on cluster creation or update or when creating a particular node pool
(which you want to be autoscaled) by passing flags `--enable-autoscaling` `--min-nodes` and `--max-nodes`
to the corresponding `gcloud` commands.
Examples:
```shell
gcloud container clusters create mytestcluster --zone=us-central1-b --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
```
```shell
gcloud container clusters update mytestcluster --enable-autoscaling --min-nodes=1 --max-nodes=15
```
**Cluster autoscaler expects that nodes have not been manually modified (e.g. by adding labels via kubectl) as those properties would not be propagated to the new nodes within the same instance group.**
For more details about how the cluster autoscaler decides whether, when and how
to scale a cluster, please refer to the [FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md)
documentation from the autoscaler project.
## Maintenance on a Node
If you need to reboot a node (such as for a kernel upgrade, libc upgrade, hardware repair, etc.), and the downtime is
brief, then when the Kubelet restarts, it will attempt to restart the pods scheduled to it. If the reboot takes longer
(the default time is 5 minutes, controlled by `--pod-eviction-timeout` on the controller-manager),
then the node controller will terminate the pods that are bound to the unavailable node. If there is a corresponding
replica set (or replication controller), then a new copy of the pod will be started on a different node. So, in the case where all
pods are replicated, upgrades can be done without special coordination, assuming that not all nodes will go down at the same time.
If you want more control over the upgrading process, you may use the following workflow:
Use `kubectl drain` to gracefully terminate all pods on the node while marking the node as unschedulable:
```shell
kubectl drain $NODENAME
```
This keeps new pods from landing on the node while you are trying to get them off.
For pods with a replica set, the pod will be replaced by a new pod which will be scheduled to a new node. Additionally, if the pod is part of a service, then clients will automatically be redirected to the new pod.
For pods with no replica set, you need to bring up a new copy of the pod, and assuming it is not part of a service, redirect clients to it.
Perform maintenance work on the node.
Make the node schedulable again:
```shell
kubectl uncordon $NODENAME
```
If you deleted the node's VM instance and created a new one, then a new schedulable node resource will
be created automatically (if you're using a cloud provider that supports
node discovery; currently this is only Google Compute Engine, not including CoreOS on Google Compute Engine using kube-register).
See [Node](/docs/concepts/architecture/nodes/) for more details.
## Advanced Topics
### Turn on or off an API version for your cluster
Specific API versions can be turned on or off by passing `--runtime-config=api/<version>` flag while bringing up the API server. For example: to turn off v1 API, pass `--runtime-config=api/v1=false`.
runtime-config also supports 2 special keys: api/all and api/legacy to control all and legacy APIs respectively.
For example, for turning off all API versions except v1, pass `--runtime-config=api/all=false,api/v1=true`.
For the purposes of these flags, _legacy_ APIs are those APIs which have been explicitly deprecated (e.g. `v1beta3`).
### Switching your cluster's storage API version
The objects that are stored to disk for a cluster's internal representation of the Kubernetes resources active in the cluster are written using a particular version of the API.
When the supported API changes, these objects may need to be rewritten in the newer API. Failure to do this will eventually result in resources that are no longer decodable or usable
by the Kubernetes API server.
### Switching your config files to a new API version
You can use `kubectl convert` command to convert config files between different API versions.
```shell
kubectl convert -f pod.yaml --output-version v1
```
For more options, please refer to the usage of [kubectl convert](/docs/reference/generated/kubectl/kubectl-commands#convert) command.
content/en/docs/tasks/administer-cluster/cluster-upgrade.md
+93
View File
@@ -0,0 +1,93 @@
---
title: Upgrade A Cluster
content_type: task
---
<!-- overview -->
This page provides an overview of the steps you should follow to upgrade a
Kubernetes cluster.
The way that you upgrade a cluster depends on how you initially deployed it
and on any subsequent changes.
At a high level, the steps you perform are:
- Upgrade the {{< glossary_tooltip text="control plane" term_id="control-plane" >}}
- Upgrade the nodes in your cluster
- Upgrade clients such as {{< glossary_tooltip text="kubectl" term_id="kubectl" >}}
- Adjust manifests and other resources based on the API changes that accompany the
new Kubernetes version
## {{% heading "prerequisites" %}}
You must have an existing cluster. This page is about upgrading from Kubernetes
{{< skew prevMinorVersion >}} to Kubernetes {{< skew latestVersion >}}. If your cluster
is not currently running Kubernetes {{< skew prevMinorVersion >}} then please check
the documentation for the version of Kubernetes that you plan to upgrade to.
## Upgrade approaches
### kubeadm {#upgrade-kubeadm}
If your cluster was deployed using the `kubeadm` tool, refer to
[Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)
for detailed information on how to upgrade the cluster.
Once you have upgraded the cluster, remember to
[install the latest version of `kubectl`](/docs/tasks/tools/install-kubectl/).
### Manual deployments
{{< caution >}}
These steps do not account for third-party extensions such as network and storage
plugins.
{{< /caution >}}
You should manually update the control plane following this sequence:
- etcd (all instances)
- kube-apiserver (all control plane hosts)
- kube-controller-manager
- kube-scheduler
- cloud controller manager, if you use one
At this point you should
[install the latest version of `kubectl`](/docs/tasks/tools/install-kubectl/).
For each node in your cluster, [drain](/docs/tasks/administer-cluster/safely-drain-node/)
that node and then either replace it with a new node that uses the {{< skew latestVersion >}}
kubelet, or upgrade the kubelet on that node and bring the node back into service.
### Other deployments {#upgrade-other}
Refer to the documentation for your cluster deployment tool to learn the recommended set
up steps for maintenance.
## Post-upgrade tasks
### Switch your cluster's storage API version
The objects that are serialized into etcd for a cluster's internal
representation of the Kubernetes resources active in the cluster are
written using a particular version of the API.
When the supported API changes, these objects may need to be rewritten
in the newer API. Failure to do this will eventually result in resources
that are no longer decodable or usable by the Kubernetes API server.
For each affected object, fetch it using the latest supported API and then
write it back also using the latest supported API.
### Update manifests
Upgrading to a new Kubernetes version can provide new APIs.
You can use `kubectl convert` command to convert manifests between different API versions.
For example:
```shell
kubectl convert -f pod.yaml --output-version v1
```
The `kubectl` tool replaces the contents of `pod.yaml` with a manifest that sets `kind` to
Pod (unchanged), but with a revised `apiVersion`.
content/en/docs/tasks/administer-cluster/enable-disable-api.md
+29
View File
@@ -0,0 +1,29 @@
---
title: Enable Or Disable A Kubernetes API
content_type: task
---
<!-- overview -->
This page shows how to enable or disable an API version from your cluster's
{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
<!-- steps -->
Specific API versions can be turned on or off by passing `--runtime-config=api/<version>` as a
command line argument to the API server. The values for this argument are a comma-separated
list of API versions. Later values override earlier values.
The `runtime-config` command line argument also supports 2 special keys:
- `api/all`, representing all known APIs
- `api/legacy`, representing only legacy APIs. Legacy APIs are any APIs that have been
explicitly [deprecated](/docs/reference/using-api/deprecation-policy/).
For example, to turning off all API versions except v1, pass `--runtime-config=api/all=false,api/v1=true`
to the `kube-apiserver`.
## {{% heading "whatsnext" %}}
Read the [full documentation](/docs/reference/command-line-tools-reference/kube-apiserver/)
for the `kube-apiserver` component.
content/en/docs/tasks/administer-cluster/safely-drain-node.md
+11 -5
View File
@@ -4,14 +4,14 @@ reviewers:
- mml
- foxish
- kow3ns
title: Safely Drain a Node while Respecting the PodDisruptionBudget
title: Safely Drain a Node
content_type: task
min-kubernetes-server-version: 1.5
---
<!-- overview -->
This page shows how to safely drain a {{< glossary_tooltip text="node" term_id="node" >}},
respecting the PodDisruptionBudget you have defined.
optionally respecting the PodDisruptionBudget you have defined.
## {{% heading "prerequisites" %}}
@@ -27,6 +27,15 @@ This task also assumes that you have met the following prerequisites:
<!-- steps -->
## (Optional) Configure a disruption budget {#configure-poddisruptionbudget}
To endure that your workloads remain available during maintenance, you can
configure a [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/).
If availability is important for any applications that run or could run on the node(s)
that you are draining, [configure a PodDisruptionBudgets](/docs/tasks/run-application/configure-pdb/)
first and the continue following this guide.
## Use `kubectl drain` to remove a node from service
You can use `kubectl drain` to safely evict all of your pods from a
@@ -158,7 +167,4 @@ application owners and cluster owners to establish an agreement on behavior in t
* Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/).
* Learn more about [maintenance on a node](/docs/tasks/administer-cluster/cluster-management/#maintenance-on-a-node).
content/en/docs/tasks/debug-application-cluster/audit.md
+60 -225
View File
@@ -9,10 +9,11 @@ title: Auditing
<!-- overview -->
Kubernetes auditing provides a security-relevant chronological set of records documenting
the sequence of activities that have affected system by individual users, administrators
or other components of the system. It allows cluster administrator to
answer the following questions:
Kubernetes _auditing_ provides a security-relevant, chronological set of records documenting
the sequence of actions in a cluster. The cluster audits the activities generated by users,
by applications that use the Kubernetes API, and by the control plane itself.
Auditing allows cluster administrators to answer the following questions:
- what happened?
- when did it happen?
@@ -32,7 +33,7 @@ a certain policy and written to a backend. The policy determines what's recorded
and the backends persist the records. The current backend implementations
include logs files and webhooks.
Each request can be recorded with an associated "stage". The known stages are:
Each request can be recorded with an associated _stage_. The defined stages are:
- `RequestReceived` - The stage for events generated as soon as the audit
handler receives the request, and before it is delegated down the handler
@@ -45,19 +46,23 @@ Each request can be recorded with an associated "stage". The known stages are:
- `Panic` - Events generated when a panic occurred.
{{< note >}}
The audit logging feature increases the memory consumption of the API server
because some context required for auditing is stored for each request.
Additionally, memory consumption depends on the audit logging configuration.
Audit events are different from the
[Event](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#event-v1-core)
API object.
{{< /note >}}
## Audit Policy
The audit logging feature increases the memory consumption of the API server
because some context required for auditing is stored for each request.
Memory consumption depends on the audit logging configuration.
## Audit policy
Audit policy defines rules about what events should be recorded and what data
they should include. The audit policy object structure is defined in the
[`audit.k8s.io` API group](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go).
When an event is processed, it's
compared against the list of rules in order. The first matching rule sets the
"audit level" of the event. The known audit levels are:
_audit level_ of the event. The defined audit levels are:
- `None` - don't log events that match this rule.
- `Metadata` - log request metadata (requesting user, timestamp, resource,
@@ -86,26 +91,27 @@ rules:
- level: Metadata
```
The audit profile used by GCE should be used as reference by admins constructing their own audit profiles. You can check the
If you're crafting your own audit profile, you can use the audit profile for Google Container-Optimized OS as a starting point. You can check the
[configure-helper.sh](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh)
script, which generates the audit policy file. You can see most of the audit policy file by looking directly at the script.
script, which generates an audit policy file. You can see most of the audit policy file by looking directly at the script.
## Audit backends
Audit backends persist audit events to an external storage.
Out of the box, the kube-apiserver provides two backends:
- Log backend, which writes events to a disk
- Webhook backend, which sends events to an external API
- Log backend, which writes events into the filesystem
- Webhook backend, which sends events to an external HTTP API
In all cases, audit events structure is defined by the API in the
`audit.k8s.io` API group. The current version of the API is
In all cases, audit events follow a structure defined by the Kubernetes API in the
`audit.k8s.io` API group. For Kubernetes {{< param "fullversion" >}}, that
API is at version
[`v1`](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go).
{{< note >}}
In case of patches, request body is a JSON array with patch operations, not a JSON object
with an appropriate Kubernetes API object. For example, the following request body is a valid patch
request to `/apis/batch/v1/namespaces/some-namespace/jobs/some-job-name`.
request to `/apis/batch/v1/namespaces/some-namespace/jobs/some-job-name`:
```json
[
@@ -125,8 +131,8 @@ request to `/apis/batch/v1/namespaces/some-namespace/jobs/some-job-name`.
### Log backend
Log backend writes audit events to a file in JSON format. You can configure
log audit backend using the following `kube-apiserver` flags:
The log backend writes audit events to a file in [JSONlines](https://jsonlines.org/) format.
You can configure the log audit backend using the following `kube-apiserver` flags:
- `--audit-log-path` specifies the log file path that log backend uses to write
audit events. Not specifying this flag disables log backend. `-` means standard out
@@ -134,15 +140,16 @@ log audit backend using the following `kube-apiserver` flags:
- `--audit-log-maxbackup` defines the maximum number of audit log files to retain
- `--audit-log-maxsize` defines the maximum size in megabytes of the audit log file before it gets rotated
In case kube-apiserver is configured as a Pod,remember to mount the hostPath to the location of the policy file and log file. For example,
`
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/audit.log
`
If your cluster's control plane runs the kube-apiserver as a Pod, remember to mount the `hostPath`
to the location of the policy file and log file, so that audit records are persisted. For example:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--audit-log-path=/var/log/audit.log
```
then mount the volumes:
```
```yaml
...
volumeMounts:
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
@@ -151,9 +158,10 @@ volumeMounts:
name: audit-log
readOnly: false
```
finally the hostPath:
and finally configure the `hostPath`:
```
```yaml
...
- name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
@@ -163,19 +171,19 @@ finally the hostPath:
hostPath:
path: /var/log/audit.log
type: FileOrCreate
```
### Webhook backend
Webhook backend sends audit events to a remote API, which is assumed to be the
same API as `kube-apiserver` exposes. You can configure webhook
audit backend using the following kube-apiserver flags:
The webhook audit backend sends audit events to a remote web API, which is assumed to
be a form of the Kubernetes API, including means of authentication. You can configure
a webhook audit backend using the following kube-apiserver flags:
- `--audit-webhook-config-file` specifies the path to a file with a webhook
configuration. Webhook configuration is effectively a
configuration. The webhook configuration is effectively a specialized
[kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters).
- `--audit-webhook-initial-backoff` specifies the amount of time to wait after the first failed
request before retrying. Subsequent requests are retried with exponential backoff.
@@ -183,7 +191,7 @@ audit backend using the following kube-apiserver flags:
The webhook config file uses the kubeconfig format to specify the remote address of
the service and credentials used to connect to it.
### Batching
## Event batching {#batching}
Both log and webhook backends support batching. Using webhook as an example, here's the list of
available flags. To get the same flag for log backend, replace `webhook` with `log` in the flag
@@ -193,9 +201,10 @@ throttling is enabled in `webhook` and disabled in `log`.
- `--audit-webhook-mode` defines the buffering strategy. One of the following:
- `batch` - buffer events and asynchronously process them in batches. This is the default.
- `blocking` - block API server responses on processing each individual event.
- `blocking-strict` - Same as blocking, but when there is a failure during audit logging at RequestReceived stage, the whole request to apiserver will fail.
- `blocking-strict` - Same as blocking, but when there is a failure during audit logging at the
RequestReceived stage, the whole request to the kube-apiserver fails.
The following flags are used only in the `batch` mode.
The following flags are used only in the `batch` mode:
- `--audit-webhook-batch-buffer-size` defines the number of events to buffer before batching.
If the rate of incoming events overflows the buffer, events are dropped.
@@ -207,16 +216,16 @@ The following flags are used only in the `batch` mode.
- `--audit-webhook-batch-throttle-burst` defines the maximum number of batches generated at the same
moment if the allowed QPS was underutilized previously.
#### Parameter tuning
## Parameter tuning
Parameters should be set to accommodate the load on the apiserver.
Parameters should be set to accommodate the load on the API server.
For example, if kube-apiserver receives 100 requests each second, and each request is audited only
on `ResponseStarted` and `ResponseComplete` stages, you should account for ~200 audit
on `ResponseStarted` and `ResponseComplete` stages, you should account for ≅200 audit
events being generated each second. Assuming that there are up to 100 events in a batch,
you should set throttling level at least 2 QPS. Assuming that the backend can take up to
5 seconds to write events, you should set the buffer size to hold up to 5 seconds of events, i.e.
10 batches, i.e. 1000 events.
you should set throttling level at least 2 queries per second. Assuming that the backend can take up to
5 seconds to write events, you should set the buffer size to hold up to 5 seconds of events;
that is: 10 batches, or 1000 events.
In most cases however, the default parameters should be sufficient and you don't have to worry about
setting them manually. You can look at the following Prometheus metrics exposed by kube-apiserver
@@ -226,192 +235,18 @@ and in the logs to monitor the state of the auditing subsystem.
- `apiserver_audit_error_total` metric contains the total number of events dropped due to an error
during exporting.
### Truncate
### Log entry truncation {#truncate}
Both log and webhook backends support truncating. As an example, the following is the list of flags
available for the log backend:
Both log and webhook backends support limiting the size of events that are logged.
As an example, the following is the list of flags available for the log backend:
- `audit-log-truncate-enabled` whether event and batch truncating is enabled.
- `audit-log-truncate-max-batch-size` maximum size in bytes of the batch sent to the underlying backend.
- `audit-log-truncate-max-event-size` maximum size in bytes of the audit event sent to the underlying backend.
By default truncate is disabled in both `webhook` and `log`, a cluster administrator should set `audit-log-truncate-enabled` or `audit-webhook-truncate-enabled` to enable the feature.
## Setup for multiple API servers
If you're extending the Kubernetes API with the [aggregation
layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/),
you can also set up audit logging for the aggregated apiserver. To do this,
pass the configuration options in the same format as described above to the
aggregated apiserver and set up the log ingesting pipeline to pick up audit
logs. Different apiservers can have different audit configurations and
different audit policies.
## Log Collector Examples
### Use fluentd to collect and distribute audit events from log file
[Fluentd](https://www.fluentd.org/) is an open source data collector for unified logging layer.
In this example, we will use fluentd to split audit events by different namespaces.
{{< note >}}
The `fluent-plugin-forest` and `fluent-plugin-rewrite-tag-filter` are plugins for fluentd.
You can get details about plugin installation from
[fluentd plugin-management](https://docs.fluentd.org/v1.0/articles/plugin-management).
{{< /note >}}
1. Install [`fluentd`](https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd),
`fluent-plugin-forest` and `fluent-plugin-rewrite-tag-filter` in the kube-apiserver node
1. Create a config file for fluentd
```
cat <<'EOF' > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
@type tail
# audit log path of kube-apiserver
path /var/log/kube-audit
pos_file /var/log/audit.pos
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%N%z
tag audit
</source>
<filter audit>
#https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
@type record_transformer
enable_ruby
<record>
namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ? "none":record["objectRef"]["namespace"])}
</record>
</filter>
<match audit>
# route audit according to namespace element in context
@type rewrite_tag_filter
<rule>
key namespace
pattern /^(.+)/
tag ${tag}.$1
</rule>
</match>
<filter audit.**>
@type record_transformer
remove_keys namespace
</filter>
<match audit.**>
@type forest
subtype file
remove_prefix audit
<template>
time_slice_format %Y%m%d%H
compress gz
path /var/log/audit-${tag}.*.log
format json
include_time_key true
</template>
</match>
EOF
```
1. Start fluentd
```shell
fluentd -c /etc/fluentd/config -vv
```
1. Start kube-apiserver with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
```
1. Check audits for different namespaces in `/var/log/audit-*.log`
### Use logstash to collect and distribute audit events from webhook backend
[Logstash](https://www.elastic.co/products/logstash)
is an open source, server-side data processing tool. In this example,
we will use logstash to collect audit events from webhook backend, and save events of
different users into different files.
1. install [logstash](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html)
1. create config file for logstash
```
cat <<EOF > /etc/logstash/config
input{
http{
#TODO, figure out a way to use kubeconfig file to authenticate to logstash
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
port=>8888
}
}
filter{
split{
# Webhook audit backend sends several events together with EventList
# split each event here.
field=>[items]
# We only need event subelement, remove others.
remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
}
mutate{
rename => {items=>event}
}
}
output{
file{
# Audit events from different users will be saved into different files.
path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
}
}
EOF
```
1. start logstash
```shell
bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/
```
1. create a [kubeconfig file](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) for kube-apiserver webhook audit backend
cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
server: http://<ip_of_logstash>:8888
name: logstash
contexts:
- context:
cluster: logstash
user: ""
name: default-context
current-context: default-context
preferences: {}
users: []
EOF
1. start kube-apiserver with the following options:
```shell
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
```
1. check audits in logstash node's directories `/var/log/kube-audit-*/audit`
Note that in addition to file output plugin, logstash has a variety of outputs that
let users route data where they want. For example, users can emit audit events to elasticsearch
plugin which supports full-text search and analytics.
- `audit-log-truncate-enabled` whether event and batch truncating is enabled.
- `audit-log-truncate-max-batch-size` maximum size in bytes of the batch sent to the underlying backend.
- `audit-log-truncate-max-event-size` maximum size in bytes of the audit event sent to the underlying backend.
By default truncate is disabled in both `webhook` and `log`, a cluster administrator should set
`audit-log-truncate-enabled` or `audit-webhook-truncate-enabled` to enable the feature.
## {{% heading "whatsnext" %}}
Learn about [Mutating webhook auditing annotations](/docs/reference/access-authn-authz/extensible-admission-controllers/#mutating-webhook-auditing-annotations).
* Learn about [Mutating webhook auditing annotations](/docs/reference/access-authn-authz/extensible-admission-controllers/#mutating-webhook-auditing-annotations).
content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md
+1 -1
View File
@@ -47,7 +47,7 @@ can not schedule your pod. Reasons include:
You may have exhausted the supply of CPU or Memory in your cluster. In this
case you can try several things:
* [Add more nodes](/docs/tasks/administer-cluster/cluster-management/#resizing-a-cluster) to the cluster.
* Add more nodes to the cluster.
* [Terminate unneeded pods](/docs/concepts/workloads/pods/#pod-termination)
to make room for pending pods.
content/en/docs/tasks/extend-kubectl/kubectl-plugins.md
+3 -3
View File
@@ -32,7 +32,7 @@ using [Krew](https://krew.dev/). Krew is a plugin manager maintained by
the Kubernetes SIG CLI community.
{{< caution >}}
Kubectl plugins available via the Krew [plugin index](https://index.krew.dev/)
Kubectl plugins available via the Krew [plugin index](https://krew.sigs.k8s.io/plugins/)
are not audited for security. You should install and run third-party plugins at your
own risk, since they are arbitrary programs running on your machine.
{{< /caution >}}
@@ -46,7 +46,7 @@ A warning will also be included for any valid plugin files that overlap each oth
You can use [Krew](https://krew.dev/) to discover and install `kubectl`
plugins from a community-curated
[plugin index](https://index.krew.dev/).
[plugin index](https://krew.sigs.k8s.io/plugins/).
#### Limitations
@@ -354,7 +354,7 @@ package it, distribute it and deliver updates to your users.
distribute your plugins. This way, you use a single packaging format for all
target platforms (Linux, Windows, macOS etc) and deliver updates to your users.
Krew also maintains a [plugin
index](https://index.krew.dev/) so that other people can
index](https://krew.sigs.k8s.io/plugins/) so that other people can
discover your plugin and install it.
content/en/docs/tasks/tls/manual-rotation-of-ca-certificates.md
+7 -4
View File
@@ -33,14 +33,17 @@ Configurations with a single API server will experience unavailability while the
(ex: `ca.crt`, `ca.key`, `front-proxy-ca.crt`, and `front-proxy-ca.key`)
to all your control plane nodes in the Kubernetes certificates directory.
1. Update *Kubernetes controller manager's* `--root-ca-file` to include both old and new CA and restart controller manager.
1. Update {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}'s `--root-ca-file` to
include both old and new CA. Then restart the component.
Any service account created after this point will get secrets that include both old and new CAs.
{{< note >}}
Remove the flag `--client-ca-file` from the *Kubernetes controller manager* configuration.
You can also replace the existing client CA file or change this configuration item to reference a new, updated CA.
[Issue 1350](https://github.com/kubernetes/kubeadm/issues/1350) tracks an issue with *Kubernetes controller manager* being unable to accept a CA bundle.
The files specified by the kube-controller-manager flags `--client-ca-file` and `--cluster-signing-cert-file`
cannot be CA bundles. If these flags and `--root-ca-file` point to the same `ca.crt` file which is now a
bundle (includes both old and new CA) you will face an error. To workaround this problem you can copy the new CA to a separate
file and make the flags `--client-ca-file` and `--cluster-signing-cert-file` point to the copy. Once `ca.crt` is no longer
a bundle you can restore the problem flags to point to `ca.crt` and delete the copy.
{{< /note >}}
1. Update all service account tokens to include both old and new CA certificates.
content/en/docs/tasks/tools/_index.md
+1 -1
View File
@@ -23,7 +23,7 @@ You can also read the
## kind
[`kind`](https://kind.sigs.k8s.io/docs/) lets you run Kubernetes on
your local computer. This tool it requires that you have
your local computer. This tool requires that you have
[Docker](https://docs.docker.com/get-docker/) installed and configured.
The kind [Quick Start](https://kind.sigs.k8s.io/docs/user/quick-start/) page