Merge branch 'master' into issue#28062
This commit is contained in:
hardikshah197
+4
-4
@@ -23,7 +23,7 @@ of Containers for each.
|
||||
|
||||
- Fetch all Pods in all namespaces using `kubectl get pods --all-namespaces`
|
||||
- Format the output to include only the list of Container image names
|
||||
using `-o jsonpath={..image}`. This will recursively parse out the
|
||||
using `-o jsonpath={.items[*].spec.containers[*].image}`. This will recursively parse out the
|
||||
`image` field from the returned json.
|
||||
- See the [jsonpath reference](/docs/reference/kubectl/jsonpath/)
|
||||
for further information on how to use jsonpath.
|
||||
@@ -33,7 +33,7 @@ of Containers for each.
|
||||
- Use `uniq` to aggregate image counts
|
||||
|
||||
```shell
|
||||
kubectl get pods --all-namespaces -o jsonpath="{..image}" |\
|
||||
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
|
||||
tr -s '[[:space:]]' '\n' |\
|
||||
sort |\
|
||||
uniq -c
|
||||
@@ -80,7 +80,7 @@ To target only Pods matching a specific label, use the -l flag. The
|
||||
following matches only Pods with labels matching `app=nginx`.
|
||||
|
||||
```shell
|
||||
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx
|
||||
kubectl get pods --all-namespaces -o=jsonpath="{.items[*].spec.containers[*].image}" -l app=nginx
|
||||
```
|
||||
|
||||
## List Container images filtering by Pod namespace
|
||||
@@ -89,7 +89,7 @@ To target only pods in a specific namespace, use the namespace flag. The
|
||||
following matches only Pods in the `kube-system` namespace.
|
||||
|
||||
```shell
|
||||
kubectl get pods --namespace kube-system -o jsonpath="{..image}"
|
||||
kubectl get pods --namespace kube-system -o jsonpath="{.items[*].spec.containers[*].image}"
|
||||
```
|
||||
|
||||
## List Container images using a go-template instead of jsonpath
|
||||
|
||||
+54
-46
@@ -1,16 +1,17 @@
|
||||
---
|
||||
reviewers:
|
||||
- jszczepkowski
|
||||
title: Set up High-Availability Kubernetes Masters
|
||||
title: Set up a High-Availability Control Plane
|
||||
content_type: task
|
||||
aliases: [ '/docs/tasks/administer-cluster/highly-available-master/' ]
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
{{< feature-state for_k8s_version="v1.5" state="alpha" >}}
|
||||
|
||||
You can replicate Kubernetes masters in `kube-up` or `kube-down` scripts for Google Compute Engine.
|
||||
This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for use with GCE.
|
||||
You can replicate Kubernetes control plane nodes in `kube-up` or `kube-down` scripts for Google Compute Engine.
|
||||
This document describes how to use kube-up/down scripts to manage a highly available (HA) control plane and how HA control planes are implemented for use with GCE.
|
||||
|
||||
|
||||
|
||||
@@ -28,17 +29,17 @@ This document describes how to use kube-up/down scripts to manage highly availab
|
||||
|
||||
To create a new HA-compatible cluster, you must set the following flags in your `kube-up` script:
|
||||
|
||||
* `MULTIZONE=true` - to prevent removal of master replicas kubelets from zones different than server's default zone.
|
||||
Required if you want to run master replicas in different zones, which is recommended.
|
||||
* `MULTIZONE=true` - to prevent removal of control plane kubelets from zones different than server's default zone.
|
||||
Required if you want to run control plane nodes in different zones, which is recommended.
|
||||
|
||||
* `ENABLE_ETCD_QUORUM_READ=true` - to ensure that reads from all API servers will return most up-to-date data.
|
||||
If true, reads will be directed to leader etcd replica.
|
||||
Setting this value to true is optional: reads will be more reliable but will also be slower.
|
||||
|
||||
Optionally, you can specify a GCE zone where the first master replica is to be created.
|
||||
Optionally, you can specify a GCE zone where the first control plane node is to be created.
|
||||
Set the following flag:
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the first master replica will run.
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the first control plane node will run.
|
||||
|
||||
The following sample command sets up a HA-compatible cluster in the GCE zone europe-west1-b:
|
||||
|
||||
@@ -46,50 +47,52 @@ The following sample command sets up a HA-compatible cluster in the GCE zone eur
|
||||
MULTIZONE=true KUBE_GCE_ZONE=europe-west1-b ENABLE_ETCD_QUORUM_READS=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
Note that the commands above create a cluster with one master;
|
||||
however, you can add new master replicas to the cluster with subsequent commands.
|
||||
Note that the commands above create a cluster with one control plane node;
|
||||
however, you can add new control plane nodes to the cluster with subsequent commands.
|
||||
|
||||
## Adding a new master replica
|
||||
## Adding a new control plane node
|
||||
|
||||
After you have created an HA-compatible cluster, you can add master replicas to it.
|
||||
You add master replicas by using a `kube-up` script with the following flags:
|
||||
After you have created an HA-compatible cluster, you can add control plane nodes to it.
|
||||
You add control plane nodes by using a `kube-up` script with the following flags:
|
||||
|
||||
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing
|
||||
master.
|
||||
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing control plane
|
||||
node.
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the master replica will run.
|
||||
Must be in the same region as other replicas' zones.
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the control plane node will run.
|
||||
Must be in the same region as other control plane nodes' zones.
|
||||
|
||||
You don't need to set the `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags,
|
||||
as those are inherited from when you started your HA-compatible cluster.
|
||||
|
||||
The following sample command replicates the master on an existing HA-compatible cluster:
|
||||
The following sample command replicates the control plane node on an existing
|
||||
HA-compatible cluster:
|
||||
|
||||
```shell
|
||||
KUBE_GCE_ZONE=europe-west1-c KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
## Removing a master replica
|
||||
## Removing a control plane node
|
||||
|
||||
You can remove a master replica from an HA cluster by using a `kube-down` script with the following flags:
|
||||
You can remove a control plane node from an HA cluster by using a `kube-down` script with the following flags:
|
||||
|
||||
* `KUBE_DELETE_NODES=false` - to restrain deletion of kubelets.
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - the zone from where master replica will be removed.
|
||||
* `KUBE_GCE_ZONE=zone` - the zone from where the control plane node will be removed.
|
||||
|
||||
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of master replica to remove.
|
||||
If empty: any replica from the given zone will be removed.
|
||||
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of control plane node to
|
||||
remove. If empty: any replica from the given zone will be removed.
|
||||
|
||||
The following sample command removes a master replica from an existing HA cluster:
|
||||
The following sample command removes a control plane node from an existing HA cluster:
|
||||
|
||||
```shell
|
||||
KUBE_DELETE_NODES=false KUBE_GCE_ZONE=europe-west1-c ./cluster/kube-down.sh
|
||||
```
|
||||
|
||||
## Handling master replica failures
|
||||
## Handling control plane node failures
|
||||
|
||||
If one of the master replicas in your HA cluster fails,
|
||||
the best practice is to remove the replica from your cluster and add a new replica in the same zone.
|
||||
If one of the control plane nodes in your HA cluster fails,
|
||||
the best practice is to remove the node from your cluster and add a new control plane
|
||||
node in the same zone.
|
||||
The following sample commands demonstrate this process:
|
||||
|
||||
1. Remove the broken replica:
|
||||
@@ -98,26 +101,31 @@ The following sample commands demonstrate this process:
|
||||
KUBE_DELETE_NODES=false KUBE_GCE_ZONE=replica_zone KUBE_REPLICA_NAME=replica_name ./cluster/kube-down.sh
|
||||
```
|
||||
|
||||
<ol start="2"><li>Add a new replica in place of the old one:</li></ol>
|
||||
<ol start="2"><li>Add a new node in place of the old one:</li></ol>
|
||||
|
||||
```shell
|
||||
KUBE_GCE_ZONE=replica-zone KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
## Best practices for replicating masters for HA clusters
|
||||
## Best practices for replicating control plane nodes for HA clusters
|
||||
|
||||
* Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail.
|
||||
* Try to place control plane nodes in different zones. During a zone failure, all
|
||||
control plane nodes placed inside the zone will fail.
|
||||
To survive zone failure, also place nodes in multiple zones
|
||||
(see [multiple-zones](/docs/setup/best-practices/multiple-zones/) for details).
|
||||
|
||||
* Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing persistent state.
|
||||
As a result, both replicas are needed and a failure of any replica turns cluster into majority failure state.
|
||||
A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.
|
||||
* Do not use a cluster with two control plane nodes. Consensus on a two-node
|
||||
control plane requires both nodes running when changing persistent state.
|
||||
As a result, both nodes are needed and a failure of any node turns the cluster
|
||||
into majority failure state.
|
||||
A two-node control plane is thus inferior, in terms of HA, to a cluster with
|
||||
one control plane node.
|
||||
|
||||
* When you add a master replica, cluster state (etcd) is copied to a new instance.
|
||||
* When you add a control plane node, cluster state (etcd) is copied to a new instance.
|
||||
If the cluster is large, it may take a long time to duplicate its state.
|
||||
This operation may be sped up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration)
|
||||
(we are considering adding support for etcd data dir migration in future).
|
||||
This operation may be sped up by migrating the etcd data directory, as described in
|
||||
the [etcd administration guide](https://etcd.io/docs/v2.3/admin_guide/#member-migration)
|
||||
(we are considering adding support for etcd data dir migration in the future).
|
||||
|
||||
|
||||
|
||||
@@ -129,7 +137,7 @@ This operation may be sped up by migrating etcd data directory, as described [he
|
||||
|
||||
### Overview
|
||||
|
||||
Each of master replicas will run the following components in the following mode:
|
||||
Each of the control plane nodes will run the following components in the following mode:
|
||||
|
||||
* etcd instance: all instances will be clustered together using consensus;
|
||||
|
||||
@@ -143,9 +151,9 @@ In addition, there will be a load balancer in front of API servers that will rou
|
||||
|
||||
### Load balancing
|
||||
|
||||
When starting the second master replica, a load balancer containing the two replicas will be created
|
||||
When starting the second control plane node, a load balancer containing the two replicas will be created
|
||||
and the IP address of the first replica will be promoted to IP address of load balancer.
|
||||
Similarly, after removal of the penultimate master replica, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
|
||||
Similarly, after removal of the penultimate control plane node, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
|
||||
Please note that creation and removal of load balancer are complex operations and it may take some time (~20 minutes) for them to propagate.
|
||||
|
||||
### Master service & kubelets
|
||||
@@ -153,17 +161,17 @@ Please note that creation and removal of load balancer are complex operations an
|
||||
Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service,
|
||||
the system directs all traffic to the external IP:
|
||||
|
||||
* in one master cluster the IP points to the single master,
|
||||
* in case of a single node control plane, the IP points to the control plane node,
|
||||
|
||||
* in multi-master cluster the IP points to the load balancer in-front of the masters.
|
||||
* in case of an HA control plane, the IP points to the load balancer in-front of the masters.
|
||||
|
||||
Similarly, the external IP will be used by kubelets to communicate with master.
|
||||
Similarly, the external IP will be used by kubelets to communicate with the control plane.
|
||||
|
||||
### Master certificates
|
||||
### Control plane node certificates
|
||||
|
||||
Kubernetes generates Master TLS certificates for the external public IP and local IP for each replica.
|
||||
There are no certificates for the ephemeral public IP for replicas;
|
||||
to access a replica via its ephemeral public IP, you must skip TLS verification.
|
||||
Kubernetes generates TLS certificates for the external public IP and local IP for each control plane node.
|
||||
There are no certificates for the ephemeral public IP for control plane nodes;
|
||||
to access a control plane node via its ephemeral public IP, you must skip TLS verification.
|
||||
|
||||
### Clustering etcd
|
||||
|
||||
@@ -172,7 +180,7 @@ To make such deployment secure, communication between etcd instances is authoriz
|
||||
|
||||
### API server identity
|
||||
|
||||
{{< feature-state state="alpha" for_k8s_version="v1.20" >}}
|
||||
{{< feature-state state="alpha" for_k8s_version="v1.20" >}}
|
||||
|
||||
The API Server Identity feature is controlled by a
|
||||
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
||||
@@ -188,7 +188,7 @@ To install a specific version of containerD specify the version with -ContainerD
|
||||
|
||||
```powershell
|
||||
# Example
|
||||
.\Install-Containerd.ps1 -ContainerDVersion v1.4.1
|
||||
.\Install-Containerd.ps1 -ContainerDVersion 1.4.1
|
||||
```
|
||||
|
||||
{{< /note >}}
|
||||
|
||||
@@ -239,7 +239,7 @@ The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet servin
|
||||
certificates by requesting them from the `certificates.k8s.io` API. One known limitation
|
||||
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
|
||||
approved by the default signer in the kube-controller-manager -
|
||||
[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
|
||||
[`kubernetes.io/kubelet-serving`](/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
|
||||
This will require action from the user or a third party controller.
|
||||
|
||||
These CSRs can be viewed using:
|
||||
|
||||
@@ -167,8 +167,8 @@ The output is similar to this:
|
||||
Name: build-robot-secret
|
||||
Namespace: default
|
||||
Labels: <none>
|
||||
Annotations: kubernetes.io/service-account.name=build-robot
|
||||
kubernetes.io/service-account.uid=da68f9c6-9d26-11e7-b84e-002dc52800da
|
||||
Annotations: kubernetes.io/service-account.name: build-robot
|
||||
kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da
|
||||
|
||||
Type: kubernetes.io/service-account-token
|
||||
|
||||
|
||||
@@ -54,7 +54,7 @@ If you use a Docker credentials store, you won't see that `auth` entry but a `cr
|
||||
|
||||
## Create a Secret based on existing Docker credentials {#registry-secret-existing-credentials}
|
||||
|
||||
A Kubernetes cluster uses the Secret of `docker-registry` type to authenticate with
|
||||
A Kubernetes cluster uses the Secret of `kubernetes.io/dockerconfigjson` type to authenticate with
|
||||
a container registry to pull a private image.
|
||||
|
||||
If you already ran `docker login`, you can copy that credential into Kubernetes:
|
||||
|
||||
@@ -45,8 +45,12 @@ kubectl create namespace qos-example
|
||||
|
||||
For a Pod to be given a QoS class of Guaranteed:
|
||||
|
||||
* Every Container, including init containers, in the Pod must have a memory limit and a memory request, and they must be the same.
|
||||
* Every Container, including init containers, in the Pod must have a CPU limit and a CPU request, and they must be the same.
|
||||
* Every Container in the Pod must have a memory limit and a memory request.
|
||||
* For every Container in the Pod, the memory limit must equal the memory request.
|
||||
* Every Container in the Pod must have a CPU limit and a CPU request.
|
||||
* For every Container in the Pod, the CPU limit must equal the CPU request.
|
||||
|
||||
These restrictions apply to init containers and app containers equally.
|
||||
|
||||
Here is the configuration file for a Pod that has one Container. The Container has a memory limit and a
|
||||
memory request, both equal to 200 MiB. The Container has a CPU limit and a CPU request, both equal to 700 milliCPU:
|
||||
@@ -272,5 +276,3 @@ kubectl delete namespace qos-example
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -176,7 +176,6 @@ securityContext:
|
||||
fsGroupChangePolicy: "OnRootMismatch"
|
||||
```
|
||||
|
||||
This is an alpha feature. To use it, enable the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `ConfigurableFSGroupPolicy` for the kube-api-server, the kube-controller-manager, and for the kubelet.
|
||||
|
||||
{{< note >}}
|
||||
This field has no effect on ephemeral volume types such as
|
||||
|
||||
@@ -1,371 +0,0 @@
|
||||
---
|
||||
reviewers:
|
||||
- piosz
|
||||
- x13n
|
||||
title: Logging Using Stackdriver
|
||||
content_type: concept
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Before reading this page, it's highly recommended to familiarize yourself
|
||||
with the [overview of logging in Kubernetes](/docs/concepts/cluster-administration/logging).
|
||||
|
||||
{{< note >}}
|
||||
By default, Stackdriver logging collects only your container's standard output and
|
||||
standard error streams. To collect any logs your application writes to a file (for example),
|
||||
see the [sidecar approach](/docs/concepts/cluster-administration/logging#sidecar-container-with-a-logging-agent)
|
||||
in the Kubernetes logging overview.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Deploying
|
||||
|
||||
To ingest logs, you must deploy the Stackdriver Logging agent to each node in your cluster.
|
||||
The agent is a configured `fluentd` instance, where the configuration is stored in a `ConfigMap`
|
||||
and the instances are managed using a Kubernetes `DaemonSet`. The actual deployment of the
|
||||
`ConfigMap` and `DaemonSet` for your cluster depends on your individual cluster setup.
|
||||
|
||||
### Deploying to a new cluster
|
||||
|
||||
#### Google Kubernetes Engine
|
||||
|
||||
Stackdriver is the default logging solution for clusters deployed on Google Kubernetes Engine.
|
||||
Stackdriver Logging is deployed to a new cluster by default unless you explicitly opt-out.
|
||||
|
||||
#### Other platforms
|
||||
|
||||
To deploy Stackdriver Logging on a *new* cluster that you're
|
||||
creating using `kube-up.sh`, do the following:
|
||||
|
||||
1. Set the `KUBE_LOGGING_DESTINATION` environment variable to `gcp`.
|
||||
1. **If not running on GCE**, include the `beta.kubernetes.io/fluentd-ds-ready=true`
|
||||
in the `KUBE_NODE_LABELS` variable.
|
||||
|
||||
Once your cluster has started, each node should be running the Stackdriver Logging agent.
|
||||
The `DaemonSet` and `ConfigMap` are configured as addons. If you're not using `kube-up.sh`,
|
||||
consider starting a cluster without a pre-configured logging solution and then deploying
|
||||
Stackdriver Logging agents to the running cluster.
|
||||
|
||||
{{< warning >}}
|
||||
The Stackdriver logging daemon has known issues on platforms other
|
||||
than Google Kubernetes Engine. Proceed at your own risk.
|
||||
{{< /warning >}}
|
||||
|
||||
### Deploying to an existing cluster
|
||||
|
||||
1. Apply a label on each node, if not already present.
|
||||
|
||||
The Stackdriver Logging agent deployment uses node labels to determine to which nodes
|
||||
it should be allocated. These labels were introduced to distinguish nodes with the
|
||||
Kubernetes version 1.6 or higher. If the cluster was created with Stackdriver Logging
|
||||
configured and node has version 1.5.X or lower, it will have fluentd as static pod. Node
|
||||
cannot have more than one instance of fluentd, therefore only apply labels to the nodes
|
||||
that don't have fluentd pod allocated already. You can ensure that your node is labelled
|
||||
properly by running `kubectl describe` as follows:
|
||||
|
||||
```
|
||||
kubectl describe node $NODE_NAME
|
||||
```
|
||||
|
||||
The output should be similar to this:
|
||||
|
||||
```
|
||||
Name: NODE_NAME
|
||||
Role:
|
||||
Labels: beta.kubernetes.io/fluentd-ds-ready=true
|
||||
...
|
||||
```
|
||||
|
||||
Ensure that the output contains the label `beta.kubernetes.io/fluentd-ds-ready=true`. If it
|
||||
is not present, you can add it using the `kubectl label` command as follows:
|
||||
|
||||
```
|
||||
kubectl label node $NODE_NAME beta.kubernetes.io/fluentd-ds-ready=true
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
If a node fails and has to be recreated, you must re-apply the label to
|
||||
the recreated node. To make this easier, you can use Kubelet's command-line parameter
|
||||
for applying node labels in your node startup script.
|
||||
{{< /note >}}
|
||||
|
||||
1. Deploy a `ConfigMap` with the logging agent configuration by running the following command:
|
||||
|
||||
```
|
||||
kubectl apply -f https://k8s.io/examples/debug/fluentd-gcp-configmap.yaml
|
||||
```
|
||||
|
||||
The command creates the `ConfigMap` in the `default` namespace. You can download the file
|
||||
manually and change it before creating the `ConfigMap` object.
|
||||
|
||||
1. Deploy the logging agent `DaemonSet` by running the following command:
|
||||
|
||||
```
|
||||
kubectl apply -f https://k8s.io/examples/debug/fluentd-gcp-ds.yaml
|
||||
```
|
||||
|
||||
You can download and edit this file before using it as well.
|
||||
|
||||
## Verifying your Logging Agent Deployment
|
||||
|
||||
After Stackdriver `DaemonSet` is deployed, you can discover logging agent deployment status
|
||||
by running the following command:
|
||||
|
||||
```shell
|
||||
kubectl get ds --all-namespaces
|
||||
```
|
||||
|
||||
If you have 3 nodes in the cluster, the output should looks similar to this:
|
||||
|
||||
```
|
||||
NAMESPACE NAME DESIRED CURRENT READY NODE-SELECTOR AGE
|
||||
...
|
||||
default fluentd-gcp-v2.0 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 5m
|
||||
...
|
||||
```
|
||||
|
||||
To understand how logging with Stackdriver works, consider the following
|
||||
synthetic log generator pod specification [counter-pod.yaml](/examples/debug/counter-pod.yaml):
|
||||
|
||||
{{< codenew file="debug/counter-pod.yaml" >}}
|
||||
|
||||
This pod specification has one container that runs a bash script
|
||||
that writes out the value of a counter and the datetime once per
|
||||
second, and runs indefinitely. Let's create this pod in the default namespace.
|
||||
|
||||
```shell
|
||||
kubectl apply -f https://k8s.io/examples/debug/counter-pod.yaml
|
||||
```
|
||||
|
||||
You can observe the running pod:
|
||||
|
||||
```shell
|
||||
kubectl get pods
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
counter 1/1 Running 0 5m
|
||||
```
|
||||
|
||||
For a short period of time you can observe the 'Pending' pod status, because the kubelet
|
||||
has to download the container image first. When the pod status changes to `Running`
|
||||
you can use the `kubectl logs` command to view the output of this counter pod.
|
||||
|
||||
```shell
|
||||
kubectl logs counter
|
||||
```
|
||||
```
|
||||
0: Mon Jan 1 00:00:00 UTC 2001
|
||||
1: Mon Jan 1 00:00:01 UTC 2001
|
||||
2: Mon Jan 1 00:00:02 UTC 2001
|
||||
...
|
||||
```
|
||||
|
||||
As described in the logging overview, this command fetches log entries
|
||||
from the container log file. If the container is killed and then restarted by
|
||||
Kubernetes, you can still access logs from the previous container. However,
|
||||
if the pod is evicted from the node, log files are lost. Let's demonstrate this
|
||||
by deleting the currently running counter container:
|
||||
|
||||
```shell
|
||||
kubectl delete pod counter
|
||||
```
|
||||
```
|
||||
pod "counter" deleted
|
||||
```
|
||||
|
||||
and then recreating it:
|
||||
|
||||
```shell
|
||||
kubectl create -f https://k8s.io/examples/debug/counter-pod.yaml
|
||||
```
|
||||
```
|
||||
pod/counter created
|
||||
```
|
||||
|
||||
After some time, you can access logs from the counter pod again:
|
||||
|
||||
```shell
|
||||
kubectl logs counter
|
||||
```
|
||||
```
|
||||
0: Mon Jan 1 00:01:00 UTC 2001
|
||||
1: Mon Jan 1 00:01:01 UTC 2001
|
||||
2: Mon Jan 1 00:01:02 UTC 2001
|
||||
...
|
||||
```
|
||||
|
||||
As expected, only recent log lines are present. However, for a real-world
|
||||
application you will likely want to be able to access logs from all containers,
|
||||
especially for the debug purposes. This is exactly when the previously enabled
|
||||
Stackdriver Logging can help.
|
||||
|
||||
## Viewing logs
|
||||
|
||||
Stackdriver Logging agent attaches metadata to each log entry, for you to use later
|
||||
in queries to select only the messages you're interested in: for example,
|
||||
the messages from a particular pod.
|
||||
|
||||
The most important pieces of metadata are the resource type and log name.
|
||||
The resource type of a container log is `container`, which is named
|
||||
`GKE Containers` in the UI (even if the Kubernetes cluster is not on Google Kubernetes Engine).
|
||||
The log name is the name of the container, so that if you have a pod with
|
||||
two containers, named `container_1` and `container_2` in the spec, their logs
|
||||
will have log names `container_1` and `container_2` respectively.
|
||||
|
||||
System components have resource type `compute`, which is named
|
||||
`GCE VM Instance` in the interface. Log names for system components are fixed.
|
||||
For a Google Kubernetes Engine node, every log entry from a system component has one of the following
|
||||
log names:
|
||||
|
||||
* docker
|
||||
* kubelet
|
||||
* kube-proxy
|
||||
|
||||
You can learn more about viewing logs on [the dedicated Stackdriver page](https://cloud.google.com/logging/docs/view/logs_viewer).
|
||||
|
||||
One of the possible ways to view logs is using the
|
||||
[`gcloud logging`](https://cloud.google.com/logging/docs/api/gcloud-logging)
|
||||
command line interface from the [Google Cloud SDK](https://cloud.google.com/sdk/).
|
||||
It uses Stackdriver Logging [filtering syntax](https://cloud.google.com/logging/docs/view/advanced_filters)
|
||||
to query specific logs. For example, you can run the following command:
|
||||
|
||||
```none
|
||||
gcloud beta logging read 'logName="projects/$YOUR_PROJECT_ID/logs/count"' --format json | jq '.[].textPayload'
|
||||
```
|
||||
```
|
||||
...
|
||||
"2: Mon Jan 1 00:01:02 UTC 2001\n"
|
||||
"1: Mon Jan 1 00:01:01 UTC 2001\n"
|
||||
"0: Mon Jan 1 00:01:00 UTC 2001\n"
|
||||
...
|
||||
"2: Mon Jan 1 00:00:02 UTC 2001\n"
|
||||
"1: Mon Jan 1 00:00:01 UTC 2001\n"
|
||||
"0: Mon Jan 1 00:00:00 UTC 2001\n"
|
||||
```
|
||||
|
||||
As you can see, it outputs messages for the count container from both
|
||||
the first and second runs, despite the fact that the kubelet already deleted
|
||||
the logs for the first container.
|
||||
|
||||
### Exporting logs
|
||||
|
||||
You can export logs to [Google Cloud Storage](https://cloud.google.com/storage/)
|
||||
or to [BigQuery](https://cloud.google.com/bigquery/) to run further
|
||||
analysis. Stackdriver Logging offers the concept of sinks, where you can
|
||||
specify the destination of log entries. More information is available on
|
||||
the Stackdriver [Exporting Logs page](https://cloud.google.com/logging/docs/export/configure_export_v2).
|
||||
|
||||
## Configuring Stackdriver Logging Agents
|
||||
|
||||
Sometimes the default installation of Stackdriver Logging may not suit your needs, for example:
|
||||
|
||||
* You may want to add more resources because default performance doesn't suit your needs.
|
||||
* You may want to introduce additional parsing to extract more metadata from your log messages,
|
||||
like severity or source code reference.
|
||||
* You may want to send logs not only to Stackdriver or send it to Stackdriver only partially.
|
||||
|
||||
In this case you need to be able to change the parameters of `DaemonSet` and `ConfigMap`.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
If you're using GKE and Stackdriver Logging is enabled in your cluster, you
|
||||
cannot change its configuration, because it's managed and supported by GKE.
|
||||
However, you can disable the default integration and deploy your own.
|
||||
|
||||
{{< note >}}
|
||||
You will have to support and maintain a newly deployed configuration
|
||||
yourself: update the image and configuration, adjust the resources and so on.
|
||||
{{< /note >}}
|
||||
|
||||
To disable the default logging integration, use the following command:
|
||||
|
||||
```
|
||||
gcloud beta container clusters update --logging-service=none CLUSTER
|
||||
```
|
||||
|
||||
You can find notes on how to then install Stackdriver Logging agents into
|
||||
a running cluster in the [Deploying section](#deploying).
|
||||
|
||||
### Changing `DaemonSet` parameters
|
||||
|
||||
When you have the Stackdriver Logging `DaemonSet` in your cluster, you can modify the
|
||||
`template` field in its spec. The DaemonSet controller manages the pods for you.
|
||||
For example, assume you've installed the Stackdriver Logging as described above. Now you want to
|
||||
change the memory limit to give fluentd more memory to safely process more logs.
|
||||
|
||||
Get the spec of `DaemonSet` running in your cluster:
|
||||
|
||||
```shell
|
||||
kubectl get ds fluentd-gcp-v2.0 --namespace kube-system -o yaml > fluentd-gcp-ds.yaml
|
||||
```
|
||||
|
||||
Then edit resource requirements in the spec file and update the `DaemonSet` object
|
||||
in the apiserver using the following command:
|
||||
|
||||
```shell
|
||||
kubectl replace -f fluentd-gcp-ds.yaml
|
||||
```
|
||||
|
||||
After some time, Stackdriver Logging agent pods will be restarted with the new configuration.
|
||||
|
||||
### Changing fluentd parameters
|
||||
|
||||
Fluentd configuration is stored in the `ConfigMap` object. It is effectively a set of configuration
|
||||
files that are merged together. You can learn about fluentd configuration on the
|
||||
[official site](https://docs.fluentd.org).
|
||||
|
||||
Imagine you want to add a new parsing logic to the configuration, so that fluentd can understand
|
||||
default Python logging format. An appropriate fluentd filter looks similar to this:
|
||||
|
||||
```
|
||||
<filter reform.**>
|
||||
type parser
|
||||
format /^(?<severity>\w):(?<logger_name>\w):(?<log>.*)/
|
||||
reserve_data true
|
||||
suppress_parse_error_log true
|
||||
key_name log
|
||||
</filter>
|
||||
```
|
||||
|
||||
Now you have to put it in the configuration and make Stackdriver Logging agents pick it up.
|
||||
Get the current version of the Stackdriver Logging `ConfigMap` in your cluster
|
||||
by running the following command:
|
||||
|
||||
```shell
|
||||
kubectl get cm fluentd-gcp-config --namespace kube-system -o yaml > fluentd-gcp-configmap.yaml
|
||||
```
|
||||
|
||||
Then in the value of the key `containers.input.conf` insert a new filter right after
|
||||
the `source` section.
|
||||
|
||||
{{< note >}}
|
||||
Order is important.
|
||||
{{< /note >}}
|
||||
|
||||
Updating `ConfigMap` in the apiserver is more complicated than updating `DaemonSet`. It's better
|
||||
to consider `ConfigMap` to be immutable. Then, in order to update the configuration, you should
|
||||
create `ConfigMap` with a new name and then change `DaemonSet` to point to it
|
||||
using [guide above](#changing-daemonset-parameters).
|
||||
|
||||
### Adding fluentd plugins
|
||||
|
||||
Fluentd is written in Ruby and allows to extend its capabilities using
|
||||
[plugins](https://www.fluentd.org/plugins). If you want to use a plugin, which is not included
|
||||
in the default Stackdriver Logging container image, you have to build a custom image. Imagine
|
||||
you want to add Kafka sink for messages from a particular container for additional processing.
|
||||
You can re-use the default [container image sources](https://git.k8s.io/contrib/fluentd/fluentd-gcp-image)
|
||||
with minor changes:
|
||||
|
||||
* Change Makefile to point to your container repository, for example `PREFIX=gcr.io/<your-project-id>`.
|
||||
* Add your dependency to the Gemfile, for example `gem 'fluent-plugin-kafka'`.
|
||||
|
||||
Then run `make build push` from this directory. After updating `DaemonSet` to pick up the
|
||||
new image, you can use the plugin you installed in the fluentd configuration.
|
||||
|
||||
|
||||
@@ -31,6 +31,7 @@ The following methods exist for installing kubectl on macOS:
|
||||
{{< tabs name="download_binary_macos" >}}
|
||||
{{< tab name="Intel" codelang="bash" >}}
|
||||
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl"
|
||||
chmod +x kubectl
|
||||
{{< /tab >}}
|
||||
{{< tab name="Apple Silicon" codelang="bash" >}}
|
||||
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl"
|
||||
|
||||
Reference in New Issue
Block a user