From f4fd9e79edc435b977c9fe37219fe39c67f699b3 Mon Sep 17 00:00:00 2001 From: Mik Vyatskov Date: Wed, 31 May 2017 19:50:04 +0200 Subject: [PATCH] Fix log rotation description in the logging doc (#3918) * Fix log rotation description in the logging doc * Review comments * Address review comments * Address review comments --- .../cluster-administration/logging.md | 37 +++++++++++++++---- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/concepts/cluster-administration/logging.md b/docs/concepts/cluster-administration/logging.md index 5059a3d81f..283a2c3812 100644 --- a/docs/concepts/cluster-administration/logging.md +++ b/docs/concepts/cluster-administration/logging.md @@ -61,13 +61,32 @@ Everything a containerized application writes to `stdout` and `stderr` is handle By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs. -An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes uses the [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html) tool to implement log rotation. +An important consideration in node-level logging is implementing log rotation, +so that logs don't consume all available storage on the node. Kubernetes +currently is not responsible for rotating logs, but rather a deployment tool +should set up a solution to address that. +For example, in Kubernetes clusters, deployed by the `kube-up.sh` script, +there is a [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html) +tool configured to run each hour. You can also set up a container runtime to +rotate application's logs automatically, e.g. by using Docker's `log-opt`. +In the `kube-up.sh` script, the latter approach is used for COS image on GCP, +and the former approach is used in any other environment. In both cases, by +default rotation is configured to take place when log file exceeds 10MB. -Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size. Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost. By default, Kubernetes keeps up to five logging rotations per container. +As an example, you can find detailed information about how `kube-up.sh` sets +up logging for COS image on GCP in the corresponding [script] +[cosConfigureHelper]. -The Kubernetes logging configuration differs depending on the node type. For example, you can find detailed information for GCI in the corresponding [configure helper](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L96). +When you run [`kubectl logs`](/docs/user-guide/kubectl/v1.6/#logs) as in +the basic logging example, the kubelet on the node handles the request and +reads directly from the log file, returning the contents in the response. +**Note:** currently, if some external system has performed the rotation, +only the contents of the latest log file will be available through +`kubectl logs`. E.g. if there's a 10MB file, `logrotate` performs +the rotation and there are two files, one 10MB in size and one empty, +`kubectl logs` will return an empty response. -When you run [`kubectl logs`](/docs/user-guide/kubectl/v1.6/#logs), as in the basic logging example, the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response. Note that `kubectl logs` **only returns the last rotation**; you must manually extract prior rotations, if desired and cluster-level logging is not enabled. +[cosConfigureHelper]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh ### System component logs @@ -80,14 +99,16 @@ that do not run in a container. For example: On machines with systemd, the kubelet and container runtime write to journald. If systemd is not present, they write to `.log` files in the `/var/log` directory. System components inside containers always write to the `/var/log` directory, -bypassing the default logging mechanism. They use the [glog](https://godoc.org/github.com/golang/glog) +bypassing the default logging mechanism. They use the [glog][glog] logging library. You can find the conventions for logging severity for those components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). Similarly to the container logs, system component logs in the `/var/log` -directory are rotated daily and based on the log size. However, -system component logs have a higher size retention: by default, -they can store up to 100MB. +directory should be rotated. In Kubernetes clusters brought up by +the `kube-up.sh` script, those logs are configured to be rotated by +the `logrotate` tool daily or once the size exceeds 100MB. + +[glog]: https://godoc.org/github.com/golang/glog ## Cluster-level logging architectures