whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix log rotation description in the logging doc (#3918)

Browse Source 
* Fix log rotation description in the logging doc

* Review comments

* Address review comments

* Address review comments
This commit is contained in:
Mik Vyatskov
2017-05-31 19:50:04 +02:00
committed by Andrew Chen
parent 5b26c42be5
commit f4fd9e79ed
1 changed files with 29 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts/cluster-administration/logging.md
+29 -8
View File
@@ -61,13 +61,32 @@ Everything a containerized application writes to `stdout` and `stderr` is handle
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs. By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs.
An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes uses the [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html) tool to implement log rotation. An important consideration in node-level logging is implementing log rotation,
so that logs don't consume all available storage on the node. Kubernetes
currently is not responsible for rotating logs, but rather a deployment tool
should set up a solution to address that.
For example, in Kubernetes clusters, deployed by the `kube-up.sh` script,
there is a [`logrotate`](http://www.linuxcommand.org/man_pages/logrotate8.html)
tool configured to run each hour. You can also set up a container runtime to
rotate application's logs automatically, e.g. by using Docker's `log-opt`.
In the `kube-up.sh` script, the latter approach is used for COS image on GCP,
and the former approach is used in any other environment. In both cases, by
default rotation is configured to take place when log file exceeds 10MB.
Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size. Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost. By default, Kubernetes keeps up to five logging rotations per container. As an example, you can find detailed information about how `kube-up.sh` sets
up logging for COS image on GCP in the corresponding [script]
[cosConfigureHelper].
The Kubernetes logging configuration differs depending on the node type. For example, you can find detailed information for GCI in the corresponding [configure helper](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh#L96). When you run [`kubectl logs`](/docs/user-guide/kubectl/v1.6/#logs) as in
the basic logging example, the kubelet on the node handles the request and
reads directly from the log file, returning the contents in the response.
**Note:** currently, if some external system has performed the rotation,
only the contents of the latest log file will be available through
`kubectl logs`. E.g. if there's a 10MB file, `logrotate` performs
the rotation and there are two files, one 10MB in size and one empty,
`kubectl logs` will return an empty response.
When you run [`kubectl logs`](/docs/user-guide/kubectl/v1.6/#logs), as in the basic logging example, the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response. Note that `kubectl logs` **only returns the last rotation**; you must manually extract prior rotations, if desired and cluster-level logging is not enabled. [cosConfigureHelper]: https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/cluster/gce/gci/configure-helper.sh
### System component logs ### System component logs
@@ -80,14 +99,16 @@ that do not run in a container. For example:
On machines with systemd, the kubelet and container runtime write to journald. If On machines with systemd, the kubelet and container runtime write to journald. If
systemd is not present, they write to `.log` files in the `/var/log` directory. systemd is not present, they write to `.log` files in the `/var/log` directory.
System components inside containers always write to the `/var/log` directory, System components inside containers always write to the `/var/log` directory,
bypassing the default logging mechanism. They use the [glog](https://godoc.org/github.com/golang/glog) bypassing the default logging mechanism. They use the [glog][glog]
logging library. You can find the conventions for logging severity for those logging library. You can find the conventions for logging severity for those
components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md). components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md).
Similarly to the container logs, system component logs in the `/var/log` Similarly to the container logs, system component logs in the `/var/log`
directory are rotated daily and based on the log size. However, directory should be rotated. In Kubernetes clusters brought up by
system component logs have a higher size retention: by default, the `kube-up.sh` script, those logs are configured to be rotated by
they can store up to 100MB. the `logrotate` tool daily or once the size exceeds 100MB.
[glog]: https://godoc.org/github.com/golang/glog
## Cluster-level logging architectures ## Cluster-level logging architectures