From fd97fcd9d58976c8ded4b8bdf65e5adb4507ef07 Mon Sep 17 00:00:00 2001 From: Qiutong Song Date: Fri, 25 Sep 2020 13:49:35 -0700 Subject: [PATCH 1/2] Add a new known issue to the out-of-resource doc: active_file memory is not considered as available memory --- content/en/docs/tasks/administer-cluster/out-of-resource.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/en/docs/tasks/administer-cluster/out-of-resource.md b/content/en/docs/tasks/administer-cluster/out-of-resource.md index b989ceac62..d0d03e7d28 100644 --- a/content/en/docs/tasks/administer-cluster/out-of-resource.md +++ b/content/en/docs/tasks/administer-cluster/out-of-resource.md @@ -341,4 +341,10 @@ to prevent system OOMs, and promote eviction of workloads so cluster state can r The Pod eviction may evict more Pods than needed due to stats collection timing gap. This can be mitigated by adding the ability to get root container stats on an on-demand basis [(https://github.com/google/cadvisor/issues/1247)](https://github.com/google/cadvisor/issues/1247) in the future. +### active_file memory is not considered as available memory + +Currently, kubelet considers active_file (# of bytes of file-backed memory on active LRU list) as not reclaimable. For I/O intense workload, this might trigger pod eviction due to memory pressure. There is an open discussion: [https://github.com/kubernetes/kubernetes/issues/43916](https://github.com/kubernetes/kubernetes/issues/43916) + +The recommended workaround is to set a memory limit equal to memory request, which requires to estimate the optimal memory limit value for the workload. + From 3a0302d12895ad066442e9b39ad6771d2fb01f37 Mon Sep 17 00:00:00 2001 From: Qiutong Song Date: Thu, 1 Oct 2020 14:02:43 -0700 Subject: [PATCH 2/2] Address comments and take the suggested wording --- content/en/docs/tasks/administer-cluster/out-of-resource.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/out-of-resource.md b/content/en/docs/tasks/administer-cluster/out-of-resource.md index d0d03e7d28..9444e197e2 100644 --- a/content/en/docs/tasks/administer-cluster/out-of-resource.md +++ b/content/en/docs/tasks/administer-cluster/out-of-resource.md @@ -343,8 +343,9 @@ the ability to get root container stats on an on-demand basis [(https://github.c ### active_file memory is not considered as available memory -Currently, kubelet considers active_file (# of bytes of file-backed memory on active LRU list) as not reclaimable. For I/O intense workload, this might trigger pod eviction due to memory pressure. There is an open discussion: [https://github.com/kubernetes/kubernetes/issues/43916](https://github.com/kubernetes/kubernetes/issues/43916) +On Linux, the kernel tracks the number of bytes of file-backed memory on active LRU list as the `active_file` statistic. The kubelet treats `active_file` memory areas as not reclaimable. For workloads that make intensive use of block-backed local storage, including ephemeral local storage, kernel-level caches of file and block data means that many recently accessed cache pages are likely to be counted as `active_file`. If enough of these kernel block buffers are on the active LRU list, the kubelet is liable to observe this as high resource use and taint the node as experiencing memory pressure - triggering Pod eviction. -The recommended workaround is to set a memory limit equal to memory request, which requires to estimate the optimal memory limit value for the workload. +For more more details, see [https://github.com/kubernetes/kubernetes/issues/43916](https://github.com/kubernetes/kubernetes/issues/43916) +You can work around that behavior by setting the memory limit and memory request the same for containers likely to perform intensive I/O activity. You will need to estimate or measure an optimal memory limit value for that container.