Update hugepages documentation (#19008)

* Update hugepages documentation

- described support for multiple huge page sizes
- described container isolation of the huge pages

* Add HugePageStorageMediumSize description

* update description for container isolation of hugepages

Signed-off-by: Byonggon Chun <bg.chun@samsung.com>

Co-authored-by: Byonggon Chun <bg.chun@samsung.com>

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -103,6 +103,7 @@ different Kubernetes components.
 | `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | |
 | `EvenPodsSpread` | `false` | Alpha | 1.16 | |
 | `HPAScaleToZero` | `false` | Alpha | 1.16 | |
+| `HugePageStorageMediumSize` | `false` | Alpha | 1.18 | |
 | `HyperVContainer` | `false` | Alpha | 1.10 | |
 | `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 |
 | `KubeletPodResources` | `true` | Beta | 1.15 | |
@@ -395,6 +396,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
    See [Enabling Endpoint Slices](/docs/tasks/administer-cluster/enabling-endpointslices/).
 - `GCERegionalPersistentDisk`: Enable the regional PD feature on GCE.
 - `HugePages`: Enable the allocation and consumption of pre-allocated [huge pages](/docs/tasks/manage-hugepages/scheduling-hugepages/).
+- `HugePageStorageMediumSize`: Enable support for multiple sizes pre-allocated [huge pages](/docs/tasks/manage-hugepages/scheduling-hugepages/).
 - `HyperVContainer`: Enable [Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container) for Windows containers.
 - `HPAScaleToZero`: Enables setting `minReplicas` to 0 for `HorizontalPodAutoscaler` resources when using custom or external metrics.
 - `KubeletConfigFile`: Enable loading kubelet configuration from a file specified using a config file.

content/en/docs/tasks/manage-hugepages/scheduling-hugepages.md

@@ -17,11 +17,11 @@ can consume huge pages and the current limitations.
 {{% capture prerequisites %}}
 
 1. Kubernetes nodes must pre-allocate huge pages in order for the node to report
-   its huge page capacity. A node may only pre-allocate huge pages for a single
+   its huge page capacity. A node can pre-allocate huge pages for multiple
-   size.
+   sizes.
 
-The nodes will automatically discover and report all huge page resources as a
+The nodes will automatically discover and report all huge page resources as
-schedulable resource.
+schedulable resources.
 
 {{% /capture %}}
 
@@ -30,12 +30,51 @@ schedulable resource.
 ## API
 
 Huge pages can be consumed via container level resource requirements using the
-resource name `hugepages-<size>`, where size is the most compact binary notation
+resource name `hugepages-<size>`, where `<size>` is the most compact binary
-using integer values supported on a particular node. For example, if a node
+notation using integer values supported on a particular node. For example, if a
-supports 2048KiB page sizes, it will expose a schedulable resource
+node supports 2048KiB and 1048576KiB page sizes, it will expose a schedulable
-`hugepages-2Mi`. Unlike CPU or memory, huge pages do not support overcommit. Note
+resources `hugepages-2Mi` and `hugepages-1Gi`. Unlike CPU or memory, huge pages
-that when requesting hugepage resources, either memory or CPU resources must
+do not support overcommit. Note that when requesting hugepage resources, either
-be requested as well.
+memory or CPU resources must be requested as well.
+
+A pod may consume multiple huge page sizes in a single pod spec. In this case it
+must use `medium: HugePages-<hugepagesize>` notation for all volume mounts.
+
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: huge-pages-example
+spec:
+  containers:
+  - name: example
+    image: fedora:latest
+    command:
+    - sleep
+    - inf
+    volumeMounts:
+    - mountPath: /hugepages-2Mi
+      name: hugepage-2mi
+    - mountPath: /hugepages-1Gi
+      name: hugepage-1gi
+    resources:
+      limits:
+        hugepages-2Mi: 100Mi
+        hugepages-1Gi: 2Gi
+        memory: 100Mi
+      requests:
+        memory: 100Mi
+  volumes:
+  - name: hugepage-2mi
+    emptyDir:
+      medium: HugePages-2Mi
+  - name: hugepage-1gi
+    emptyDir:
+      medium: HugePages-1Gi
+```
+
+A pod may use `medium: HugePages` only if it requests huge pages of one size.
 
 ```yaml
 apiVersion: v1
@@ -66,8 +105,7 @@ spec:
 
 - Huge page requests must equal the limits. This is the default if limits are
   specified, but requests are not.
-- Huge pages are isolated at a pod scope, container isolation is planned in a
+- Huge pages are isolated at a container scope, so each container has own limit on their cgroup sandbox as requested in a container spec.
-  future iteration.
 - EmptyDir volumes backed by huge pages may not consume more huge page memory
   than the pod request.
 - Applications that consume huge pages via `shmget()` with `SHM_HUGETLB` must
@@ -75,10 +113,15 @@ spec:
 - Huge page usage in a namespace is controllable via ResourceQuota similar
 to other compute resources like `cpu` or `memory` using the `hugepages-<size>`
 token.
+- Support of multiple sizes huge pages is feature gated. It can be
+  enabled with the `HugePageStorageMediumSize` [feature
+gate](/docs/reference/command-line-tools-reference/feature-gates/) on the {{<
+glossary_tooltip text="kubelet" term_id="kubelet" >}} and {{<
+glossary_tooltip text="kube-apiserver"
+term_id="kube-apiserver" >}} (`--feature-gates=HugePageStorageMediumSize=true`).
 
 ## Future
 
-- Support container isolation of huge pages in addition to pod isolation.
 - NUMA locality guarantees as a feature of quality of service.
 - LimitRange support.