pod-overhead: add documentation (#15970)

Create a concepts page, link to it from scheduler concept page,
update feature gates reference and describe briefly in the
runtime-class page.

Signed-off-by: Eric Ernst <eric.ernst@intel.com>

content/en/docs/concepts/configuration/pod-overhead.md

@@ -0,0 +1,62 @@
+---
+reviewers:
+- dchen1107
+- egernst
+- tallclair
+title: Pod Overhead
+content_template: templates/concept
+weight: 20
+---
+
+{{% capture overview %}}
+
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
+
+
+When you run a Pod on a Node, the Pod itself takes an amount of system resources. These
+resources are additional to the resources needed to run the container(s) inside the Pod.
+_Pod Overhead_ is a feature for accounting for the resources consumed by the pod infrastructure
+on top of the container requests & limits.
+
+
+{{% /capture %}}
+
+
+{{% capture body %}}
+
+## Pod Overhead
+
+In Kubernetes, the pod's overhead is set at
+[admission](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#what-are-admission-webhooks)
+time according to the overhead associated with the pod's
+[RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/).
+
+When Pod Overhead is enabled, the overhead is considered in addition to the sum of container
+resource requests when scheduling a pod. Similarly, Kubelet will include the pod overhead when sizing
+the pod cgroup, and when carrying out pod eviction ranking.
+
+### Set Up
+
+You need to make sure that the `PodOverhead`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled (it is off by default)
+across your cluster. This means:
+
+- in {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}
+- in {{< glossary_tooltip text="kube-apiserver" term_id="kube-apiserver" >}}
+- in the {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} on each Node
+- in any custom API servers that use feature gates
+
+{{< note >}}
+Users who can write to RuntimeClass resources are able to have cluster-wide impact on
+workload performance. You can limit access to this ability using Kubernetes access controls.
+See [Authorization Overview](/docs/reference/access-authn-authz/authorization/) for more details.
+{{< /note >}}
+
+{{% /capture %}}
+
+{{% capture whatsnext %}}
+
+* [RuntimeClass](/docs/concepts/containers/runtime-class/)
+* [PodOverhead Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20190226-pod-overhead.md)
+
+{{% /capture %}}

content/en/docs/concepts/containers/runtime-class.md

@@ -171,6 +171,20 @@ Nodes](/docs/concepts/configuration/assign-pod-node/).
 
 [admission controller]: /docs/reference/access-authn-authz/admission-controllers/
 
+### Pod Overhead
+
+{{< feature-state for_k8s_version="v1.16" state="alpha" >}}
+
+As of Kubernetes v1.16, RuntimeClass includes support for specifying overhead associated with
+running a pod, as part of the [`PodOverhead`](/docs/concepts/configuration/pod-overhead.md) feature.
+To use `PodOverhead`, you must have the PodOverhead [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+enabled (it is off by default).
+
+
+Pod overhead is defined in RuntimeClass through the `Overhead` fields. Through the use of these fields,
+you can specify the overhead of running pods utilizing this RuntimeClass and ensure these overheads
+are accounted for in Kubernetes.
+
 ### Upgrading RuntimeClass from Alpha to Beta
 
 The RuntimeClass Beta feature includes the following changes:
@@ -202,5 +216,7 @@ RuntimeClass feature to the beta version:
 
 - [RuntimeClass Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/runtime-class.md)
 - [RuntimeClass Scheduling Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/runtime-class-scheduling.md)
+- Read about the [Pod Overhead](/docs/concepts/configuration/pod-overhead/) concept
+- [PodOverhead Feature Design](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20190226-pod-overhead.md)
 
 {{% /capture %}}

content/en/docs/concepts/scheduling/kube-scheduler.md

@@ -184,4 +184,5 @@ kube-scheduler has a default set of scheduling policies.
 * Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/)
 * Read the [reference documentation](/docs/reference/command-line-tools-reference/kube-scheduler/) for kube-scheduler
 * Learn about [configuring multiple schedulers](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/)
+* Learn about [Pod Overhead](/docs/concepts/configuration/pod-overhead/)
 {{% /capture %}}

content/en/docs/reference/command-line-tools-reference/feature-gates.md

@@ -128,6 +128,7 @@ different Kubernetes components.
 | `PersistentLocalVolumes` | `false` | Alpha | 1.7 | 1.9 |
 | `PersistentLocalVolumes` | `true` | Beta | 1.10 | 1.13 |
 | `PersistentLocalVolumes` | `true` | GA | 1.14 | |
+| `PodOverhead` | `false` | Alpha | 1.16 | - |
 | `PodPriority` | `false` | Alpha | 1.8 | 1.10 |
 | `PodPriority` | `true` | Beta | 1.11 | 1.13 |
 | `PodPriority` | `true` | GA | 1.14 | |
@@ -315,6 +316,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `NonPreemptingPriority`: Enable NonPreempting option for PriorityClass and Pod.
 - `PersistentLocalVolumes`: Enable the usage of `local` volume type in Pods.
   Pod affinity has to be specified if requesting a `local` volume.
+- `PodOverhead`: Enable the [PodOverhead](/docs/concepts/configuration/pod-overhead/) feature to account for pod overheads.
 - `PodPriority`: Enable the descheduling and preemption of Pods based on their [priorities](/docs/concepts/configuration/pod-priority-preemption/).
 - `PodReadinessGates`: Enable the setting of `PodReadinessGate` field for extending
   Pod readiness evaluation.