From 8d6da018576a9a50acfcdb9f451021d5c1a000dc Mon Sep 17 00:00:00 2001 From: Swati Sehgal Date: Fri, 21 May 2021 00:39:30 +0100 Subject: [PATCH] podresource-api: Graduate GetAllocatableResources to Beta Also, explicitly clarify the behavior of GetAllocatableResources The explanation that GetAllocatableResources can be used to obtain available resources on the node can be misinterpretted as an API that is used to obtain free/unallocated resources on a node. This PR adds additional text to clarify that this API endpoint only returns allocatable resources which are resources exposed to kubelet as defined here: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable. Signed-off-by: Swati Sehgal --- .../compute-storage-net/device-plugins.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 868d8d56e8..6e766a86d6 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -197,6 +197,8 @@ service PodResourcesLister { } ``` +### `List` gRPC endpoint {#grpc-endpoint-list} + The `List` endpoint provides information on resources of running pods, with details such as the id of exclusively allocated CPUs, device id as it was reported by device plugins and id of the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container. @@ -247,9 +249,25 @@ message ContainerDevices { } ``` +### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources} + +{{< feature-state state="beta" for_k8s_version="v1.23" >}} + GetAllocatableResources provides information on resources initially available on the worker node. It provides more information than kubelet exports to APIServer. +{{< note >}} +`GetAllocatableResources` should only be used to evaluate [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +resources on a node. If the goal is to evaluate free/unallocated resources it should be used in +conjunction with the List() endpoint. The result obtained by `GetAllocatableResources` would remain +the same unless the underlying resources exposed to kubelet change. This happens rarely but when +it does (for example: hotplug/hotunplug, device health changes), client is expected to call +`GetAlloctableResources` endpoint. +However, calling `GetAllocatableResources` endpoint is not sufficient in case of cpu and/or memory +update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable. +{{< /note >}} + + ```gRPC // AllocatableResourcesResponses contains informations about all the devices known by the kubelet message AllocatableResourcesResponse { @@ -259,6 +277,13 @@ message AllocatableResourcesResponse { } ``` +Starting from Kubernetes v1.23, the `GetAllocatableResources` is enabled by default. +You can disable it by turning off the +`KubeletPodResourcesGetAllocatable` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). + +Preceding Kubernetes v1.23, to enable this feature `kubelet` must be started with the following flag: + +`--feature-gates=KubeletPodResourcesGetAllocatable=true` `ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to what device