From 8d6da018576a9a50acfcdb9f451021d5c1a000dc Mon Sep 17 00:00:00 2001 From: Swati Sehgal Date: Fri, 21 May 2021 00:39:30 +0100 Subject: [PATCH 1/2] podresource-api: Graduate GetAllocatableResources to Beta Also, explicitly clarify the behavior of GetAllocatableResources The explanation that GetAllocatableResources can be used to obtain available resources on the node can be misinterpretted as an API that is used to obtain free/unallocated resources on a node. This PR adds additional text to clarify that this API endpoint only returns allocatable resources which are resources exposed to kubelet as defined here: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable. Signed-off-by: Swati Sehgal --- .../compute-storage-net/device-plugins.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 868d8d56e8..6e766a86d6 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -197,6 +197,8 @@ service PodResourcesLister { } ``` +### `List` gRPC endpoint {#grpc-endpoint-list} + The `List` endpoint provides information on resources of running pods, with details such as the id of exclusively allocated CPUs, device id as it was reported by device plugins and id of the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container. @@ -247,9 +249,25 @@ message ContainerDevices { } ``` +### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources} + +{{< feature-state state="beta" for_k8s_version="v1.23" >}} + GetAllocatableResources provides information on resources initially available on the worker node. It provides more information than kubelet exports to APIServer. +{{< note >}} +`GetAllocatableResources` should only be used to evaluate [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +resources on a node. If the goal is to evaluate free/unallocated resources it should be used in +conjunction with the List() endpoint. The result obtained by `GetAllocatableResources` would remain +the same unless the underlying resources exposed to kubelet change. This happens rarely but when +it does (for example: hotplug/hotunplug, device health changes), client is expected to call +`GetAlloctableResources` endpoint. +However, calling `GetAllocatableResources` endpoint is not sufficient in case of cpu and/or memory +update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable. +{{< /note >}} + + ```gRPC // AllocatableResourcesResponses contains informations about all the devices known by the kubelet message AllocatableResourcesResponse { @@ -259,6 +277,13 @@ message AllocatableResourcesResponse { } ``` +Starting from Kubernetes v1.23, the `GetAllocatableResources` is enabled by default. +You can disable it by turning off the +`KubeletPodResourcesGetAllocatable` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). + +Preceding Kubernetes v1.23, to enable this feature `kubelet` must be started with the following flag: + +`--feature-gates=KubeletPodResourcesGetAllocatable=true` `ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to what device From c7231c8d6d0045bba24ea475f44e5daa3525df3c Mon Sep 17 00:00:00 2001 From: Swati Sehgal Date: Tue, 5 Oct 2021 21:08:43 +0100 Subject: [PATCH 2/2] Explicitly state that GetCpuIds returns exclusive cpus Based on the discussion here: https://github.com/kubernetes/kubernetes/pull/97415#discussion_r722548437 we explictly state that the GetCpuIds returned for a ContainerResource in the ListPodResourcesResponse represent only exclusively allocated CPUs. In order to evaluate the CPUs corresponding to the shared pool, List endpoint should be used in conjunction with GetAllocatableResources endpoint. We highlight the steps that the client needs to take evaluate this. Signed-off-by: Swati Sehgal --- .../compute-storage-net/device-plugins.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 6e766a86d6..f14f78b13b 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -248,6 +248,15 @@ message ContainerDevices { TopologyInfo topology = 3; } ``` +{{< note >}} +cpu_ids in the `ContainerResources` in the `List` endpoint correspond to exclusive CPUs allocated +to a partilar container. If the goal is to evaluate CPUs that belong to the shared pool, the `List` +endpoint needs to be used in conjunction with the `GetAllocatableResources` endpoint as explained +below: +1. Call `GetAllocatableResources` to get a list of all the allocatable CPUs +2. Call `GetCpuIds` on all `ContainerResources` in the system +3. Subtract out all of the CPUs from the `GetCpuIds` calls from the `GetAllocatableResources` call +{{< /note >}} ### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources}