From 9b99280a0d1034abfa1145f14770dd2303b88bfc Mon Sep 17 00:00:00 2001 From: Rohit Agarwal Date: Thu, 28 Dec 2017 11:42:27 -0800 Subject: [PATCH] Rewrote doc on GPU support in Kubernetes. (#6736) --- docs/tasks/manage-gpus/scheduling-gpus.md | 289 +++++++++++++--------- 1 file changed, 173 insertions(+), 116 deletions(-) diff --git a/docs/tasks/manage-gpus/scheduling-gpus.md b/docs/tasks/manage-gpus/scheduling-gpus.md index 4ad2126b40..5b33e93579 100644 --- a/docs/tasks/manage-gpus/scheduling-gpus.md +++ b/docs/tasks/manage-gpus/scheduling-gpus.md @@ -4,154 +4,211 @@ approvers: title: Schedule GPUs --- -{% capture overview %} +Kubernetes includes **experimental** support for managing NVIDIA GPUs spread +across nodes. The support for NVIDIA GPUs was added in v1.6 and has gone through +multiple backwards incompatible iterations. This page describes how users can +consume GPUs across different Kubernetes versions and the current limitations. -Kubernetes includes **experimental** support for managing NVIDIA GPUs spread across nodes. -This page describes how users can consume GPUs and the current limitations. +## v1.6 and v1.7 +To enable GPU support in 1.6 and 1.7, a special **alpha** feature gate +`Accelerators` has to be set to true across the system: +`--feature-gates="Accelerators=true"`. It also requires using the Docker +Engine as the container runtime. -{% endcapture %} +Further, the Kubernetes nodes have to be pre-installed with NVIDIA drivers. +Kubelet will not detect NVIDIA GPUs otherwise. -{% capture prerequisites %} - -1. Kubernetes nodes have to be pre-installed with Nvidia drivers. Kubelet will not detect Nvidia GPUs otherwise. Try to re-install Nvidia drivers if kubelet fails to expose Nvidia GPUs as part of Node Capacity. After installing the driver, run `nvidia-docker-plugin` to confirm that all drivers have been loaded. -2. A special **alpha** feature gate `Accelerators` has to be set to true across the system: `--feature-gates="Accelerators=true"`. -3. Nodes must be using `docker engine` as the container runtime. - -The nodes will automatically discover and expose all Nvidia GPUs as a schedulable resource. - -{% endcapture %} - -{% capture steps %} - -## API - -Nvidia GPUs can be consumed via container level resource requirements using the resource name `alpha.kubernetes.io/nvidia-gpu`. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: gpu-pod -spec: - containers: - - - name: gpu-container-1 - image: k8s.gcr.io/pause:2.0 - resources: - limits: - alpha.kubernetes.io/nvidia-gpu: 2 # requesting 2 GPUs - - - name: gpu-container-2 - image: k8s.gcr.io/pause:2.0 - resources: - limits: - alpha.kubernetes.io/nvidia-gpu: 3 # requesting 3 GPUs -``` +When you start Kubernetes components after all the above conditions are true, +Kubernetes will expose `alpha.kubernetes.io/nvidia-gpu` as a schedulable +resource. +You can consume these GPUs from your containers by requesting +`alpha.kubernetes.io/nvidia-gpu` just like you request `cpu` or `memory`. +However, there are some limitations in how you specify the resource requirements +when using GPUs: - GPUs are only supposed to be specified in the `limits` section, which means: - * You can specify GPU `limits` without specifying `requests` because Kubernetes - will use the limit as the request value by default. - * You can specify GPU in both `limits` and `requests` but these two values must equal. + * You can specify GPU `limits` without specifying `requests` because + Kubernetes will use the limit as the request value by default. + * You can specify GPU in both `limits` and `requests` but these two values + must be equal. * You cannot specify GPU `requests` without specifying `limits`. -- Containers (and pods) do not share GPUs. -- Each container can request one or more GPUs. -- It is not possible to request a portion of a GPU. -- Nodes are expected to be homogenous, i.e. run the same GPU hardware. +- Containers (and pods) do not share GPUs. There's no overcommitting of GPUs. +- Each container can request one or more GPUs. It is not possible to request a + fraction of a GPU. -If your nodes are running different versions of GPUs, then use Node Labels and Node Selectors to schedule pods to appropriate GPUs. -Following is an illustration of this workflow: +When using `alpha.kubernetes.io/nvidia-gpu` as the resource, you also have to +mount host directories containing NVIDIA libraries (libcuda.so, libnvidia.so +etc.) to the container. -As part of your Node bootstrapping, identify the GPU hardware type on your nodes and expose it as a node label. - -```shell -NVIDIA_GPU_NAME=$(nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 | sed -e 's/ /-/g') -source /etc/default/kubelet -KUBELET_OPTS="$KUBELET_OPTS --node-labels='alpha.kubernetes.io/nvidia-gpu-name=$NVIDIA_GPU_NAME'" -echo "KUBELET_OPTS=$KUBELET_OPTS" > /etc/default/kubelet -``` - -Specify the GPU types a pod can use via [Node Affinity](/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) rules. +Here's an example: ```yaml -kind: pod apiVersion: v1 +kind: Pod metadata: - annotations: - scheduler.alpha.kubernetes.io/affinity: > - { - "nodeAffinity": { - "requiredDuringSchedulingIgnoredDuringExecution": { - "nodeSelectorTerms": [ - { - "matchExpressions": [ - { - "key": "alpha.kubernetes.io/nvidia-gpu-name", - "operator": "In", - "values": ["Tesla K80", "Tesla P100"] - } - ] - } - ] - } - } - } + name: cuda-vector-add spec: + restartPolicy: OnFailure containers: - - - name: gpu-container-1 + - name: cuda-vector-add + # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile + image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: - alpha.kubernetes.io/nvidia-gpu: 2 + alpha.kubernetes.io/nvidia-gpu: 1 # requesting 1 GPU + volumeMounts: + - name: "nvidia-libraries" + mountPath: "/usr/local/nvidia/lib64" + volumes: + - name: "nvidia-libraries" + hostPath: + path: "/usr/lib/nvidia-375" ``` -This will ensure that the pod will be scheduled to a node that has a `Tesla K80` or a `Tesla P100` Nvidia GPU. +The `Accelerators` feature gate and `alpha.kubernetes.io/nvidia-gpu` resource +works on 1.8 and 1.9 as well. It will be deprecated in 1.10 and removed in +1.11. -### Warning +## v1.8 onwards -The API presented here **will change** in an upcoming release to better support GPUs, and hardware accelerators in general, in Kubernetes. +**From 1.8 onwards, the recommended way to consume GPUs is to use [device +plugins](/docs/concepts/cluster-administration/device-plugins).** -## Access to CUDA libraries +To enable GPU support through device plugins, a special **alpha** feature gate +`DevicePlugins` has to be set to true across the system: +`--feature-gates="DevicePlugins=true"`. -As of now, CUDA libraries are expected to be pre-installed on the nodes. +Then you have to install NVIDIA drivers on the nodes and run an NVIDIA GPU device +plugin ([see below](#deploying-nvidia-gpu-device-plugin)). -To mitigate this, you can copy the libraries to a more permissive folder in ``/var/lib/`` or change the permissions directly. (Future releases will automatically perform this operation) +When the above conditions are true, Kubernetes will expose `nvidia.com/gpu` as +a schedulable resource. -Pods can access the libraries using `hostPath` volumes. +You can consume these GPUs from your containers by requesting +`nvidia.com/gpu` just like you request `cpu` or `memory`. +However, there are some limitations in how you specify the resource requirements +when using GPUs: +- GPUs are only supposed to be specified in the `limits` section, which means: + * You can specify GPU `limits` without specifying `requests` because + Kubernetes will use the limit as the request value by default. + * You can specify GPU in both `limits` and `requests` but these two values + must be equal. + * You cannot specify GPU `requests` without specifying `limits`. +- Containers (and pods) do not share GPUs. There's no overcommitting of GPUs. +- Each container can request one or more GPUs. It is not possible to request a + fraction of a GPU. + +Unlike with `alpha.kubernetes.io/nvidia-gpu`, when using `nvidia.com/gpu` as +the resource, you don't have to mount any special directories in your pod +specs. The device plugin is expected to inject them automatically in the +container. + +Here's an example: ```yaml -kind: Pod apiVersion: v1 +kind: Pod metadata: - name: gpu-pod + name: cuda-vector-add spec: + restartPolicy: OnFailure containers: - - name: gpu-container-1 - image: k8s.gcr.io/pause:2.0 - resources: - limits: - alpha.kubernetes.io/nvidia-gpu: 1 - volumeMounts: - - mountPath: /usr/local/nvidia/bin - name: bin - - mountPath: /usr/lib/nvidia - name: lib - volumes: - - hostPath: - path: /usr/lib/nvidia-375/bin - name: bin - - hostPath: - path: /usr/lib/nvidia-375 - name: lib + - name: cuda-vector-add + # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile + image: "k8s.gcr.io/cuda-vector-add:v0.1" + resources: + limits: + nvidia.com/gpu: 1 # requesting 1 GPU ``` -## Future +### Deploying NVIDIA GPU device plugin -- Support for hardware accelerators is in its early stages in Kubernetes. -- GPUs and other accelerators will soon be a native compute resource across the system. +There are currently two device plugin implementations for NVIDIA GPUs: + +#### Official NVIDIA GPU device plugin + +The [official NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin) +has the following requirements: +- Kubernetes nodes have to be pre-installed with NVIDIA drivers. +- Kubernetes nodes have to be pre-installed with [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker) +- nvidia-container-runtime must be configured as the [default runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime) + for docker instead of runc. +- NVIDIA drivers ~= 361.93 + +To deploy the NVIDIA device plugin once your cluster is running and the above +requirements are satisfied: + +``` +# For Kubernetes v1.8 +kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.8/nvidia-device-plugin.yml + +# For Kubernetes v1.9 +kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml +``` + +Report issues with this device plugin to [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin). + +#### NVIDIA GPU device plugin used by GKE/GCE + +The [NVIDIA GPU device plugin used by GKE/GCE](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu) +doesn't require using nvidia-docker and should work with any container runtime +that is compatible the Kubernetes Container Runtime Interface (CRI). It's tested +on [Container-Optimized OS](https://cloud.google.com/container-optimized-os/) +and has experimental code for Ubuntu from 1.9 onwards. + +On your 1.9 cluster, you can use the following commands to install the NVIDIA drivers and device plugin: + +``` +# Install NVIDIA drivers on Container-Optimized OS: +kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/daemonset.yaml + +# Install NVIDIA drivers on Ubuntu (experimental): +kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/ubuntu/daemonset.yaml + +# Install the device plugin: +kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.9/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml +``` + +Report issues with this device plugin and installation method to [GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators). + +## Clusters containing different types of NVIDIA GPUs + +If different nodes in your cluster have different types of NVIDIA GPUs, then you +can use [Node Labels and Node Selectors](/docs/tasks/configure-pod-container/assign-pods-nodes/) +to schedule pods to appropriate nodes. + +For example: + +```shell +# Label your nodes with the accelerator type they have. +kubectl label nodes accelerator=nvidia-tesla-k80 +kubectl label nodes accelerator=nvidia-tesla-p100 +``` + +Specify the GPU type in the pod spec: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cuda-vector-add +spec: + restartPolicy: OnFailure + containers: + - name: cuda-vector-add + # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile + image: "k8s.gcr.io/cuda-vector-add:v0.1" + resources: + limits: + nvidia.com/gpu: 1 + nodeSelector: + accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc. +``` + +This will ensure that the pod will be scheduled to a node that has the GPU type +you specified. + +## Future +- Support for hardware accelerators in Kubernetes is still in alpha. - Better APIs will be introduced to provision and consume accelerators in a scalable manner. - Kubernetes will automatically ensure that applications consuming GPUs get the best possible performance. -- Key usability problems like access to CUDA libraries will be addressed. - -{% endcapture %} - -{% include templates/task.md %}