whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrote doc on GPU support in Kubernetes. (#6736)

Browse Source
This commit is contained in:
Rohit Agarwal
2017-12-28 11:42:27 -08:00
committed by Steve Perry
parent b5f411fc64
commit 9b99280a0d
1 changed files with 173 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tasks/manage-gpus/scheduling-gpus.md
+169 -112
View File
@@ -4,154 +4,211 @@ approvers:
title: Schedule GPUs title: Schedule GPUs
--- ---
{% capture overview %} Kubernetes includes **experimental** support for managing NVIDIA GPUs spread
across nodes. The support for NVIDIA GPUs was added in v1.6 and has gone through
multiple backwards incompatible iterations. This page describes how users can
consume GPUs across different Kubernetes versions and the current limitations.
Kubernetes includes **experimental** support for managing NVIDIA GPUs spread across nodes. ## v1.6 and v1.7
This page describes how users can consume GPUs and the current limitations. To enable GPU support in 1.6 and 1.7, a special **alpha** feature gate
`Accelerators` has to be set to true across the system:
`--feature-gates="Accelerators=true"`. It also requires using the Docker
Engine as the container runtime.
{% endcapture %} Further, the Kubernetes nodes have to be pre-installed with NVIDIA drivers.
Kubelet will not detect NVIDIA GPUs otherwise.
{% capture prerequisites %} When you start Kubernetes components after all the above conditions are true,
Kubernetes will expose `alpha.kubernetes.io/nvidia-gpu` as a schedulable
resource.
1. Kubernetes nodes have to be pre-installed with Nvidia drivers. Kubelet will not detect Nvidia GPUs otherwise. Try to re-install Nvidia drivers if kubelet fails to expose Nvidia GPUs as part of Node Capacity. After installing the driver, run `nvidia-docker-plugin` to confirm that all drivers have been loaded. You can consume these GPUs from your containers by requesting
2. A special **alpha** feature gate `Accelerators` has to be set to true across the system: `--feature-gates="Accelerators=true"`. `alpha.kubernetes.io/nvidia-gpu` just like you request `cpu` or `memory`.
3. Nodes must be using `docker engine` as the container runtime. However, there are some limitations in how you specify the resource requirements
when using GPUs:
- GPUs are only supposed to be specified in the `limits` section, which means:
* You can specify GPU `limits` without specifying `requests` because
Kubernetes will use the limit as the request value by default.
* You can specify GPU in both `limits` and `requests` but these two values
must be equal.
* You cannot specify GPU `requests` without specifying `limits`.
- Containers (and pods) do not share GPUs. There's no overcommitting of GPUs.
- Each container can request one or more GPUs. It is not possible to request a
fraction of a GPU.
The nodes will automatically discover and expose all Nvidia GPUs as a schedulable resource. When using `alpha.kubernetes.io/nvidia-gpu` as the resource, you also have to
mount host directories containing NVIDIA libraries (libcuda.so, libnvidia.so
etc.) to the container.
{% endcapture %} Here's an example:
{% capture steps %}
## API
Nvidia GPUs can be consumed via container level resource requirements using the resource name `alpha.kubernetes.io/nvidia-gpu`.
```yaml ```yaml
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
name: gpu-pod name: cuda-vector-add
spec: spec:
restartPolicy: OnFailure
containers: containers:
- - name: cuda-vector-add
name: gpu-container-1 # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
image: k8s.gcr.io/pause:2.0 image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources: resources:
limits: limits:
alpha.kubernetes.io/nvidia-gpu: 2 # requesting 2 GPUs alpha.kubernetes.io/nvidia-gpu: 1 # requesting 1 GPU
- volumeMounts:
name: gpu-container-2 - name: "nvidia-libraries"
image: k8s.gcr.io/pause:2.0 mountPath: "/usr/local/nvidia/lib64"
resources: volumes:
limits: - name: "nvidia-libraries"
alpha.kubernetes.io/nvidia-gpu: 3 # requesting 3 GPUs hostPath:
path: "/usr/lib/nvidia-375"
``` ```
The `Accelerators` feature gate and `alpha.kubernetes.io/nvidia-gpu` resource
works on 1.8 and 1.9 as well. It will be deprecated in 1.10 and removed in
1.11.
## v1.8 onwards
**From 1.8 onwards, the recommended way to consume GPUs is to use [device
plugins](/docs/concepts/cluster-administration/device-plugins).**
To enable GPU support through device plugins, a special **alpha** feature gate
`DevicePlugins` has to be set to true across the system:
`--feature-gates="DevicePlugins=true"`.
Then you have to install NVIDIA drivers on the nodes and run an NVIDIA GPU device
plugin ([see below](#deploying-nvidia-gpu-device-plugin)).
When the above conditions are true, Kubernetes will expose `nvidia.com/gpu` as
a schedulable resource.
You can consume these GPUs from your containers by requesting
`nvidia.com/gpu` just like you request `cpu` or `memory`.
However, there are some limitations in how you specify the resource requirements
when using GPUs:
- GPUs are only supposed to be specified in the `limits` section, which means: - GPUs are only supposed to be specified in the `limits` section, which means:
* You can specify GPU `limits` without specifying `requests` because Kubernetes * You can specify GPU `limits` without specifying `requests` because
will use the limit as the request value by default. Kubernetes will use the limit as the request value by default.
* You can specify GPU in both `limits` and `requests` but these two values must equal. * You can specify GPU in both `limits` and `requests` but these two values
must be equal.
* You cannot specify GPU `requests` without specifying `limits`. * You cannot specify GPU `requests` without specifying `limits`.
- Containers (and pods) do not share GPUs. - Containers (and pods) do not share GPUs. There's no overcommitting of GPUs.
- Each container can request one or more GPUs. - Each container can request one or more GPUs. It is not possible to request a
- It is not possible to request a portion of a GPU. fraction of a GPU.
- Nodes are expected to be homogenous, i.e. run the same GPU hardware.
If your nodes are running different versions of GPUs, then use Node Labels and Node Selectors to schedule pods to appropriate GPUs. Unlike with `alpha.kubernetes.io/nvidia-gpu`, when using `nvidia.com/gpu` as
Following is an illustration of this workflow: the resource, you don't have to mount any special directories in your pod
specs. The device plugin is expected to inject them automatically in the
container.
As part of your Node bootstrapping, identify the GPU hardware type on your nodes and expose it as a node label. Here's an example:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: cuda-vector-add
spec:
restartPolicy: OnFailure
containers:
- name: cuda-vector-add
# https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
```
### Deploying NVIDIA GPU device plugin
There are currently two device plugin implementations for NVIDIA GPUs:
#### Official NVIDIA GPU device plugin
The [official NVIDIA GPU device plugin](https://github.com/NVIDIA/k8s-device-plugin)
has the following requirements:
- Kubernetes nodes have to be pre-installed with NVIDIA drivers.
- Kubernetes nodes have to be pre-installed with [nvidia-docker 2.0](https://github.com/NVIDIA/nvidia-docker)
- nvidia-container-runtime must be configured as the [default runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime)
for docker instead of runc.
- NVIDIA drivers ~= 361.93
To deploy the NVIDIA device plugin once your cluster is running and the above
requirements are satisfied:
```
# For Kubernetes v1.8
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.8/nvidia-device-plugin.yml
# For Kubernetes v1.9
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml
```
Report issues with this device plugin to [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin).
#### NVIDIA GPU device plugin used by GKE/GCE
The [NVIDIA GPU device plugin used by GKE/GCE](https://github.com/GoogleCloudPlatform/container-engine-accelerators/tree/master/cmd/nvidia_gpu)
doesn't require using nvidia-docker and should work with any container runtime
that is compatible the Kubernetes Container Runtime Interface (CRI). It's tested
on [Container-Optimized OS](https://cloud.google.com/container-optimized-os/)
and has experimental code for Ubuntu from 1.9 onwards.
On your 1.9 cluster, you can use the following commands to install the NVIDIA drivers and device plugin:
```
# Install NVIDIA drivers on Container-Optimized OS:
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/daemonset.yaml
# Install NVIDIA drivers on Ubuntu (experimental):
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/ubuntu/daemonset.yaml
# Install the device plugin:
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.9/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml
```
Report issues with this device plugin and installation method to [GoogleCloudPlatform/container-engine-accelerators](https://github.com/GoogleCloudPlatform/container-engine-accelerators).
## Clusters containing different types of NVIDIA GPUs
If different nodes in your cluster have different types of NVIDIA GPUs, then you
can use [Node Labels and Node Selectors](/docs/tasks/configure-pod-container/assign-pods-nodes/)
to schedule pods to appropriate nodes.
For example:
```shell ```shell
NVIDIA_GPU_NAME=$(nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 | sed -e 's/ /-/g') # Label your nodes with the accelerator type they have.
source /etc/default/kubelet kubectl label nodes <node-with-k80> accelerator=nvidia-tesla-k80
KUBELET_OPTS="$KUBELET_OPTS --node-labels='alpha.kubernetes.io/nvidia-gpu-name=$NVIDIA_GPU_NAME'" kubectl label nodes <node-with-p100> accelerator=nvidia-tesla-p100
echo "KUBELET_OPTS=$KUBELET_OPTS" > /etc/default/kubelet
``` ```
Specify the GPU types a pod can use via [Node Affinity](/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) rules. Specify the GPU type in the pod spec:
```yaml ```yaml
kind: pod
apiVersion: v1 apiVersion: v1
kind: Pod
metadata: metadata:
annotations: name: cuda-vector-add
scheduler.alpha.kubernetes.io/affinity: >
{
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "alpha.kubernetes.io/nvidia-gpu-name",
"operator": "In",
"values": ["Tesla K80", "Tesla P100"]
}
]
}
]
}
}
}
spec: spec:
restartPolicy: OnFailure
containers: containers:
- - name: cuda-vector-add
name: gpu-container-1 # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources: resources:
limits: limits:
alpha.kubernetes.io/nvidia-gpu: 2 nvidia.com/gpu: 1
nodeSelector:
accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc.
``` ```
This will ensure that the pod will be scheduled to a node that has a `Tesla K80` or a `Tesla P100` Nvidia GPU. This will ensure that the pod will be scheduled to a node that has the GPU type
you specified.
### Warning
The API presented here **will change** in an upcoming release to better support GPUs, and hardware accelerators in general, in Kubernetes.
## Access to CUDA libraries
As of now, CUDA libraries are expected to be pre-installed on the nodes.
To mitigate this, you can copy the libraries to a more permissive folder in ``/var/lib/`` or change the permissions directly. (Future releases will automatically perform this operation)
Pods can access the libraries using `hostPath` volumes.
```yaml
kind: Pod
apiVersion: v1
metadata:
name: gpu-pod
spec:
containers:
- name: gpu-container-1
image: k8s.gcr.io/pause:2.0
resources:
limits:
alpha.kubernetes.io/nvidia-gpu: 1
volumeMounts:
- mountPath: /usr/local/nvidia/bin
name: bin
- mountPath: /usr/lib/nvidia
name: lib
volumes:
- hostPath:
path: /usr/lib/nvidia-375/bin
name: bin
- hostPath:
path: /usr/lib/nvidia-375
name: lib
```
## Future ## Future
- Support for hardware accelerators in Kubernetes is still in alpha.
- Support for hardware accelerators is in its early stages in Kubernetes.
- GPUs and other accelerators will soon be a native compute resource across the system.
- Better APIs will be introduced to provision and consume accelerators in a scalable manner. - Better APIs will be introduced to provision and consume accelerators in a scalable manner.
- Kubernetes will automatically ensure that applications consuming GPUs get the best possible performance. - Kubernetes will automatically ensure that applications consuming GPUs get the best possible performance.
- Key usability problems like access to CUDA libraries will be addressed.
{% endcapture %}
{% include templates/task.md %}