You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split readme into smaller chunks, show only one "easy installation"
and hide the rest. Add some notes about tile resources.
Co-authored-by: Eero Tamminen <[email protected]>
Signed-off-by: Tuomas Katila <[email protected]>
| -shared-dev-num | int | 1 | Number of containers that can share the same GPU device |
56
46
| -allocation-policy | string | none | 3 possible values: balanced, packed, none. For shared-dev-num > 1: _balanced_ mode spreads workloads among GPU devices, _packed_ mode fills one GPU fully before moving to next, and _none_ selects first available device from kubelet. Default is _none_. Allocation policy does not have an effect when resource manager is enabled. |
57
47
@@ -60,145 +50,45 @@ Please use the -h option to see the complete list of logging related options.
60
50
61
51
## Operation modes for different workload types
62
52
53
+
<imgsrc="usage-scenarios.png"/>
54
+
63
55
Intel GPU-plugin supports a few different operation modes. Depending on the workloads the cluster is running, some modes make more sense than others. Below is a table that explains the differences between the modes and suggests workload types for each mode. Mode selection applies to the whole GPU plugin deployment, so it is a cluster wide decision.
64
56
65
57
| Mode | Sharing | Intended workloads | Suitable for time critical workloads |
66
58
|:---- |:-------- |:------- |:------- |
67
59
| shared-dev-num == 1 | No, 1 container per GPU | Workloads using all GPU capacity, e.g. AI training | Yes |
68
60
| shared-dev-num > 1 | Yes, >1 containers per GPU | (Batch) workloads using only part of GPU resources, e.g. inference, media transcode/analytics, or CPU bound GPU workloads | No |
69
-
| shared-dev-num > 1 && resource-management |Yes and no, 1>= containers per GPU | Any. For best results, all workloads should declare their expected GPU resource usage (memory, millicores). Requires [GAS](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling). See also [fractional use](#fractional-resources-details)| Yes. 1000 millicores = exclusive GPU usage. See note below. |
61
+
| shared-dev-num > 1 && resource-management |Depends on resource requests | Any. For requirements and usage, see [fractional resource management](./fractional.md)| Yes. 1000 millicores = exclusive GPU usage. See note below. |
70
62
71
63
> **Note**: Exclusive GPU usage with >=1000 millicores requires that also *all other GPU containers* specify (non-zero) millicores resource usage.
72
64
73
-
## Installation
74
-
75
-
The following sections detail how to obtain, build, deploy and test the GPU device plugin.
76
-
77
-
Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis.
78
-
79
-
### Prerequisites
80
-
81
-
Access to a GPU device requires firmware, kernel and user-space
82
-
drivers supporting it. Firmware and kernel driver need to be on the
83
-
host, user-space drivers in the GPU workload containers.
84
-
85
-
Intel GPU devices supported by the current kernel can be listed with:
86
-
```
87
-
$ grep i915 /sys/class/drm/card?/device/uevent
88
-
/sys/class/drm/card0/device/uevent:DRIVER=i915
89
-
/sys/class/drm/card1/device/uevent:DRIVER=i915
90
-
```
91
-
92
-
#### Drivers for discrete GPUs
93
-
94
-
> **Note**: Kernel (on host) and user-space drivers (in containers)
95
-
> should be installed from the same repository as there are some
96
-
> differences between DKMS and upstream GPU driver uAPI.
97
-
98
-
##### Kernel driver
99
-
100
-
###### Intel DKMS packages
101
-
102
-
`i915` GPU driver DKMS[^dkms] package is recommended for Intel
103
-
discrete GPUs, until their support in upstream is complete. DKMS
104
-
package(s) can be installed from Intel package repositories for a
105
-
subset of older kernel versions used in enterprise / LTS
of this component are available on the Docker hub. These images are automatically built and uploaded
164
73
to the hub from the latest main branch of this repository.
165
74
166
75
Release tagged images of the components are also available on the Docker hub, tagged with their
167
76
release version numbers in the format `x.y.z`, corresponding to the branches and releases in this
168
-
repository. Thus the easiest way to deploy the plugin in your cluster is to run this command
169
-
170
-
> **Note**: Replace `<RELEASE_VERSION>` with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images.
171
-
172
-
> **Note**: Add ```--dry-run=client -o yaml``` to the ```kubectl``` commands below to visualize the yaml content being applied.
77
+
repository.
173
78
174
79
See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin.
175
80
176
-
#### Install to all nodes
177
-
178
-
Simplest option to enable use of Intel GPUs in Kubernetes Pods.
Deploying GPU plugin to only nodes that have Intel GPU attached. [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) is required to detect the presence of Intel GPUs.
81
+
## Installation
187
82
188
-
```bash
189
-
# Start NFD - if your cluster doesn't have NFD installed yet
There are multiple ways to install Intel GPU plugin to a cluster. The most common methods are described below. For alternative methods, see [advanced install](./advanced-install.md) page.
191
84
192
-
# Create NodeFeatureRules for detecting GPUs on nodes
> **Note**: Replace `<RELEASE_VERSION>` with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images.
> **Note**: Add ```--dry-run=client -o yaml``` to the ```kubectl``` commands below to visualize the yaml content being applied.
198
88
199
-
####Install to nodes with NFD, Monitoring and Shared-dev
89
+
### Install with NFD
200
90
201
-
Same as above, but configures GPU plugin with logging, [monitoring and shared-dev](#modes-and-configuration-options) features enabled. This option is useful when there is a desire to retrieve GPU metrics from nodes. For example with [XPU-Manager](https://github.com/intel/xpumanager/) or [collectd](https://github.com/collectd/collectd/tree/collectd-6.0).
91
+
Deploy GPU plugin with the help of NFD ([Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery)). It detects the presence of Intel GPUs and labels them accordingly. GPU plugin's node selector is used to deploy plugin to nodes which have such a GPU label.
202
92
203
93
```bash
204
94
# Start NFD - if your cluster doesn't have NFD installed yet
#### Install to nodes with Intel GPUs with Fractional resources
215
-
216
-
With the experimental fractional resource feature you can use additional kubernetes extended
217
-
resources, such as GPU memory, which can then be consumed by deployments. PODs will then only
218
-
deploy to nodes where there are sufficient amounts of the extended resources for the containers.
104
+
### Install with Operator
219
105
220
-
(For this to work properly, all GPUs in a given node should provide equal amount of resources
221
-
i.e. heteregenous GPU nodes are not supported.)
106
+
GPU plugin can be installed with the Intel Device Plugin Operator. It allows configuring GPU plugin's parameters without kustomizing the deployment files. The general installation is described in the [install documentation](../operator/README.md#installation). For configuring the GPU Custom Resource (CR), see the [configuration options](#modes-and-configuration-options) and [operation modes](#operation-modes-for-different-workload-types).
222
107
223
-
Enabling the fractional resource feature isn't quite as simple as just enabling the related
224
-
command line flag. The DaemonSet needs additional RBAC-permissions
225
-
and access to the kubelet podresources gRPC service, plus there are other dependencies to
226
-
take care of, which are explained below. For the RBAC-permissions, gRPC service access and
227
-
the flag enabling, it is recommended to use kustomization by running:
108
+
### Install alongside with GPU Aware Scheduling
228
109
229
-
```bash
230
-
# Start NFD - if your cluster doesn't have NFD installed yet
GPU plugin can be installed alongside with GPU Aware Scheduling (GAS). It allows scheduling Pods which e.g. request only partial use of a GPU. The installation is described in [fractional resources](./fractional.md) page.
239
111
240
-
##### Fractional resources details
241
-
242
-
Usage of these fractional GPU resources requires that the cluster has node
243
-
extended resources with the name prefix `gpu.intel.com/`. Those can be created with NFD
244
-
by running the [hook](/cmd/gpu_nfdhook/) installed by the plugin initcontainer. When fractional resources are
245
-
enabled, the plugin lets a [scheduler extender](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling)
246
-
do card selection decisions based on resource availability and the amount of extended
247
-
resources requested in the [pod spec](https://github.com/intel/platform-aware-scheduling/blob/master/gpu-aware-scheduling/docs/usage.md#pods).
248
-
249
-
The scheduler extender then needs to annotate the pod objects with unique
250
-
increasing numeric timestamps in the annotation `gas-ts` and container card selections in
251
-
`gas-container-cards` annotation. The latter has container separator '`|`' and card separator
252
-
'`,`'. Example for a pod with two containers and both containers getting two cards:
253
-
`gas-container-cards:card0,card1|card2,card3`. Enabling the fractional-resource support
254
-
in the plugin without running such an annotation adding scheduler extender in the cluster
255
-
will only slow down GPU-deployments, so do not enable this feature unnecessarily.
256
-
257
-
In multi-tile systems, containers can request individual tiles to improve GPU resource usage.
258
-
Tiles targeted for containers are specified to pod via `gas-container-tiles` annotation where the the annotation
259
-
value describes a set of card and tile combinations. For example in a two container pod, the annotation
260
-
could be `gas-container-tiles:card0:gt0+gt1|card1:gt1,card2:gt0`. Similarly to `gas-container-cards`, the container
261
-
details are split via `|`. In the example above, the first container gets tiles 0 and 1 from card 0,
262
-
and the second container gets tile 1 from card 1 and tile 0 from card 2.
263
-
264
-
> **Note**: It is also possible to run the GPU device plugin using a non-root user. To do this,
265
-
the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration.
266
-
Furthermore, the deployments `securityContext` must be configured with appropriate `runAsUser/runAsGroup`.
267
-
268
-
### Verify Plugin Registration
112
+
### Verify Plugin Installation
269
113
270
-
You can verify the plugin has been registered with the expected nodes by searching for the relevant
114
+
You can verify that the plugin has been installed on the expected nodes by searching for the relevant
271
115
resource allocation status on the nodes:
272
116
273
117
```bash
@@ -341,17 +185,27 @@ The GPU plugin functionality can be verified by deploying an [OpenCL image](../.
It is possible to run the GPU device plugin using a non-root user. To do this,
193
+
the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration.
194
+
Furthermore, the deployments `securityContext` must be configured with appropriate `runAsUser/runAsGroup`.
195
+
196
+
More info: https://kubernetes.io/blog/2021/11/09/non-root-containers-and-devices/
197
+
198
+
### Labels created by GPU plugin
345
199
346
200
If installed with NFD and started with resource-management, plugin will export a set of labels for the node. For detailed info, see [labeling documentation](./labels.md).
347
201
348
-
## SR-IOV use with the plugin
202
+
### SR-IOV use with the plugin
349
203
350
204
GPU plugin does __not__ setup SR-IOV. It has to be configured by the cluster admin.
351
205
352
206
GPU plugin does however support provisioning Virtual Functions (VFs) to containers for a SR-IOV enabled GPU. When the plugin detects a GPU with SR-IOV VFs configured, it will only provision the VFs and leaves the PF device on the host.
353
207
354
-
## Issues with media workloads on multi-GPU setups
208
+
### Issues with media workloads on multi-GPU setups
355
209
356
210
OneVPL media API, 3D and compute APIs provide device discovery
357
211
functionality for applications and work fine in multi-GPU setups.
0 commit comments