For Cloud Service Providers (CSPs), who have multi-tenant use cases, MIG ensures one client cannot impact the work or scheduling of other clients, in addition to providing enhanced isolation for customers.

With MIG, each instance’s processors have separate and isolated paths through the entire memory system - the on-chip crossbar ports, L2 cache banks, memory controllers, and DRAM address busses are all assigned uniquely to an individual instance. This ensures that an individual user’s workload can run with predictable throughput and latency, with the same L2 cache allocation and DRAM bandwidth, even if other tasks are thrashing their own caches or saturating their DRAM interfaces. MIG can partition available GPU compute resources (including streaming multiprocessors or SMs, and GPU engines such as copy engines or decoders), to provide a defined quality of service (QoS) with fault isolation for different clients such as VMs, containers or processes. MIG enables multiple GPU Instances to run in parallel on a single, physical NVIDIA Ampere architecture GPU.

With MIG, users will be able to see and schedule jobs on their new virtual GPU Instances as if they were physical GPUs. MIG works with Linux operating systems, supports containers using Docker Engine, with support for Kubernetes and virtual machines using hypervisors such as Red Hat Virtualization and VMware vSphere. MIG supports the following deployment configurations:

• Bare-metal, including containers

• GPU pass-through virtualization to Linux guests on top of supported hypervisors

• vGPU on top of supported hypervisors

MIG allows multiple vGPUs (and thereby VMs) to run in parallel on a single GPU, while preserving the isolation guarantees that vGPU provides. For more information on GPU partitioning using vGPU and MIG, refer to the technical brief.

• No graphics APIs are supported (for example, OpenGL, Vulkan and so on). The exception to this is RTX Pro 6000 Blackwell GPUs where certain MIG profiles support graphics.

• No GPU to GPU P2P (either PCIe or NVLink) is supported.

• CUDA applications treat a Compute Instance and its parent GPU Instance as a single CUDA device. See this section on device enumeration by CUDA.

• CUDA IPC across GPU instances is not supported. CUDA IPC across Compute instances is supported.

• CUDA debugging (e.g. using cuda-gdb) and memory/race checking (for example, using cuda-memcheck or compute-sanitizer) is supported.

• CUDA MPS is supported on top of MIG. The only limitation is that the maximum number of clients (48) is lowered proportionally to the Compute Instance size.

• GPUDirect RDMA is supported when used from GPU Instances.

MIG supports running CUDA applications by specifying the CUDA device on which the application should be run. With CUDA 11/R450 and CUDA 12/R525, only enumeration of a single MIG instance is supported. In other words, regardless of how many MIG devices are created (or made available to a container), a single CUDA process can only enumerate a single MIG device. CUDA applications treat a CI and its parent GI as a single CUDA device. CUDA is limited to use a single CI and will pick the first one available if several of them are visible. To summarize, there are two constraints:

1. CUDA can only enumerate a single compute instance

2. CUDA will not enumerate non-MIG GPU if any compute instance is enumerated on any other GPU

Note that these constraints may be relaxed in future NVIDIA driver releases for MIG. CUDA_VISIBLE_DEVICES has been extended to add support for MIG. Depending on the driver versions being used, two formats are supported:

1. With drivers >= R470 (470.42.01+), each MIG device is assigned a GPU UUID starting with MIG-<UUID>.

2. With drivers < R470 (for example, R450 and R460), each MIG device is enumerated by specifying the CI and the corresponding parent GI. The format follows this convention: MIG-<GPU-UUID>/<GPU instance ID>/<compute instance ID>.

ubuntu@h100-test:~/gpu-burn$  nvidia-smi mig -lgip
 +-------------------------------------------------------------------------------+
 | GPU instance profiles:                                                        |
 | GPU   Name               ID    Instances   Memory     P2P    SM    DEC   ENC  |
 |                                Free/Total   GiB              CE    JPEG  OFA  |
 |===============================================================================|
 |   0  MIG 1g.10gb         19     7/7        9.75       No     14     1     0   |
 |                                                               1     1     0   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 1g.10gb+me      20     1/1        9.75       No     14     1     0   |
 |                                                               1     1     1   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 1g.20gb         15     4/4        19.62      No     14     1     0   |
 |                                                               1     1     0   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 2g.20gb         14     3/3        19.62      No     30     2     0   |
 |                                                               2     2     0   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 3g.40gb          9     2/2        39.50      No     46     3     0   |
 |                                                               3     3     0   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 4g.40gb          5     1/1        39.50      No     62     4     0   |
 |                                                               4     4     0   |
 +-------------------------------------------------------------------------------+
 |   0  MIG 7g.80gb          0     1/1        79.25      No     114    7     0   |
 |                                                               8     7     1   |
 +-------------------------------------------------------------------------------+

ubuntu@h100-test:~/gpu-burn$ nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-b7993cf0-88bc-732c-4f8a-12f4234a7cf9)
  MIG 7g.80gb     Device  0: (UUID: MIG-761616d7-1d64-58ca-bd17-0c0134f5e1bb)

ubuntu@h100-test:~/gpu-burn$ sudo nvidia-smi mig -lgi
+---------------------------------------------------------+
| GPU instances:                                          |
| GPU   Name               Profile  Instance   Placement  |
|                            ID       ID       Start:Size |
|=========================================================|
|   0  MIG 3g.40gb            9        1          4:4     |
+---------------------------------------------------------+
|   0  MIG 3g.40gb            9        2          0:4     |
+---------------------------------------------------------+

ubuntu@h100-test:~$ nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-b7993cf0-88bc-732c-4f8a-12f4234a7cf9)
  MIG 3g.40gb     Device  0: (UUID: MIG-42753a49-5c48-5303-b7fa-1b474afcfe6a)
  MIG 3g.40gb     Device  1: (UUID: MIG-ae71f6a7-5253-5734-8cbc-5e6ebd157675)

ubuntu@h100-test:~$ nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-b7993cf0-88bc-732c-4f8a-12f4234a7cf9)
  MIG 2g.20gb     Device  0: (UUID: MIG-5c355e34-9007-538c-93e8-28d4ceb6d442)
  MIG 2g.20gb     Device  1: (UUID: MIG-457964b3-a935-5f29-bdb7-2699e8ebf53a)
  MIG 2g.20gb     Device  2: (UUID: MIG-003358eb-f6d9-5d95-aa83-313e49258456)

ubuntu@h100-test:~$ nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-b7993cf0-88bc-732c-4f8a-12f4234a7cf9)
  MIG 1g.10gb     Device  0: (UUID: MIG-4fb27904-f37d-53eb-a625-6bbc5a2d37fe)
  MIG 1g.10gb     Device  1: (UUID: MIG-9829f3a0-98c5-54a1-8060-8b4f8df7cee4)
  MIG 1g.10gb     Device  2: (UUID: MIG-743a0c03-4b20-5785-8787-35ac6a127f0c)
  MIG 1g.10gb     Device  3: (UUID: MIG-5041ede5-a7ef-5e4b-a22b-65af1b1f6d5e)
  MIG 1g.10gb     Device  4: (UUID: MIG-1d1275b7-1da5-5c1e-aab2-58c6020c595a)
  MIG 1g.10gb     Device  5: (UUID: MIG-75e14770-31e9-5475-9100-7507ebf533ca)
  MIG 1g.10gb     Device  6: (UUID: MIG-59ae9f20-3c64-5e48-b2f2-799246bafeb3)