Which node types have a GPU + ConnectX NIC on the same PCIe root (PIX/PXB) for GPUDirect?

General/User Questions
1

Hi,

I’m doing line-rate GPU packet processing and need GPUDirect NIC-GPU (DOCA GPUNetIO / DPDK gpudev), which requires an NVIDIA GPU and a Mellanox ConnectX-6 Dx/ConnectX-7 or BlueField NIC on the same PCIe root complex (so nvidia-smi topo -m shows PIX/PXB between them).

a) Which bare-metal node types have a GPU and a ConnectX/BlueField NIC co-located on the same PCIe root?
b) Are DPDK + DOCA/OFED + hugepages + privileged binding of the NIC allowed on those bare-metal nodes?
c) Any node where the GigaIO composable PCIe fabric could place a GPU and NIC under one root?

I just need to reserve one such node briefly to verify topology. Thanks!

2

Hello,

Thank you for your questions!

I can answer some of that, but you’ll need to do a bit of digging to find the answers to the rest.

You can use our new resource discovery browser: discover.chameleoncloud.org to filter for things like GPU and ConnectX. E.g., if you check “has GPU” in the filter bar, then under advanced filters → network → adapter model you can find the ConnectX ones. That should narrow down to a set of nodes. From there you can reserve any of the nodes you believe to be a closest match (e.g. specific gigaio nodes) and once you are on the box you can examine the details more closely to see if it will meet your needs.

Please feel free to open a help desk ticket if you need any bios settings changed or have a specific question about one of the nodes you’ve reserved, etc.

Best,

Paul