MOFED support on CX3-Pro cards

[jithinjosepkl]

CX3-Pro cards are not supported in newer Mellanox OFED versions, and these cards are supported through Mellanox OFED LTS version (4.9-0.1.7.0). For more information, see Linux Drivers.

Thus, the latest Azure Marketplace HPC images that have Mellanox OFED 5.1 and later, do not support ConnectX3-Pro cards. Please check the Mellanox OFED version in the HPC image before using it on VMs with ConnectX3-Pro cards.

Following are the latest CentOS-HPC images that support ConnectX3-Pro cards:

• OpenLogic:CentOS-HPC:7.6:7.6.2020062900
• OpenLogic:CentOS-HPC:7_6gen2:7.6.2020062901
• OpenLogic:CentOS-HPC:7.7:7.7.2020062600
• OpenLogic:CentOS-HPC:7_7-gen2:7.7.2020062601
• OpenLogic:CentOS-HPC:8_1:8.1.2020062400
• OpenLogic:CentOS-HPC:8_1-gen2:8.1.2020062401

[matt-chan]

I got bitten by this recently. Would it be possible to publish a list of Azure families that use ConnectX3-pro? Mine was NDv2. It uses ConnectX3 for the ethernet controller.

[Nayanshivhare]

You could use any image with a supported Infiniband instance, and you can build Mellanox from scratch. I would suggest go with the stable 5.8-4.1.5.0-LTS.
https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

If need some suggestions on how to build mofed on vanila image follow this- https://github.com/Azure/azhpc-images/blob/master/alma/alma-8.x/alma-8.7-hpc/install_mellanoxofed.sh

[LiquidPT]

You could use any image with a supported Infiniband instance, and you can build Mellanox from scratch. I would suggest go with the stable 5.8-4.1.5.0-LTS. https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

If need some suggestions on how to build mofed on vanila image follow this- https://github.com/Azure/azhpc-images/blob/master/alma/alma-8.x/alma-8.7-hpc/install_mellanoxofed.sh

This issue was opened almost 4 years ago by the team lead on the project.

Mellanox OFED 5.8 does not support the CX3-Pro cards. As stated on that page, for that hardware you must use MLNX_OFED 4.9-x LTS.

That said, the fact that they support Ubuntu 20.04 with this driver is brand new information to me. We asked NVIDIA for this almost a year ago as Ubuntu 18.04 was ending support, and as recent as a few months ago we were told they wouldn't be supporting 20.04 with this driver.