The Nvidia GB10 ConnectX-7 200GbE Networking Is Different
10 days ago
- #GB10
- #NVIDIA
- #ConnectX-7
- NVIDIA DGX Spark and GB10 platforms feature NVIDIA ConnectX-7 networking, which is a key highlight.
- The GB10 platforms use two PCIe Gen5 x4 links to achieve 200Gbps networking, instead of a single PCIe Gen5 x8 link.
- Four network interfaces are presented to the OS, each connected via PCIe Gen5 x4 links to the ConnectX-7 NIC.
- Achieving full 200Gbps requires careful configuration, including using RoCE and NVIDIA Perftest tool, and aligning interfaces correctly.
- Scaling out GB10 platforms can be challenging due to the need to manage multiple interfaces and ensure optimal performance.
- NVIDIA confirmed that the dual x4 link design is due to a limitation in the GB10 chip, which cannot provide more than x4-wide PCIe per device.
- The article serves as a reference for understanding the peculiarities of ConnectX-7 implementation in GB10 platforms.