Nvidia Mellanox ConnectX-5 User manual

Exported on Sep/19/2022 09:36 AM



NVIDIA ConnectX-5 Ethernet Adapter Cards

for OCP Spec 3.0 User Manual



2
Table of Contents
Introduction..................................................................................... 7
Features and Benefits ...............................................................................9
Operating Systems/Distributions ................................................................ 11
Connectivity ........................................................................................ 11
Interfaces ......................................................................................12
Ethernet SFP28 and QSFP28 Interfaces ......................................................... 12
Networking Port LEDS Interface ................................................................. 12
PCI Express Interface.............................................................................. 13
FRU EEPROM ........................................................................................ 13
Heat Sink Interface ................................................................................ 13
SMBus Interface .................................................................................... 14
Voltage Regulators ................................................................................. 14
CPLD Interface ..................................................................................... 14
Thermal Sensors.................................................................................... 14
Hardware Installation ........................................................................15
Safety Warnings .................................................................................... 15
Installation Procedure Overview................................................................. 16
System Requirements ............................................................................. 17
Hardware Requirements ...................................................................... 17
Airflow Requirements ......................................................................... 17
Software Requirements ....................................................................... 17
Safety Precautions ................................................................................. 18
Pre-Installation Checklist ......................................................................... 18
OCP 3.0 Bracket Replacement Instructions .................................................... 18
OCP 3.0 Adapter Card Installation Instructions................................................ 21
Cables and Modules................................................................................ 22
Identifying the Card in Your System............................................................. 22
Cards Extraction Instructions .................................................................... 24
Driver Installation.............................................................................25
Windows Driver Installation ...................................................................... 25
Software Requirements ....................................................................... 25
Downloading NVIDIA WinOF-2 Driver ........................................................ 25

3
Installing NVIDIA WinOF-2 Driver ............................................................ 26
Uninstalling NVIDIA WinOF-2 Driver ......................................................... 33
Extracting Files Without Running Installation ............................................. 34
Firmware Upgrade ............................................................................. 36
Linux Driver Installation .......................................................................... 36
Prerequisites.................................................................................... 36
Downloading NVIDIA OFED .................................................................... 37
Installing NVIDIA OFED ........................................................................ 37
Installing MLNX_OFED Using YUM............................................................ 43
Installing MLNX_OFED Using apt-get Tool .................................................. 45
Updating Firmware After Installation....................................................... 46
UEFI Secure Boot............................................................................... 48
Performance Tuning ........................................................................... 49
VMware Driver Installation ....................................................................... 49
Hardware and Software Requirements ..................................................... 49
Installing NVIDIA NATIVE ESXi Driver for VMware vSphere ............................... 50
Removing Earlier NVIDIA Drivers ............................................................. 50
Firmware Programming ....................................................................... 51
Updating Adapter Firmware .................................................................52
Troubleshooting ...............................................................................53
General Troubleshooting.......................................................................... 53
Linux Troubleshooting ............................................................................. 53
Windows Troubleshooting......................................................................... 54
Specifications..................................................................................55
MCX562A-ACAI / MCX562A-ACAB Specifications............................................... 55
MCX566A-CCAI Specifications .................................................................... 57
MCX565A-CCAB Specifications ................................................................... 59
MCX566A-CDAB / MCX566A-CDAI Specifications............................................... 61
MCX565M-CDAI / MCX565M-CDAB Specifications .............................................. 63
Board Mechanical Drawing and Dimensions.................................................... 65
Brackets Mechanical Drawing and Dimensions ................................................ 66
Finding MAC and Serial Number on the Adapter Card ...................................67
Document Revision History ..................................................................69


About This Manual

This is the User Guide for NVIDIA® Ethernet adapter cards based on the ConnectX®-5 integrated

circuit device for Open Compute Project Spec 3.0. These adapters' connectivity provide the highest

performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers

used in Enterprise Data Centers and High-Performance Computing environments.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-5 Ex and

ConnectX-5 Ethernet adapter cards for OCP Spec 3.0.

IC in Use OPN Marketing Description

ConnectX®-5 MCX562A-ACAI ConnectX®-5 EN network interface card for OCP 3.0, with host

management, 25GbE Dual-port SFP28, PCIe3.0 x16, Internal Lock bracket

MCX562A-ACAB ConnectX®-5 EN network interface card for OCP 3.0, with host

management, 25GbE Dual-port SFP28, PCIe3.0 x16, Thumbscrew (Pull

Tab) bracket

MCX566A-CCAI ConnectX®-5 EN network interface card for OCP 3.0, with host

management, 100GbE Dual-port QSFP28, PCIe3.0 x16, Internal Lock

bracket

MCX565A-CCAB ConnectX®-5 EN network interface card for OCP 3.0, with host

management, 100GbE Single-port QSFP28, PCIe3.0 x16, Thumbscrew

(Pull Tab) bracket

ConnectX®-5 Ex MCX566M-GDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with Multi-Host

or Socket Direct and host management, 50GbE Dual-port QSFP28, PCIe

4.0/3.0 x16, Internal Lock bracket

MCX565M-CDAI

ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host

or Socket Direct and host management, 100GbE Single-port QSFP28,

PCIe4.0 x16, Internal Lock bracket

MCX565M-CDAB ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host

or Socket Direct and host management, 100GbE Single-port QSFP28,

PCIe4.0 x16, Thumbscrew (Pull Tab) bracket

MCX566A-CDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with host

management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Internal Lock

bracket

MCX566A-CDAB

ConnectX®-5 Ex EN network interface card for OCP 3.0, with host

management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Thumbscrew (Pull

Tab) bracket

Intended Audience

This manual is intended for the installer and user of these cards.The manual assumes basic

familiarity with Ethernet network and architecture specifications.

Technical Support

•

Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough

the following methods:

URL:https://www.nvidia.com> Support

E-mail:[email protected]

Customers who purchased NVIDIA Global Support Services, please see your contract for details

regarding Technical Support.

Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek

assistance through their reseller.

Related Documentation

NVIDIAMLNX_OFED for Linux User Manual

and Release Notes

User Manual and Release Notesdescribing

MLNx_OFEDfeatures, performance, band diagnostic,

tools content and configuration. SeeNVIDIA MLNX_OFED

for Linux Documentation.

WinOF-2 for WindowsUser Manual and

Release Notes

User Manual describing WinOF-2 features, performance,

Ethernet diagnostic, tools content and configuration.

SeeWinOF-2 for Windows Documentation.

NVIDIA VMware for Ethernet User Manual and

Release Notes

User Manual describing the various components of the

NVIDIA ConnectX® NATIVE ESXi stack.SeeVMware® ESXi

Documentation.

NVIDIA Firmware Utility (mlxup) User Manual

and Release Notes

NVIDIA firmware update and query utility used to update

the firmware.SeeNVIDIA Firmware Utility (mlxup)

Documentation.

NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware

management tools for a single node. SeeMFT User

Manual.

Open Compute Project 3.0 Specifications https://www.opencompute.org/

IEEE Std 802.3 Specification IEEE Ethernet Specifications

PCI Express Specifications Industry Standard PCI Express Base and Card

Electromechanical Specifications. Refer toPCI-SIG

Specifications.

LinkX Interconnect Solutions LinkX Ethernet cables and transceivers are designed to

maximize the performance of High-Performance

Computing networks, requiring high-bandwidth, low-

latency connections between compute nodes and switch

nodes.NVIDIA offers one of the industry’s broadest

portfolio of 40GbE, 56GbE, 100GbE, 200GbE and400GbE

cables, includingDirect Attach Copper cables (DACs),

copper splitter cables, Active Optical Cables (AOCs) and

transceivers in a wide range of lengths from 0.5m to

10km. In addition to meeting Ethernetstandards, NVIDIA

tests every product in an end-to-end environment

ensuring a Bit Error Rate of less than 1E-15.Read more

atLinkX Cables and Transceivers.

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to

mean PCI Express.

Introduction

This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex

integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest

performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers

used in Enterprise Data Centers and High-Performance Computing environments.

Important Notes:

ConnectX-5 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA

specifications and setup, as the OCPspec3.0 available at that time did not contain any S&V

definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is

in the midst of retesting these cards to comply with OCP spec 3.0.

In some of the OCP 3.0 cards, the insertion force that is required to install the card into a

particular PCI connector may exceed the maximum insertion force that is allowed by the

connector’s spec.NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion

process.

The following tables provide the ordering part number, port speed, number of ports, and PCI Express

speed per ConnectX-5 and ConnectX-5 ExOPNs.

ConnectX-5 Ex Adapter Cards for OCP 3.0

Part Number MCX566M-

GDAI

MCX565M-

CDAI

MCX565M-

CDAB

MCX566A-

CDAI

MCX566A-

CDAB

Ethernet Data Rate

Up to 50Gb/s

Up to 100Gb/s

Up to 100Gb/s Up to 100Gb/s Up to 100Gb/s

Network Connector

Type

x2 QSFP28 ports

x1 QSFP28 port

x1 QSFP28 port x2 QSFP28

ports x2 QSFP28 ports

PCI Express

Connectors

PCIe Gen 3.0 / 4.0 x16 (a), (b)

SERDES @ 8.0GT/s / 16.0GT/s

Bracket

Internal-Lock

Bracket Internal-Lock

Bracket

Pull-tab

Bracket

Internal-Lock

Bracket Pull-tab bracket

Host Management

✔ ✔ ✔ ✔

✔

Multi-Host/ Socket

Direct

✔ ✔ ✔ NA NA

IC Part Number

MT28804A0-

FCCF-CEM

MT28804A0-

FCCF-CEM

MT28804A0-

FCCF-CEM MT28808A0-

FCCF-CE

MT28808A0-

FCCF-CE

Device ID 4121 for Physical Function (PF)

4122 for Virtual Function (VF)

Part Number MCX566M-

GDAI

MCX565M-

CDAI

MCX565M-

CDAB

MCX566A-

CDAI

MCX566A-

CDAB

Form Factor

Dimensions

Small Form Factor (SFF)

4.52in. x 2.99in. (115mm x 76mm)

RoHS RoHS Compliant

ConnectX-5 Adapter Cards for OCP 3.0

Part Number MCX562A-

ACAI

MCX562A-

ACAB

MCX566A-CCAI MCX565A-CCAB

Ethernet Data Rate Up to 25Gb/s

Up to 25Gb/s Up to 100Gb/s Up to 100Gb/s

Network Connector

Type

x2 SFP28 ports

x2 SFP28 ports x2 QSFP28 ports x1 QSFP28 ports

PCI Express Connectors PCIe Gen 3.0 x16

SERDES @ 8.0GT/s

Bracket Internal-Lock

Bracket

Pull-tab Bracket Internal-Lock Bracket

Pull-tab Bracket

Host Management ✔ ✔ ✔

✔

IC Part Number MT27808A0-

FCCF-AE

MT27808A0-FCCF-

MT27808A0-FCCF-CE

Device ID 4119 for Physical Function (PF)

4120 for Virtual Function (VF)

Form Factor Dimensions Small Form Factor (SFF)

4.52in. x 2.99in. (115mm x 76mm)

RoHS RoHS Compliant

a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead),

and can support 200Gb/s when both network ports of the card run at 100Gb/s.

b.TheGen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be

tested when servers with Gen 4.0 support become available.

For more detailed information seeSpecifications.

•

Features and Benefits

Feature Description

PCI Express

(PCIe)

Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge connector. Gen 1.1

and 2.0 compatible.

Up to 100

Gigabit

Ethernet

NVIDIA adapters comply with the following IEEE 802.3 standards:

• 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet

• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes

• IEEE 802.3ba 40 Gigabit Ethernet

• IEEE 802.3by 25 Gigabit Ethernet

• IEEE 802.3ae 10 Gigabit Ethernet

• IEEE 802.3ap based auto-negotiation and KR startup

• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)

• IEEE 802.3ad, 802.1AX Link Aggregation

• IEEE 802.1Q, 802.1P VLAN tags and priority

• IEEE 802.1Qau (QCN)

• Congestion Notification

• IEEE 802.1Qaz (ETS)

• IEEE 802.1Qbb (PFC)

• IEEE 802.1Qbg

• IEEE 1588v2

• Jumbo frame support (9.6KB)

Memory EEPROM - The EEPROM capacity is 32Kbit.FRU I2C address is (0x50) and is accessible

through the PCIe SMBus (Note: Address 0x58 is reserved.)

SPI - includes 128Mb SPI Flash device (by WINBOND-NUVOTON).

Overlay

Networks

In order to better scale their networks, data center operators often create overlay networks

that carry traffic from individual virtual machines over logical tunnels in encapsulated formats

such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet

from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5

effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines

that encapsulate and de-capsulate the overlay protocol.

RDMA and

RDMA

overConver

ged

Ethernet

(RoCE)

ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged

Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet

networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced

congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services

over Layer 2 and Layer 3 networks.

NVIDIA

PeerDirect™

PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary

internal data copies between components on the PCIe bus (for example, from GPU to CPU), and

therefore significantly reduces application run time. ConnectX-5 advanced acceleration

technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

This section describes hardware features and capabilities. Please refer to the relevant

driver and/or firmware release notes for feature availability.



Feature Description

CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available CPU for

computation tasks.

Open vSwitch (OVS) offload using ASAP2(TM)

• Flexible match-action flow tables

• Tunneling encapsulation/decapsulation

Quality of

Service

(QoS)

Support for port-based Quality of Service enabling various application requirements for latency

and SLA.

Hardware-

based I/

OVirtualiza

tion

ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for

virtual machines within the server.

Storage

Acceleratio

A consolidated compute and storage network achieves significant cost-performance advantages

over multi-fabric networks. Standard block and file access protocols can leverage RDMA for

high-performance storage access.

• NVMe over Fabric offloads for target machine

SR-IOV ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation

and protection for virtual machines (VM) within the server.

NC-SI over

RMII The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be

connected to a BMC.

High-

Performanc

eAccelerati

ons

• Tag Matching and Rendezvous Offloads

• Adaptive Routing on Reliable Transport

• Burst Buffer Offloads for Background Checkpointing

Host

Managemen

Technology

NVIDIA’s host management technology for standard and multi-host platforms optimizes board

management and power, performance and firmware update management via NC-SI, MCTP over

SMBus and MCTP over PCIe, as well as PLDM for Monitor and Control DSP0248 and PLDM for

Firmware Update DSP0267.

Multi-Host

Technology NVIDIA Multi-Host™ technology, when enabled, allows multiple hosts to be connected into a

single adapter by separating the PCIe interface into multiple and independent interfaces. By

Using NVIDIA Multi Host™, ConnectX-5 lowers the total cost of ownership (TCO) in the data

center by reducing CAPEX (cables, NICs, and switch port expenses), and by reducing OPEX by

cutting down on switch port management and overall power usage. With NVIDIA Multi-Host™

technology powered by a shared buffer architecture, connection-tracking offloads, and RoCE

enhancements, ConnectX-5 offers an extremely flexible solution for today’s demanding data

center and cloud application.

NVIDIA

Socket

Direct™

NVIDIA Socket Direct technology brings improved performance to multi-socket servers by

enabling direct access from each CPU in a multi-socket server to the network through its

dedicated PCIe interface. With this type of configuration, each CPU connects directly to the

network; this enables the interconnect to bypass a QPI (UPI) and the other CPU, optimizing

performance and improving latency. CPU utilization improves as each CPU handles only its own

traffic, and not the traffic from the other CPU. NVIDIA’s OCP 3.0 cards include native support

for socket direct technology for multi-socket servers and can support up to 4 CPU sockets.

Wake-on-

LAN (WoL)

Supported

Other manuals for Mellanox ConnectX-5

Table of contents

Other Nvidia Computer Hardware manuals

Nvidia

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

Nvidia

Nvidia JETSON XAVIER NX Guide

Nvidia

Nvidia JETSON TX2 User manual

Nvidia

Nvidia DGX H100 User manual

Nvidia

Nvidia DGX A100 User manual

Nvidia

Nvidia TESLA K20 User manual

Nvidia

Nvidia GeForce3 Ti200 User manual

Nvidia

Nvidia Geforce PCI-Express Series User manual

Nvidia

Nvidia GEFORCE RTX 4090 User manual

Nvidia

Nvidia Quadro User manual

Nvidia

Nvidia Quadro PLEX 7000 User manual

Nvidia

Nvidia SP5200BLT User manual

Nvidia

Nvidia DGX-2 SYSTEM User manual

Nvidia

Nvidia A800 40GB Active User manual

Nvidia

Nvidia BlueField-2 BF2500 User manual

Nvidia

Nvidia Clara AGX User manual

Nvidia

Nvidia Tegra 250 Reference manual

Nvidia

Nvidia TESLA C2050 User manual

Nvidia

Nvidia Quadro M6000 Sync User manual

Nvidia

Nvidia PCI Express Series User manual

Nvidia

Nvidia DGX-2 SYSTEM User manual

Nvidia

Nvidia ConnectX-7 User manual

Nvidia

Nvidia Jetson TX2 NX User manual

Nvidia

Nvidia BlueField-3 User manual

Popular Computer Hardware manuals by other brands

EMC2

EMC2 VNX Series Hardware Information Guide

Panasonic

Panasonic DV0PM20105 Operation manual

Mitsubishi Electric

Mitsubishi Electric Q81BD-J61BT11 user manual

Gigabyte

Gigabyte B660M DS3H AX DDR4 user manual

Raidon

Raidon iT2300 Quick installation guide

National Instruments

National Instruments PXI-8186 user manual

Hauppauge

Hauppauge WinTV-HVR-2250 Quick installation guide

Kontron

Kontron CP6002 user guide

Manhattan

Manhattan 702843 user manual

Aphex

Aphex 230 instruction manual

AMD

AMD Phenom II Product data sheet

Intel

Intel NUC 12 Pro user guide

dbx

dbx Zone Pro 1260 user manual

Galaxy

Galaxy GHDX2-2430S-24F4D Installation and hardware reference manual

Intel

Intel AXXRMFBU4 Quick installation user's guide

Kontron

Kontron DIMM-PC/MD product manual

ST X-NUCLEO-SAFEA1B How to use

STEINWAY LYNGDORF

STEINWAY LYNGDORF SP-1 installation manual