Mellanox Technologies MCX4131A-BCAT User manual

Mellanox Technologies

www.mellanox.com

ConnectX®-4 Lx Single 40/50 Gb/s Ethernet QSFP28

Port Adapter Card User Manual

P/N: MCX4131A-BCAT, MCX4131A-GCAT

Rev 1.9

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,

CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,

GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellanox

CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open

Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,

Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular

Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, NPS®, Open Ethernet logo, PhyX®,

PlatformX®, PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of

Open Ethernet logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are

registered trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-ISﺴ WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT

USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST

ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY

QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES

CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT ARE DISCLAIMED . IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR

ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION )

HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE

PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH

DAMAGE.

Doc #: MLNX-15-5644 2Mellanox Technologies

Rev 1.9 3Mellanox Technologies

Table of Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.1 Product Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.2 Operating Systems/Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.3 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.4 Features and Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1 Ethernet QSFP28 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3 LEDs Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 3 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.2 Operating Systems/Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.3 Software Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.3 Pre-installation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.4 Bracket Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.4.1 Removing the Existing Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.4.2 Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.5 Adapter Card Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.6 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.6.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.6.2 Adapter Card Un-installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.6.3 Card Un-installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.7 Identify the Card in Your System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.7.1 On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.7.2 On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Chapter 4 Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1 Linux Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Rev 1.9 4Mellanox Technologies

4.1.2 Downloading Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1.3 Installing Mellanox OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1.3.1 Installation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.3.2 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.3.3 Installation Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.3.4 openibd Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.3.5 Driver Load Upon System Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.3.6 mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.4 Uninstalling Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.5 Installing MLNX_OFED using YUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.5.1 Setting up MLNX_OFED YUM Repository . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.5.2 Installing MLNX_OFED using the YUM Tool. . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.5.3 Uninstalling Mellanox OFED using the YUM Tool . . . . . . . . . . . . . . . . . . . 32

4.1.6 Installing MLNX_OFED using apt-get . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.6.1 Setting up MLNX_OFED apt-get Repository . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.6.2 Installing MLNX_OFED using the apt-get Tool. . . . . . . . . . . . . . . . . . . . . . 33

4.1.6.3 Uninstalling Mellanox OFED using the apt-get Tool . . . . . . . . . . . . . . . . . 34

4.1.7 Updating Firmware After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.1.7.1 Updating the Device Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.1.7.2 Updating the Device Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.1.7.3 Updating the Device Firmware Automatically upon System Boot . . . . . 35

4.1.8 UEFI Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.1.8.1 Enrolling Mellanox's x.509 Public Key On your Systems . . . . . . . . . . . . . 35

4.1.8.2 Removing Signature from Kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . 36

4.1.9 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.2 Windows Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.2.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.2.2 Downloading Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.2.3 Installing Mellanox WinOF-2 Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2.3.1 Attended Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2.3.2 Unattended Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.2.4 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.2.5 Extracting Files Without Running Installation. . . . . . . . . . . . . . . . . . . . . . . 45

4.2.6 Uninstalling Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.2.6.1 Attended Uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.2.7 Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.2.8 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.3 VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.2 Installing VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.3 Removing Previous Mellanox Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.3.4 Loading/Unloading Driver Kernel Modules. . . . . . . . . . . . . . . . . . . . . . . . . 50

Rev 1.9 5Mellanox Technologies

4.3.5 Firmware Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Chapter 5 Updating Adapter Card Firmware. . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.1 Firmware Update (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Chapter 6 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6.2 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

6.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 7 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

7.1 MCX4131A-BCAT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

7.2 MCX4131A-GCAT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

7.3 Airflow Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.4 Adapter LEDs Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

7.5 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . 59

7.6 Bracket Mechanical Drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Appendix A Interface Connectors Pinouts . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

A.1 QSFP28 Connector Pinouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

A.2 PCI Express x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Appendix B Finding the MAC and Serial Number on the Adapter Card . . . . 64

Appendix C Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Appendix D Avertissements de sécurité d’installation (Warnings in French) 67

Appendix E Sicherheitshinweise (Warnings in German) . . . . . . . . . . . . . . . . 69

Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 71

Rev 1.9 6Mellanox Technologies

List of Tables

Table 1: Revision History Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Table 2: Related Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 3: Single-port 40/50 Gb/s Ethernet Adapter Card. . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Table 4: Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 5: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Table 6: mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Table 7: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Table 8: Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Table 9: MCX4131A-BCAT Specifications Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Table 10: MCX4131A-GCAT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Table 11: Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Table 12: Connector Pin Number and Name to Signal Name Map . . . . . . . . . . . . . . . . . . . 61

Rev 1.9 7Mellanox Technologies

List of Figures

Figure 1: PCI Device (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Figure 2: Installation Results (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Figure 3: Mechanical Drawing of the MCX4131A-BCAT/MCX4131A-GCAT Adapter Card . . . . . . . 59

Figure 4: Single-port Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Figure 5: Connector and Cage Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Figure 6: PCIe x8 Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Figure 7: MCX4131A-BCAT Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Figure 8: MCX4131A-GCAT Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Rev 1.9 8Mellanox Technologies

Revision History

This document was printed on January 30, 2018.

Table 1 - Revision History Tables

Date Rev Comments/Changes

January 2017 1.9 • Added a note to System Requirements on page 16.

June 2017 1.8 • Updated Cable Installation on page 19

• Updated Adapter LEDs Operations on page 59

June 2017 1.7 • Updated the following tables:

•MCX4131A-BCAT Specifications on page 56

•MCX4131A-GCAT Specifications on page 57

September 2016 1.6 • Updated Adapter Card Installation Instructions on page 18

• Added Adapter Card Un-installation InstructionsSafety Precautions on

page 20

• Updated the following tables:

•MCX4131A-BCAT Specifications on page 56

•MCX4131A-GCAT Specifications on page 57

August 2016 1.5 • Updated Cable Installation on page 19

• Updated Adapter LEDs Operations on page 59

April 2016 1.4 Updated Adapter LEDs Operations on page 59

March 2016 1.3 • Removed I2C-compatible Interface on page 21

• Updated Features and Benefits on page 13

February 2016 1.2 Updated Installing the New Bracket on page 17

February 2016 1.1 • Updated Linux Driver on page 23

• Updated Windows Driver on page 38

• Updated VMware Driver on page 49

• Added note to Finding the MAC and Serial Number on the Adapter Card

on page 64

December 2015 1.0 First Release

Rev 1.9 9Mellanox Technologies

About this Manual

This User Manual describes Mellanox Technologies ConnectX®-4 Lx 40/50 Gb/s Ethernet Sin-

gle QSFP28 port PCI Express x8 adapter cards. It provides details as to the interfaces of the

board, specifications, required software and firmware for operating the board, and relevant docu-

mentation.

Intended Audience

This manual is intended for the installer and user of these cards.

The manual assumes basic familiarity with Ethernet network and architecture specifications.

Popular Computer Hardware manuals by other brands

EMC2

EMC2 VNX Series Hardware Information Guide

Panasonic

Panasonic DV0PM20105 Operation manual

Mitsubishi Electric

Mitsubishi Electric Q81BD-J61BT11 user manual

Gigabyte

Gigabyte B660M DS3H AX DDR4 user manual

Raidon

Raidon iT2300 Quick installation guide

National Instruments

National Instruments PXI-8186 user manual

Hauppauge

Hauppauge WinTV-HVR-2250 Quick installation guide

Kontron

Kontron CP6002 user guide

Manhattan

Manhattan 702843 user manual

Aphex

Aphex 230 instruction manual

AMD

AMD Phenom II Product data sheet

Intel

Intel NUC 12 Pro user guide

dbx

dbx Zone Pro 1260 user manual

Galaxy

Galaxy GHDX2-2430S-24F4D Installation and hardware reference manual

Intel

Intel AXXRMFBU4 Quick installation user's guide

Kontron

Kontron DIMM-PC/MD product manual

ST X-NUCLEO-SAFEA1B How to use

STEINWAY LYNGDORF

STEINWAY LYNGDORF SP-1 installation manual