Mellanox Technologies IS5025D-1BRC User manual

www.mellanox.com

InfiniScale® IV 36-Port QSFP 40 Gb/s

InfiniBand Switch User Manual

P/N:

IS5025D-1BRC, IS5025Q-1BFC, IS5025Q-1SRC, IS5025Q-1SFC, IS5025Q-1BRC, IS5025Q-2SRC,

IS5025Q-2BRC, IS5030Q-1BFC, IS5030Q-1SFC, IS5030Q-1BRC, IS5030Q-2BFC, IS5030Q-2SFC,

IS5030Q-2BRC, IS5035Q-1BFC, IS5035Q-1BRC, IS5035Q-2BFC, IS5035Q-2BRC, IS5035Q-1SFC,

IS5035Q-2SFC

Rev 2.9

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

PO Box 586 Hermon Building

Yokneam 20692

Israel

Tel: +972-4-909-7200

Fax: +972-4-959-3245

Mellanox, BridgeX, ConnectX, Virtual Protocol Interconnect, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI

are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct, FabricIT, and PhyX are trademarks of Mellanox

Technologies, Ltd. All other trademarks are property of their respective owners.

All other marks and names mentioned herein may be trademarks of their respective companies.

QDR InfiniBand Switch Platform User Manual

Document Number: 3133

Rev 2.9

Mellanox Technologies

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE

PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE

OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE

CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX

TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX

TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE

DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,

INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,

PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORYOF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND

RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

QDR InfiniBand Switch Platform User Manual Rev 2.9

Mellanox Technologies

Table of Contents

Table of Contents 3

List of Figures 5

List of Tables 7

Revision History 8

About this Manual 10

Intended Audience 10

Related Documentation 10

Conventions 11

Mellanox Part Numbering Legend 11

Chapter 1 Overview 12

1.1 Serial Number and Product Version Information 12

Chapter 2 Internally Managed vs. Externally Managed 13

Chapter 3 Basic Operation and Installation 14

3.1 Switch Platform Hardware Overview 14

3.1.1 Status LEDs 14

3.1.2 Reset Button 18

3.2 Air Flow 19

3.3 QSFP Cable Power Budget Classification 20

3.4 Interfaces 20

3.4.1 Port Connector Interfaces 20

3.4.2 Management and FW updating Interfaces 21

3.5 Package Contents 23

3.6 Switch Platform Installation and Operation 24

3.6.1 Installation Safety Warnings 24

3.6.2 Mechanical Installation 26

3.6.3 Grounding the Switch 32

3.6.4 Power Connections and Initial Power On 32

3.6.5 Extracting and Inserting the Power Supply Unit 33

3.6.6 InfiniBand Cable Installation 35

3.6.7 Extracting and Inserting the Fan Unit 35

3.7 Disassembly of the Switch from the Rack 36

3.8 Disposal 37

Chapter 4 Management and Tools Overview 38

4.1 Network Management and Clustering Software 38

4.2 Internally Managed Switch System 38

4.2.1 Subnet Management 39

4.2.2 Fabric Management with UFM 39

4.2.3 Configuring the Switch for the First Time 40

4.2.4 Starting a Remote Connection to the Switch 40

4.2.5 Downloading Firmware 40

4.3 Unmanaged (Externally Managed) Switch 41

4.3.1 I2C Connector 41

4.3.2 Current Firmware Revision 41

4.3.3 How to Get Mellanox Firmware Tools (MFT) 44

4.3.4 Open SM 44

4.4 Updating Firmware on Managed Switches 45

Chapter 5 Troubleshooting 46

Rev 2.9

Mellanox Technologies

Appendix A Specification 49

Appendix B Passing the Power Cord From the Connector Side to the Power Side in a Full Rack57

Appendix C QSFP Interface 68

Appendix D RJ45 CONSOLE Interface 70

Appendix E Replacement Parts Ordering Numbers 71

Appendix F Avertissements de sécurité d’installation (French) 72

Appendix G Installation - Sicherheitshinweise (German) 74

Appendix H Advertencias de seguridad para la instalación (Spanish) 76

Appendix I Special Regulations Regarding Finland, Sweden, Denmark, and Norway 79

QDR InfiniBand Switch Platform User Manual Rev 2.9

Mellanox Technologies

List of Figures

Figure 1: Pull Out Tab 12

Figure 2: QSFP Switch System Power and Connector Side Panels 14

Figure 3: Power, Fan, and System LEDs 15

Figure 4: System Health LED 16

Figure 5: IS5030-2BRC QSFP Power Side Panel 17

Figure 6: PSU Status LEDs 17

Figure 7: Reset Button 19

Figure 8: Port Numbering 21

Figure 9: Top and Bottom Ports 21

Figure 10: Management Interfaces 21

Figure 11: RJ45 I2C Connector 22

Figure 12: Rack Rail Kit Parts MIS00083/85 27

Figure 13: Rack Rail Kit Parts MIS000079 28

Figure 14: Which Side of the Rack Do You Want the Connectors? 29

Figure 15: Making Room for the Power Cord 30

Figure 16: Screwing on the Bracket 30

Figure 17: Screwing on the Rail 31

Figure 18: Inserting the Caged Nuts 31

Figure 19: Connect Bracket to Rack Vertical support 31

Figure 20: Ground Connection 32

Figure 21: Two Power Inlets - Electric Caution Notifications 33

Figure 22: Power Supply Unit Extraction 34

Figure 23: PSU Pulled Out 34

Figure 24: Top and Bottom Ports 35

Figure 25: Air Flow Labels 36

Figure 26: Fan Module Latches 36

Figure 27: Host Connection 40

Figure 28: MTUSB-1 with Cables 43

Figure 29: I2C Cable Connected to IS5025 43

Figure 30: Rack Installation Kit Parts 57

Figure 31: Rack Rail Kit Parts MIS00083/85 58

Figure 32: Which Side of the Rack Do You Want the Connectors? 59

Figure 33: Screwing the Rail Slide onto the switch 59

Figure 34: Caged Nut Spacing 60

Rev 2.9

Mellanox Technologies

Figure 35: Screwing in the Rails 60

Figure 36: Rack Rail Kit Parts MIS000079 61

Figure 37: Which Side of the Rack Do You Want the Connectors? 62

Figure 38: Full Rack 63

Figure 39: Inserting the Caged Nuts 63

Figure 40: Put the Cables Through the 1U Space 64

Figure 41: Placing the Power Cords 64

Figure 42: Power Cord Through the Rail Slide 65

Figure 43: Screwing on the Bracket 65

Figure 44: Making Room for the Power Cord 66

Figure 45: Screwing on the Rail 66

Figure 46: Power Cord Through the Bracket 66

Figure 47: Power Side View 67

Figure 48: Connect Bracket to Rack Vertical support 67

QDR InfiniBand Switch Platform User Manual Rev 2.9

Mellanox Technologies

List of Tables

Table 1: Revision History Table 8

Table 2: Reference Documents 10

Table 3: Switch Management 13

Table 4: System Status LED Configurations 16

Table 5: Connector Physical and Logical Link Indications 16

Table 6: PSU Status LED Configurations 17

Table 7: PSU Status LED Configurations 18

Table 8: Fan Status LED Configurations 18

Table 9: Air Flow Direction 20

Table 10: IS5025 Specification Data 49

Table 11: IS5030 Specification Data 51

Table 12: IS5035 Specification Data 53

Table 13: InfiniBand QSFP Connector Pinout 68

Table 14: RJ45 CONSOLE Pinout 70

Table 15: Replacement Parts Ordering Numbers 71

Rev 2.9

Mellanox Technologies

Revision History

Table 1 - Revision History Table

Date Revision Description

June 2011 Rev 2.9 Removed references to paper clip.

Changes to reset button section

Added warning do not use the switch as a shelf.

June 2011 Rev 2.8 Changed three minute boot up time to 5 minutes.

June 2011 Rev 2.7 Removed Flashing green from Table 4

June 2011 Rev 2.6 Removed products scheduled for End of Life.

Removed Flashing green from Table 4

May 2011 Rev 2.5 Fixed power numbers

May 2011 Rev 2.4 Changes to Sections 1.1 and 2

May 2011 Rev 2.3 Changes to the mangement sections

Removed Section 6

May 2011 Rev 2.2 Changed Grounding section.

Removed note from Table 4

Changed unmanaged FW update procedure

Oct. 2010 Rev 2.1 Added information regarding the MIS5031 port numbering and port activation.

Oct. 2010 Rev 2.0 New power numbers

Cable power level changed to 2.0W level 2

July 2010 Rev 1.9.1 Added “QSFP Cable Power Budget Classification”

June 2010 Rev 1.9 Added section Getting the License

Added Note “Each Ethernet connector gets connected to Ethernet switches. These

switches must be configured to 10/100M auto-negotiation.”

April 2010 Rev 1.8 Added appendix with instructions for bringing the power cord from one side of the

switch to the other in a full rack.

Added Safety Warnings in Spanish

April 2010 Rev 1.7 Removed DDR switches.

January 2010 Rev 1.6 Added instructions in the installation section regarding putting the power cord through

the bracket Figure 15 and note on page 26.

Removed DDR switches.

January 2010 Rev 1.5 Updates figures to include the power side I2C connector.

Added Section 3.3.2.5 I2C Interface on the power side of the switch

December 2009 Rev 1.4 Added Chapter 6 regarding FabricIT-EFM licensing information.

Added Chapter 2Management modules and CPU connections

Section 4.3 on unmanaged switches including

The I2C connector should only be used when the FW cannot be updated in-band.

QDR InfiniBand Switch Platform User Manual Rev 2.9

Mellanox Technologies

October 2009 Rev 1.3 Fixed Depth of Standard switch in the Spec sheets

Added RJ45 console pinout to appendix.

Fixed Status Fan LED info on page 13. Added to Troubleshooting section.

Updated power numbers

Added IS5025Q-2SRC, IS5025Q-2BRC model numbers

September 2009 Rev 1.2 Added Spec sheets for IS5025 and IS5035

updated revision table

September 2009 Rev 1.1 Updated power numbers

September 2009 Rev 1.0 Initial Release

Table 1 - Revision History Table

Date Revision Description

Rev 2.9

Mellanox Technologies

About this Manual

This manual describes the installation and basic use of the Mellanox IS50XX switch, which is

based on the InfiniScale IV InfiniBand switch device.

Intended Audience

This manual is intended for users and system administrators responsible for installing and setting

up the switch platforms listed above.

The manual assumes familiarity with the InfiniBand

Architecture Specification.

Related Documentation

Additional Documentation available from Mellanox:

Table 2 - Reference Documents

Switch Firmware and

Firmware Update

Tools

See

http://www.mellanox.com > Support > Download Firmware Tools

Note that the Switch System described in this manual is based on Mellanox

Technologies’ InfiniScale

IV switch device.

Mellanox OFED

Stack for Linux

User’s Manual

See

http://www.mellanox.com > Support > InfiniBand Software and Drivers

Click “Mellanox OpenFabrics Enterprise Distribution for Linux

(MLNX_OFED)”

Select the Linux User’s Manual

The embedded OS and tools on the CPU in the management module is a

subset of the Mellanox OFED stack.

FabricIT Enterprise

Fabric Management

Software CLI User’s

Manual

See

http://www.mellanox.com > Products > Management Software

Select “FabricIT EFM - InfiniBand Fabric Management"

Click "Software and Documentation Download center"

select User Manual

Mellanox Firmware

Tools (MFT)

User’s Manual

Document # 2329

The MFT (Mellanox Firmware Tools) package is a set of firmware tools.

The manual supplied with this package provides an overview of the firm-

ware its installation and replacement. The MFT can be downloaded with its

documentation at:

http://www.mellanox.com > Support > Download Firmware Tools

Other manuals for IS5025D-1BRC

This manual suits for next models

Table of contents

Other Mellanox Technologies Switch manuals

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 User manual

Mellanox Technologies

Mellanox Technologies MSX6536-10R Manual

Mellanox Technologies

Mellanox Technologies SX6720 Instructions for use

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies InfiniScale III M2401G 20Gb/s InfiniBand User manual

Mellanox Technologies

Mellanox Technologies IS5025D-1BRC User manual

Mellanox Technologies

Mellanox Technologies Spectrum SN2100 User manual

Mellanox Technologies

Mellanox Technologies SwitchX User manual

Mellanox Technologies

Mellanox Technologies SX10XX Instructions for use

Mellanox Technologies

Mellanox Technologies SX1710 User manual

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 User manual

Mellanox Technologies

Mellanox Technologies Spectrum SN2700 Instructions for use

Mellanox Technologies

Mellanox Technologies Spectrum SN2700 Instructions for use

Mellanox Technologies

Mellanox Technologies SX6710 Instructions for use

Mellanox Technologies

Mellanox Technologies MIS5025 User manual

Mellanox Technologies

Mellanox Technologies MSX6512-4R Manual

Mellanox Technologies

Mellanox Technologies SwitchX-2 MSX6018F-1SFS Instructions for use

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1UNC User manual

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Instruction Manual

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Operating manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies Grid Director 4036 User manual

Mellanox Technologies

Mellanox Technologies SX1710 User manual

Mellanox Technologies

Mellanox Technologies MTS3610 User manual

Popular Switch manuals by other brands

URC

URC Total Control MFS-8 installation manual

Azbil

Azbil HPQ-D11 user manual

Sony

Sony DFS-900M operating instructions

SICK

SICK i17S operating instructions

F&F

F&F SZR-277 quick start guide

Shure

Shure UAMS/BK user guide

Dräger

Dräger X-zone Switch Off Instructions for use

Asco

Asco Joucomatic 349 Series manual

BT Mini Hub user guide

D-Link

D-Link DES-1226G manual

Johnson Controls

Johnson Controls FS80-C installation instructions

Alcatel-Lucent

Alcatel-Lucent 7450 ESS-12 Configuration guide

MicroNet

MicroNet SP1200A user manual

Ruijie

Ruijie RG-PF2920 Series Hardware installation and reference guide

Altinex

Altinex DA1914SX user guide

MiLAN

MiLAN MIL-SM2401M user guide

RJM

RJM MASTERMIND PBC/6X user manual

Atlas

Atlas MMK-KVM8 owner's manual