Mellanox Technologies MSX6512-NR Manual

www.mellanox.com

216 Port InfiniBand FDR Switch Platform

Hardware Installation Guide

PN: MSX6512-NR, MSX6512-4R

Rev 1.0

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

Beit Mellanox

PO Box 586 Yokneam 20692

Israel

Tel: +972 (0)4 909 7200 ; 

+972 (0)74 723 7200

Fax: +972 (0)4 959 3245

Mellanox, Mellanox logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, SwitchX, Virtual Protocol

Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. FabricIT, MLNX-OS, Unbreakable-Link, UFM and

Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide

Document Number: 3877

Mellanox Technologies2

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE

PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE

OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE

CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX

TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX

TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE

DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,

INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,

PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND

RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0

Mellanox Technologies 3

Table of Contents

Chapter 1 Chassis Installation 9

1.1 Environmental and Safety Recommendations 9

1.2 ChassisPackageContents 10

1.3 LeafPackageContents 10

1.4 SpinePackageContents 11

1.5 ManagementPackageContents 11

1.6 PhysicalInstallation 11

1.6.1

ESD (ElectroStatic Discharge) Connection 11

1.6.2 Installation Procedure 12

1.7 Installingthe Chassis 18

1.8 InsertingtheLeafs 22

1.9 Inserting a Management Module 23

1.10 Inserting a Spine Board 24

Chapter 2 Installing the Cable Holder 25

Chapter 3 Ground Connections 28

Chapter 4 Power Connections 29

4.1 Powering Up the Switch Platform 29

Chapter 5 InfiniBand QSFP Cable Installation 31

5.1 PortConnectorInterfaces 31

5.2 SupportedApprovedCables 32

5.3 CablePower Classes 32

Chapter 6 Chassis Power Up 33

6.1 Power Supply, Management Board, and Spine Board Indicator Status at Power ON 34

Chapter 7 InfiniBand Fabric Initialization and Management 35

7.1 Configuring the Switch Management Modules for the First Time 35

7.2 RerunningtheWizard 40

Chapter 8 MLNX-OS Management Software 41

8.1 Starting an SSH Connection to the Switch 41

8.2 Starting a WebUI Connection to the Switch 41

Chapter 9 Resetting the Switch – RST 43

Chapter 10 TroubleShooting 44

10.1 PowerSupplyUnit 44

10.2 LeafBoard 44

10.3 ManagementModule 45

10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit 45

10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit 45

10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit 46

10.4 SpineBoard 46

10.5 MLNX-OSSoftware 46

Mellanox Technologies4

Rev 1.0

List of Figures

Figure 1: Shock and Tilt Stickers 13

Figure 2: Shelf Installation Kit Parts 14

Figure 3: Inserting the Caged Nuts for the Shelf 16

Figure 4: Use These Holes for the Rail Slides 16

Figure 5: Connect Rail Slide to Rack Vertical support 17

Figure 6: Inserting the Caged Nuts for the Faceplate 17

Figure 7: Chassis Rails and Rail Slides 18

Figure 8: Screw the Handles Onto the Chassis 19

Figure 9: The Rails are Already Connected Onto the Chassis 19

Figure 10: Chassis on the Shelf 20

Figure 11: Put on the Rail Slide 21

Figure 12: Face Plate Mounting Bolt Locations 21

Figure 13: Move the Rail Slide to the Vertical Support 22

Figure 14: Ejector Latch 23

Figure 15: Ejector Latch 23

Figure 16: Spine Board Insertion 24

Figure 17: Ejector Latch 24

Figure 18: Cable Holders 25

Figure 19: Inserting the Caged Nuts for the Cable Holder 26

Figure 20: Cable Holder Placement 27

Figure 21: Ground Connection 28

Figure 22: Multiple Power Inlets - Electric Caution Notification 30

Figure 23: Spine Module 30

Figure 24: Port Numbering 32

Figure 25: Top and Bottom Ports 32

Figure 26: Spine Side Panel Display Status Indications 34

Figure 27: Management Module Status Indications for Normal Operation 34

Figure 28: Connecting the Harness to the Console 35

Figure 29: Web UI Login Page 42

Figure 30: Reset Button 43

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0

Mellanox Technologies 5

List of Tables

Table 1: Revision History of this User’s Manual 6

Table 2: Reference Documents and Web Sites 7

Table 3: Serial Terminal Program Configuration 36

Table 4: Configuration Wizard Session - IP Configuration by DHCP 36

Table 5: Configuration Wizard Session - IP Zeroconf Configuration 38

Table 6: Configuration Wizard Session - Static IP Configuration 39

Mellanox Technologies6

Rev 1.0

Revision History

Table 1 - Revision History of this User’s Manual

Revision Date Details

1.0 July2010 Initial release

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0

Mellanox Technologies 7

About this Manual

This manual provides installation and set-up instructions for the SX6512 QSFP Chassis

InfiniBand Switch Platform.

Intended Audience

This manual is intended for users and system administrators responsible for installing and

setting up the switch platform.

The manual assumes familiarity with the InfiniBand®architecture specification.

Related Documentation

The documentation set accompanying the QSFP Chassis InfiniBand Switch platform

includes the following:

Table 2 - Reference Documents and Web Sites

Document Conventions

Throughout this manual, the name SX6512 and the terms chassis and switch are used to

describe the switch, unless explicitly indicated otherwise.

The following pictures are used throughout this document to indicate information that is

important to the user.

InfiniBand Architecture Specification, Vol. 1,

Release 1.2.1 The InfiniBand Architecture Specification that is provided by

IBTA

Switch Hardware Errata For any possible errata due to hardware issues see the switch

support product page. This requires a customer support login.

Mellanox MLNX-OS™ SwitchX® Software WebUI

User’s Manual WebUI Overview for MLNX-OS software.

Mellanox MLNX-OS™ SwitchX® Software User

Manual This document contains information regarding configuring and

managing Mellanox Technologies' SwitchX Switch Platforms.

MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the

commands available through MLNX-OS with explanations and

examples.

MLNX-OS™ Software Configuration Guide Configuration Guide for MLNX-OS displaying different config-

uration scenarios.

This symbol makes recommendations to the user.

Mellanox Technologies8

Rev 1.0

This symbol indicates information that is helpful to the user.

This symbol indicates a situation that can potentially cause damage to hardware or

software.

WARNING! This symbol indicates a situation that can potentially cause personal

injury and / or damage to hardware or software.

216 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0

Mellanox Technologies 9

1 Chassis Installation

Installation and initialization of the chassis is a simple process requiring attention to the

normal mechanical, power, and thermal precautions for rack-mounted equipment. Your

chassis comes only with the power supplies and fans pre-installed. The rest of the open-

ings are populated with blanks. All of the leafs, spines, and management modules come

shipped in separate packages.

The chassis requires initial configuration to get the chassis and Fabric management up and

running through remote management. see “Configuring the Switch Management Modules

for the First Time” on page 35.

1.1 Environmental and Safety Recommendations

The following are Mellanox recommendations.

This chassis can be installed in standard 19” racks that have depths between 65cm

(25.6in.) and 80cm (31.5in.) between the vertical supports of the rack.

This unit is intended for installation in a Restricted Access Location. A restricted

access area can be accessed only through the use of a special tool, lock and key, or

other means of security.

Recommended ambient temperature in the System room is 20o± 5oC.

Recommended humidity ranges is 40% ± 15% without condensing.

It is highly recommended that the installation sites be as isolated as possible from all

sources of radio transmissions and electrical interference.

It is highly recommended that the installation site building be equipped with a light-

ning rod.

It is highly recommended that the installation site be equipped with smoke detectors

and a fire alarm warning system.

Chassis Installation

Mellanox Technologies10

Rev 1.0

1.2 Chassis Package Contents

The chassis package includes:

• 1 chassis with the following modules installed:

4 fans

4 PSUs

If you are not using a mechanical lift to install the chassis,reduce the weight of the chassis by

removing all of the power supply units, and fan units, and put aside for reinstallation after the chas-

sis is installed in the rack.

• 1 installation kit

• 1 box containing 4 power cords 250v 15a 2.0M, C14 to C13, USA UL Standard

• 1 cable management kit

• 1 Installation Guide

Before you install your new SX6512 chassis, unpack the system and check to make sure that all

the parts have been sent, check this against the parts list see “Installation and Cable Management

Kit Parts” on page 12. Check the container for visible damages that may have occurred during

shipping.

1.3 Leaf Package Contents

The leafs are ordered by the customer and are shipped 4 to a box. The customer will receive as

many boxes as needed to fill the order.

The system requires a KVA rated UPS system. It is recommended that a UPS system

be installed to protect the equipment in the event of unexpected power failure.

Make sure that the outlets and circuits will not be overloaded. Spread out the load over

at least two or three circuits or use a 3 phase circuit. N+N grid redundancy requires 2

independent power supply sources.

If anything is damaged or missing, contact your customer representative immediately.

The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Guaran-

tee proper ventilation, by leaving 8cm (3”) of space to the front and rear of the switch.

This will ensure proper air flow through the chassis. This is crucial for maintaining

good airflow at ambient temperature. In particular, route cables such that they do not

impede the air into or out of the chassis.

This manual suits for next models

Table of contents

Other Mellanox Technologies Switch manuals

Mellanox Technologies

Mellanox Technologies InfiniScale III M2401G 20Gb/s InfiniBand User manual

Mellanox Technologies

Mellanox Technologies SwitchX-2 MSX1012B-1BFS User manual

Mellanox Technologies

Mellanox Technologies InfiniScale III User manual

Mellanox Technologies

Mellanox Technologies MIS5022Q-1BFR User manual

Mellanox Technologies

Mellanox Technologies MIS5200Q-4DNC Manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies SX10XX Instructions for use

Mellanox Technologies

Mellanox Technologies MIS5035Q-2SFC User manual

Mellanox Technologies

Mellanox Technologies SwitchX-2 MSX6018F-1SFS Instructions for use

Mellanox Technologies

Mellanox Technologies TX6100 MetroX Series Instructions for use

Mellanox Technologies

Mellanox Technologies Spectrum SN2100 Instructions for use

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1UNC User manual

Mellanox Technologies

Mellanox Technologies QM8700 User manual

Mellanox Technologies

Mellanox Technologies SX6710 Instructions for use

Mellanox Technologies

Mellanox Technologies SX6015 User manual

Mellanox Technologies

Mellanox Technologies MIS5300 Manual

Mellanox Technologies

Mellanox Technologies SX1012 User manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies IS5025D-1BRC User manual

Mellanox Technologies

Mellanox Technologies SB7700 User manual

Mellanox Technologies

Mellanox Technologies IS5025D-1BRC User manual

Mellanox Technologies

Mellanox Technologies SX1710 User manual

Mellanox Technologies

Mellanox Technologies Spectrum SN2700 Instructions for use

Mellanox Technologies

Mellanox Technologies MIS5024 User manual

Popular Switch manuals by other brands

URC

URC Total Control MFS-8 installation manual

Azbil

Azbil HPQ-D11 user manual

Sony

Sony DFS-900M operating instructions

SICK

SICK i17S operating instructions

F&F

F&F SZR-277 quick start guide

Shure

Shure UAMS/BK user guide

Dräger

Dräger X-zone Switch Off Instructions for use

Asco

Asco Joucomatic 349 Series manual

BT Mini Hub user guide

D-Link

D-Link DES-1226G manual

Johnson Controls

Johnson Controls FS80-C installation instructions

Alcatel-Lucent

Alcatel-Lucent 7450 ESS-12 Configuration guide

MicroNet

MicroNet SP1200A user manual

Ruijie

Ruijie RG-PF2920 Series Hardware installation and reference guide

Altinex

Altinex DA1914SX user guide

MiLAN

MiLAN MIL-SM2401M user guide

RJM

RJM MASTERMIND PBC/6X user manual

Atlas

Atlas MMK-KVM8 owner's manual