Mellanox Technologies MIS5600Q-10DNC User manual

www.mellanox.com

648 Port InfiniBand QDR Switch Platform

Installation Guide

PN: MIS5600Q-10DNC

Rev 2.1

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies, Ltd.

PO Box 586 Hermon Building

Yokneam 20692

Israel

Tel: +972-4-909-7200

Fax: +972-4-959-3245

Mellanox®, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®, Virtual Protocol Interconnect and

Voltaire are registered trademarks of Mellanox Technologies, Ltd.

FabricIT, MLNX-OS and SwitchX are trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

648 Port InfiniBand QDR Switch Platform Installation Guide

Document Number: 3173

Rev 2.1

Mellanox Technologies

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE

PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE

OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE

CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX

TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX

TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE

DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,

INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,

PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORYOF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND

RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1

Mellanox Technologies 3

Table of Contents

Chapter 1 Chassis Installation 10

1.1 Environmental and SafetyRecommendations 10

1.2 Chassis PackageContents 11

1.3 LeafPackageContents 12

1.4 SpinePackageContents 12

1.5 ManagementPackageContents 12

1.6 PhysicalInstallation 12

1.6.1 Starting with the Rack 13

1.6.2 ESD Connection 14

1.6.3 Installation Procedure 14

1.6.4 Installing the Cable Holder 32

1.7 Installing the Leafs 33

1.8 Inserting a Management Module 33

1.9 Inserting a Spine Board 34

Chapter 2 Installing the Cable Holder 36

Chapter 3 Ground Connections 38

Chapter 4 Power Connections 39

4.1 Powering Up the Switch Platform 39

Chapter 5 InfiniBand QSFP Cable Installation 41

5.1 Cable PowerClasses 42

Chapter 6 Chassis Power Up 43

6.1 Power Supply, Management Board, and Spine Board Indicator Status at Power ON 44

Chapter 7 InfiniBand Fabric Initialization and Management 45

7.1 Configuring the Switch Management Modules for the First Time 45

7.2 Rerunning the Wizard 52

Chapter 8 Fabric Management Software 54

8.1 Downloading FabricIT Software and Documents 54

8.2 InstallingLicenses 54

8.3 Getting and Inserting the License Key 54

8.4 FabricITManagement and Inspection License 57

8.5 Starting an SSH Connection to the Switch 57

8.6 Starting a WebUI Connection to the Switch 57

Chapter 9 Resetting the Switch – RST 59

Rev 2.1

Mellanox Technologies

List of Figures

Figure 1: Distance Between the Vertical Supports 13

Figure 2: Installation Kit Parts 16

Figure 3: Container Screws 17

Figure 4: Vertical Rack Supports Dimension 19

Figure 5: Adjustable Shelf Brackets 19

Figure 6: Placing the Shelf Bracket 20

Figure 7: Caged Nut Locations for the Shelf Brackets 21

Figure 8: Shelf Brackets 22

Figure 9: Place the Lock-Down Bar on the Shelf 22

Figure 10: U Numbering Above Shelf 23

Figure 11: Putting the Chassis in the Rack 24

Figure 12: Location for the Lock-down Bar 24

Figure 13: Place the Upper Bracket Offset 25

Figure 14: Attach the Shelf Extension 25

Figure 15: Place Caged Nuts 26

Figure 16: Fork Lift 27

Figure 17: Face Plate Mounting Bolt Locations 28

Figure 18: Upper Brackets and Offset Location 29

Figure 19: Upper Bracket Installation 29

Figure 20: Offset to Upper bracket Bolts 30

Figure 21: Lock-Down Bar 30

Figure 22: Bolt and Washer Order 31

Figure 23: Cable Holder 32

Figure 24: Ejector Latch 33

Figure 25: Ejector Latch 34

Figure 26: Management Module 34

Figure 27: Ejector Latch 35

Figure 28: Spine Board Insertion 35

Figure 29: Cable Holders 36

Figure 30: Ground Connection 38

Figure 31: Multiple Power Inlets - Electric Caution Notification 40

Figure 32: Spine Module 40

Figure 33: Top and Bottom Ports 41

Figure 34: Spine Side Panel Display Status Indications 44

Figure 35: Management Module Status Indications for Normal Operation 44

Figure 36: Connecting the Harness to the Console 45

Figure 37: License Key Entitlement Number 55

648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1

Mellanox Technologies 5

Figure 38: License Key Generation Form 55

Figure 39: FabricIT Status/Licensing 56

Figure 40: Web UI Login Page 58

Figure 41: Reset Button 59

Rev 2.1

Mellanox Technologies

List of Tables

Table 1: Revision History of this Installation Guide 7

Table 2: Reference Documents and Web Sites 8

Table 3: Serial Terminal Program Configuration 46

Table 4: Configuration Wizard Session - IP Configuration by DHCP 46

Table 5: Configuration Wizard Session - IP Zeroconf Configuration 48

Table 6: Configuration Wizard Session - Static IP Configuration 49

648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1

Mellanox Technologies 7

Revision History

Table 1 - Revision History of this Installation Guide

Revision Date Details

2.1 July 2011 Changes to Reset button section

2.0 June 2011 Changes to Installation kit

1.9 Nov. 2010 Changed cable guide graphic

1.8 Oct. 2010 Added air flow direction

1.7 Oct. 2010 Added step screw in bolts from offset to upper bracket.

1.6 Aug. 2010 Added weight lifting warning

Added “Remove all protective plastic film from all sides and top of the chas-

sis.”

Added cable power classes

1.5.3 July 2010 Added last page with barcode

1.5.2 June 2010 Added Note regarding initial configuration of the master and slave MMs

Added note regarding configuring the Box IP (BIP)

1.5.1 June 2010 minor fixes

1.5 June 2010 Changed order of Revision History and About

Added TOC, LOF, LOT and Heading1NoNumber to TOC

Synced this doc installation procedure with the UM

Added Getting and Inserting the License section

1.4 May 2010 Added notes for Hotswap

Added note for EFM upgrade

Turned the file into a book with FP, TOC...

Added getting the license section

1.3 April 2010 Reworked the Installation kit

1.2 April 2010 Power numbers

1.1 Jan. 2010 Formatting

1.0 Dec. 2009 Initial release

Rev 2.1

Mellanox Technologies

About this Manual

This manual provides installation and set-up instructions for the IS5600SX GT series QSFP Chas-

sis InfiniBand Switch Platform.

Intended Audience

This manual is intended for users and system administrators responsible for installing and setting

up the switch platform.

The manual assumes familiarity with the InfiniBand

architecture specification.

Related Documentation

The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the

following:

Table 2 - Reference Documents and Web Sites

Document Conventions

Throughout this manual, the name and the term switch are used to describe the switch, unless

explicitly indicated otherwise.

The following pictures are used throughout this document to indicate information that is important

to the user.

Switch Firmware and Firmware Update Tools See

http://www.mellanox.com/content/

pages.php?pg=management_tools&menu_section=34

Note that the Switch System described in this manual is based on

Mellanox Technologies’ InfiniScale

IV switch device.

FabricIT

Enterprise Fabric Management

(EFM) Software User’s Manual Talk to your Mellanox representative for information regarding

licensing and implementation of the FabricIT Enterprise Fabric

Management Software System.

This symbol makes recommendations to the user.

This symbol indicates information that is helpful to the user.

648 Port InfiniBand QDR Switch Platform User Manual Rev 2.1

Mellanox Technologies 9

This symbol indicates a situation that can potentially cause damage to hardware or

software.

BEWARE! This symbol indicates a situation that can potentially cause personal injury

or damage to hardware or software.

Chassis InstallationRev 2.1

Mellanox Technologies

1 Chassis Installation

Installation and initialization of the chassis is a simple process requiring attention to the normal

mechanical, power, and thermal precautions for rack-mounted equipment. Your Chassis will be

shipped in two packages one with the all of the leafs (leaf package) that are ordered and an extra

management module should it be ordered, and the second package (chassis package) with all of the

rest of the parts.

The chassis requires initial configuration to get the chassis and Fabric management up and running

through remote management.

1.1 Environmental and Safety Recommendations

The following are Mellanox recommendations.

This chassis can be installed in standard 19” racks that have depths between 65cm and

80cm between the vertical supports of the rack.

This unit is intended for installation in a Restricted Access Location. A restricted

access area can be accessed only through the use of a special tool, lock and key, or

other means of security.

Recommended ambient temperature in the System room is 20

± 5

Recommended humidity ranges is 40% ± 15% without condensing.

It is highly recommended that the installation sites be as isolated as possible from all

sources of radio transmissions and electrical interference.

It is highly recommended that the installation site building be equipped with a light-

ning rod.

Other manuals for MIS5600Q-10DNC

Table of contents

Other Mellanox Technologies Switch manuals

Mellanox Technologies

Mellanox Technologies SB7700 User manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies SX6720 Instructions for use

Mellanox Technologies

Mellanox Technologies IS5025D-1BRC User manual

Mellanox Technologies

Mellanox Technologies TX6100 MetroX Series Instructions for use

Mellanox Technologies

Mellanox Technologies IS5025 User manual

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Instruction Manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies SX67X0 Instructions for use

Mellanox Technologies

Mellanox Technologies switchx MSX1035B-1BRR User manual

Mellanox Technologies

Mellanox Technologies InfiniScale III M2401G 20Gb/s InfiniBand User manual

Mellanox Technologies

Mellanox Technologies Spectrum SN2700 User manual

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

Mellanox Technologies

Mellanox Technologies SX1710 User manual

Mellanox Technologies

Mellanox Technologies MIS5022Q-1BFR User manual

Mellanox Technologies

Mellanox Technologies M2401G User manual

Mellanox Technologies

Mellanox Technologies SX6518-NR Manual

Mellanox Technologies

Mellanox Technologies SX6710 Instructions for use

Mellanox Technologies

Mellanox Technologies InfiniScale III User manual

Mellanox Technologies

Mellanox Technologies MSX6512-NR Manual

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Instruction Manual

Mellanox Technologies

Mellanox Technologies SB7700 User manual

Mellanox Technologies

Mellanox Technologies SB7700 User manual

Mellanox Technologies

Mellanox Technologies Switch-IB User manual

Popular Switch manuals by other brands

URC

URC Total Control MFS-8 installation manual

Azbil

Azbil HPQ-D11 user manual

Sony

Sony DFS-900M operating instructions

SICK

SICK i17S operating instructions

F&F

F&F SZR-277 quick start guide

Shure

Shure UAMS/BK user guide

Dräger

Dräger X-zone Switch Off Instructions for use

Asco

Asco Joucomatic 349 Series manual

BT Mini Hub user guide

D-Link

D-Link DES-1226G manual

Johnson Controls

Johnson Controls FS80-C installation instructions

Alcatel-Lucent

Alcatel-Lucent 7450 ESS-12 Configuration guide

MicroNet

MicroNet SP1200A user manual

Ruijie

Ruijie RG-PF2920 Series Hardware installation and reference guide

Altinex

Altinex DA1914SX user guide

MiLAN

MiLAN MIL-SM2401M user guide

RJM

RJM MASTERMIND PBC/6X user manual

Atlas

Atlas MMK-KVM8 owner's manual