Mellanox Technologies PowerEdge M420 User manual
Mellanox Technologies
Third party information brought
to you courtesy of Dell.
InfiniScale® IV 16+16 Port 40Gb/s
InfiniBand Switch for Dell PowerEdge
M1000e-series Blade Enclosures User Manual
PN: M3601Q
Rev 1.1
Rev 1.1 MellanoxTechnologies
2
Copyright 2009. Mellanox Technologies, Inc. All Rights Reserved.
Mellanox Technologies ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and
InfiniPCI are registered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Interconnect is a trade-
mark of Mellanox Technologies, Ltd. Dell and the Dell logo are trademarks of Dell Inc. All other marks and
names mentioned herein may be trademarks of their respective companies.
InfiniScale® IV 16+16 Port 40Gb/s InfiniBand Switch for Dell PowerEdge M1000e-series Blade
Enclosures User Manual
Document Number: 3010
Mellanox Technologies, Inc.
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.Mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
Mellanox Technologies Ltd
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: +972-4-909-7200
Fax: +972-4-959-3245
NOTE:
THIS INFORMATION IS PROVIDED BY MELLANOX FOR INFORMATIONAL PURPOSES ONLY AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
MELLANOX BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSE-
QUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLI-
GENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS HARDWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
3
Contents
Contents 3
About this Manual 4
Revision History 5
Chapter 1 Overview 6
1.1 InfiniBand Connectors 7
1.2 Switch Status Lights 8
Chapter 2 Installation and Basic Operation 9
2.1 Installation Safety Warnings 9
2.2 Mechanical Installation 11
2.2.1 Removing an Old Switch From the Chassis 11
2.2.2 Removing an I/O Module Blank From the Chassis 12
2.2.3 Installing the Mellanox M3601Q Switch Into the Dell Chassis 12
2.3 Power Connections and Initial Power On 12
2.4 InfiniBand Cable Installation 12
Chapter 3 Cluster Management and Firmware 14
3.1 Network Management and Clustering Software 14
3.2 Updating Firmware 14
3.2.1 Instructions for Reprogramming Over the InfiniBand Network 14
Chapter 4 Troubleshooting 17
Appendix A Specification 18
Appendix B EMC Certification Statements 19
Appendix C QSFP Interface 21
Appendix D Avertissements de sécurité d’installation 23
Appendix E Installation - Sicherheitshinweise 25
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
4
About this Manual
This manual describes the installation and basic use of the Mellanox M3601Q 16+16 Port 40Gb/s
InfiniBand Switch for Dell PowerEdge M1000e-series Blade Enclosures.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platforms listed above.
The manual assumes familiarity with the InfiniBand® Architecture Specification.
Related Documentation
The documentation set accompanying the Mellanox M3601Q includes the following:
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Dell Support Web pages: http://support.dell.com
Conventions
Throughout this manual, the name M3601Q and the terms switch, I/O module, and IOM are used to
describe the 16+16 port 40Gb/s InfiniBand switch for the Dell M1000e, unless explicitly indicated
otherwise.
Table 1 - Reference Documents
InfiniBand Architecture Specification Vol-
ume 1 Release 1.2.1 and Volume 2 release
1.2.1
InfiniBand architecture specification descriptions
Mellanox Firmware Tools (MFT)
User’s Manual
Document # 2329
The MFT (Mellanox Firmware Tools) package is a set of firmware tools. The manual
supplied with this package provides an overview of the firmware its installation and
replacement. The MFT can be downloaded with its documentation at:
http://www.mellanox.com > Downloads > Firmware Tools.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
5
Revision History
Table 2 - Revision History Table
Date Revision Description
March 2009 1.1 Added Dell Logo
November 2008 1.0 Initial Release
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
6
1 Overview
The Mellanox M3601Q 40Gb/s InfiniBand Switch Blade for Dell PowerEdge M1000e-series Blade
Enclosures provides a high bandwidth, low latency fabric for Enterprise Data Centers, High-Perfor-
mance Computing and Embedded environments. Based on the fourth generation InfiniScale® IV
InfiniBand switch device, this I/O module (IOM) delivers up to 40Gb/s full bisectional bandwidth
per port. When used in conjunction with ConnectX® InfiniBand dual port Mezzanine I/O cards,
clustered data bases, parallelized applications and transactional services applications will achieve
significant performance improvements resulting in reduced completion time and lower cost per
operation.
The I/O module comes pre-installed with all necessary firmware, and configured for standard oper-
ation within an InfiniBand fabric, and requires an InfiniBand compliant Subnet Manager running
from one of the hosts. All that is required for normal operation is to follow the usual precautions for
installation and connection from the switch to the HCAs or other switches. Once connected, the
Subnet Management software automatically configures and begins utilizing the switch.
It is recommended that Mellanox OpenFabrics software package be installed on all nodes con-
nected to the M3610Q. The software package provides a subnet manager and network management
tools as well as connectivity software for servers and storage, and is available on the Mellanox web
site. See Chapter 3 for more information.
Basic installation, hot-swapping components and hardware maintenance is covered in “Installation
and Basic Operation” on page 9.
The M3601Q switch has a Hot Swap controller and a PSOC Management IC.
Rev 1.1 MellanoxTechnologies
7
Figure 1: Switch Front Panel and Locking Mechanism
1.1 InfiniBand Connectors
InfiniBand connectivity has 16 QSFP connectors through the front panel. The remaining 16 inter-
faces are through the AirMax Midplane Connector out the back of the switch. Figure 1 shows the
front 16 ports.
Each of the InfiniBand ports has two LEDs located next to the connector. The green LED, when lit,
indicates that a valid physical connection to the other system (switch or HCA port) exists. The yel-
low LED when lit, indicates that the Subnet Manager is running and a valid data link exists. The
yellow LED illuminates when the InfiniBand network is discovered over the physical link. A valid
data activity link without data transfer is designated by a constant yellow LED indication. A valid
data activity link with data transfer is designated by a blinking yellow LED indication. If the LEDs
are not active, either the physical link or the logical link (or both) connections have not been estab-
lished.
Locking Mechanism
Switch Status Lights
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
8
Figure 2: Physical and Logical Link Indication LEDs
1.2 Switch Status Lights
The switch Status lights indicate whether the switch is receiving power from the chassis, and the
state of the switch.
Figure 3: Indicator LEDs
Table 3 - IOM states and LED configurations:
LED Switch Status
Indication Green Status
OFF OFF OFF
ON OFF Boot in Progress
Switch not ready
Blinking Blue
ON BLINKING BLUE The CMC is identifying
the newly installed switch
ON ON BLUE Switch is on and operating Normally
ON ON or BLINKING
AMBER Fault in System
Self-diagnosed
OFF ON or BLINKING
AMBER Fault in System
CMC-detected
LED Name Connection Status
Physical Link - Green Off – No Physical Link
ON – Physical Link
Data Activity - Yellow Blinking – indicates Data Transfer
Constant on – indicates Link exists
with no Data Transfer taking place
Off with green LED lit – indicates that
the Subnet Manager may not be running
The IO Module is on and
ready when both the blue and
green LEDs are lit.
This LED can be blue or
amber. A fault is indicated
when the amber LED is blink-
ing. See Table 3.
Rev 1.1 MellanoxTechnologies
9
2Installation and Basic Operation
Installation and initialization of the I/O module are straightforward processes, requiring attention to
the normal mechanical, power, and thermal precautions for rack-mounted equipment. The I/O mod-
ule does not require any programming or configuration to operate as a basic InfiniBand switch and
includes all of the necessary functionality to operate with external standard InfiniBand Subnet
Management software.
This section describes the installation process and basic operation of the I/O module. Please first
read the warnings sub-section carefully before carrying on with installation procedures.
2.1 Installation Safety Warnings
1. Installation Instructions
2. Over-temperature
3. Stacking the Chassis
4. During Lightning - Electrical Hazard
Suitable electrical, mechanical and fire enclosure shall be provided by the end user.
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 40°C (104°F). Moreover, to guarantee proper air flow,
allow at least 8cm (3 inches) of clearance around the ventilation openings.
The chassis should not be stacked on any other equipment. If the chassis falls, it can
cause bodily injury and equipment damage.
During periods of lightning activity, do not work on the equipment or connect or discon-
nect cables.
M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual
MellanoxTechnologies Rev 1.1
10
5. Copper InfiniBand Cable Connecting/Disconnecting
6. Rack Mounting and Servicing
7. Equipment Installation
8. Equipment Disposal
9. Local and National Electrical Codes
Copper InfiniBand cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for special
warnings and instructions.
When this product is mounted or serviced in a rack, special precautions must be taken to
ensure that the system remains stable. In general you should fill the rack with equipment
starting from the bottom to the top.
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regulations.
This equipment should be installed in compliance with local and national electrical
codes.
This manual suits for next models
3
Table of contents
Other Mellanox Technologies Network Router manuals
Mellanox Technologies
Mellanox Technologies SwitchX-2 MSX1024B-1BFS User manual
Mellanox Technologies
Mellanox Technologies SX1012 User manual
Mellanox Technologies
Mellanox Technologies SX1018HP Installation guide
Mellanox Technologies
Mellanox Technologies MSX6512-4R Instructions for use
Mellanox Technologies
Mellanox Technologies SwitchX MSX1016X-2BFR Instructions for use
Mellanox Technologies
Mellanox Technologies SwitchX-2 SX1012 User manual
