Silicon Graphics ICE X User manual

SGI®ICE™X System 

Hardware User Guide

Document Number 007-5806-004

No permission is granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part,

without the prior written permission of SGI.

LIMITED RIGHTS LEGEND

The software described in this document is "commercial computer software" provided with restricted rights (except as to included open/free source) as specified

in the FAR 52.227-19 and/or the DFAR 227.7202, or successive sections. Use beyond license provisions is a violation of worldwide intellectual property laws,

treaties and conventions. This document is provided with limited rights as defined in 52.227-14.

The electronic (software) version of this document was developed at private expense; if acquired under an agreement with the USA government or any

contractor thereto, it is acquired as “commercial computer software” subject to the provisions of its applicable license agreement, as specified in (a) 48 CFR

12.212 of the FAR; or, if acquired for Department of Defense units, (b) 48 CFR 227-7202 of the DoD FAR Supplement; or sections succeeding thereto.

Contractor/manufacturer is SGI, 900 North McCarthy Blvd. Milpitas, CA 95035.

TRADEMARKS AND ATTRIBUTIONS

SGI, and the SGI logo are registered trademarks and Rackable, SGI Lustre and SGI ICE are trademarks of, Silicon Graphics International, in the United States

and/or other countries worldwide.

Intel, Intel QuickPath Interconnect (QPI), Itanium and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States

and other countries. 

UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd. 

Infiniband is a trademark of the InfiniBand Trade Association. 

Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries.

Linux is a registered trademark of Linus Torvalds.

All other trademarks mentioned herein are the property of their respective owners.

007-5806-004 iii

Record of Revision

Version Description

-001 March, 2012

First release

-002 February, 2013

Blade and rack design updates

-003 June, 2014

Blade updates

-004 November, 2015

cpower command and service reference updates

007-5806-004 v

Contents

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . xi

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . xiii

Audience. . . . . . . . . . . . . . . . . . . . . . . . . . xv

Important Information . . . . . . . . . . . . . . . . . . . . . . xv

Chapter Descriptions . . . . . . . . . . . . . . . . . . . . . . xvi

Related Publications . . . . . . . . . . . . . . . . . . . . . . .xvii

Obtaining SGI Publications. . . . . . . . . . . . . . . . . . . xviii

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . xix

Product Support . . . . . . . . . . . . . . . . . . . . . . . . xix

Reader Comments . . . . . . . . . . . . . . . . . . . . . . . xx

1. Operation Procedures . . . . . . . . . . . . . . . . . . . . . . 1

Precautions . . . . . . . . . . . . . . . . . . . . . . . . . 1

ESD Precaution . . . . . . . . . . . . . . . . . . . . . . . 1

Safety Precautions . . . . . . . . . . . . . . . . . . . . . . 2

Console Connections . . . . . . . . . . . . . . . . . . . . . . 3

Powering the System On and Off . . . . . . . . . . . . . . . . . . . 4

Preparing to Power On . . . . . . . . . . . . . . . . . . . . . 5

Powering On and Off . . . . . . . . . . . . . . . . . . . . . 8

Console Management Power (cpower) Commands . . . . . . . . . . . . 8

Monitoring Your Server . . . . . . . . . . . . . . . . . . . . . . 12

Optional SGI Remote Services (SGI RS) . . . . . . . . . . . . . . . . . 13

SGI Remote Services Primary Capabilities . . . . . . . . . . . . . . . 14

SGI Remote Services Benefits . . . . . . . . . . . . . . . . . . . 14

SGI Remote Service Operations Overview. . . . . . . . . . . . . . . . 14

vi 007-5806-004

Contents

2. System Management . . . . . . . . . . . . . . . . . . . . . . 17

Using the 1U Console Option . . . . . . . . . . . . . . . . . . . . 19

Levels of System and Chassis Control . . . . . . . . . . . . . . . . . . 19

Chassis Controller Interaction . . . . . . . . . . . . . . . . . . . . 19

Chassis Manager Interconnects . . . . . . . . . . . . . . . . . . . 20

M-rack Chassis Manager Interconnection . . . . . . . . . . . . . . . 21

Chassis Management Control (CMC) Functions . . . . . . . . . . . . . . 22

CMC Connector Ports and Indicators . . . . . . . . . . . . . . . . . 23

System Power Status . . . . . . . . . . . . . . . . . . . . . . . 23

3. System Overview . . . . . . . . . . . . . . . . . . . . . . . 25

System Models . . . . . . . . . . . . . . . . . . . . . . . . 26

SGI ICE X System and Blade Architectures . . . . . . . . . . . . . . . . . 29

IP113 Blade Architecture Overview . . . . . . . . . . . . . . . . . 29

IP115 Blade Architecture Overview . . . . . . . . . . . . . . . . . 30

IP119 Blade Architecture Overview . . . . . . . . . . . . . . . . . 31

IP131 Blade Architecture Overview . . . . . . . . . . . . . . . . . 32

IP133 Blade Architecture Overview . . . . . . . . . . . . . . . . . 33

QuickPath Interconnect Features. . . . . . . . . . . . . . . . . . . . 34

IP113, IP115 and IP119 QPI Bandwidth . . . . . . . . . . . . . . . . 34

IP131 and IP133 QPI Bandwidth . . . . . . . . . . . . . . . . . . 34

Blade Memory Features . . . . . . . . . . . . . . . . . . . . . 35

Blade DIMM Memory Features . . . . . . . . . . . . . . . . . 35

Memory Channel Recommendation . . . . . . . . . . . . . . . . 35

Blade DIMM Bandwidth Factors . . . . . . . . . . . . . . . . . 35

System InfiniBand Switch Blades . . . . . . . . . . . . . . . . . . 36

Enclosure Switch Density Choices . . . . . . . . . . . . . . . . . 36

Contents

007-5806-004 vii

System Features and Major Components . . . . . . . . . . . . . . . . . 38

Modularity and Scalability . . . . . . . . . . . . . . . . . . . . 38

System Administration Server . . . . . . . . . . . . . . . . . . 39

Rack Leader Controller . . . . . . . . . . . . . . . . . . . 40

Multiple Chassis Manager Connections . . . . . . . . . . . . . . . 40

The RLC as Fabric Manager . . . . . . . . . . . . . . . . . . 41

Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 42

Batch Server Node . . . . . . . . . . . . . . . . . . . . . 43

I/O Gateway Node . . . . . . . . . . . . . . . . . . . . . 43

Optional Lustre Nodes Overview . . . . . . . . . . . . . . . . . . 44

MDS Node . . . . . . . . . . . . . . . . . . . . . . . 44

OSS Node . . . . . . . . . . . . . . . . . . . . . . . 45

Reliability, Availability, and Serviceability (RAS) . . . . . . . . . . . . . 45

System Components . . . . . . . . . . . . . . . . . . . . . . . 47

D-Rack Unit Numbering . . . . . . . . . . . . . . . . . . . . 51

Rack Numbering . . . . . . . . . . . . . . . . . . . . . . 51

Optional System Components . . . . . . . . . . . . . . . . . . . 51

Optional SGI Remote Services (SGI RS) . . . . . . . . . . . . . . . . . 51

SGI Remote Services Primary Capabilities . . . . . . . . . . . . . . . 52

SGI Remote Services Benefits . . . . . . . . . . . . . . . . . . . 52

SGI Remote Service Operations Overview. . . . . . . . . . . . . . . . 53

SGI Warranty Levels . . . . . . . . . . . . . . . . . . . . . 54

4. Rack Information . . . . . . . . . . . . . . . . . . . . . . . 55

Overview . . . . . . . . . . . . . . . . . . . . . . . . . 55

SGI ICE X Series D-Rack (42U) . . . . . . . . . . . . . . . . . . . 56

ICE X D-Rack Technical Specifications . . . . . . . . . . . . . . . . . 61

SGI ICE X M-Cell Rack Assemblies . . . . . . . . . . . . . . . . . . 62

M-Cell Functional Overview . . . . . . . . . . . . . . . . . . . 63

viii 007-5806-004

Contents

5. SGI ICE X Administration/Leader Servers . . . . . . . . . . . . . . . . 67

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 68

System Hierarchy . . . . . . . . . . . . . . . . . . . . . . 68

Communication Hierarchy . . . . . . . . . . . . . . . . . . . 69

1U Rack Leader Controller and Administration Server . . . . . . . . . . . . . . 71

1U Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 72

C1104G-RP5 1U Service Node . . . . . . . . . . . . . . . . . . . 72

2U Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 74

RP2 2U Service Nodes . . . . . . . . . . . . . . . . . . . . . 74

RP2 Service Node Front Controls . . . . . . . . . . . . . . . . . 74

RP2 Service Node Back Panel Components . . . . . . . . . . . . . . 76

C2110G-RP5-P 2U Service Node . . . . . . . . . . . . . . . . . . 77

SGI UV 20 2U Service Node . . . . . . . . . . . . . . . . . . . 79

6. Basic Troubleshooting . . . . . . . . . . . . . . . . . . . . . . 81

Troubleshooting Chart . . . . . . . . . . . . . . . . . . . . . . 82

LED Status Indicators . . . . . . . . . . . . . . . . . . . . . . 83

Blade Enclosure Pair Power Supply LEDs . . . . . . . . . . . . . . . . 83

IP113 Compute Blade LEDs . . . . . . . . . . . . . . . . . . . 84

IP115 Compute Blade Status LEDs . . . . . . . . . . . . . . . . . . 85

IP119 Compute Blade Status LEDs . . . . . . . . . . . . . . . . . . 87

IP131 Compute Blade Status LEDs . . . . . . . . . . . . . . . . . . 88

IP133 Compute Blade Status LEDs . . . . . . . . . . . . . . . . . . 89

Accessing Online Support Information and Services . . . . . . . . . . . . . . 90

SGI Customer Portal. . . . . . . . . . . . . . . . . . . . . . 90

Technical Assistance . . . . . . . . . . . . . . . . . . . . . 91

Other Resources . . . . . . . . . . . . . . . . . . . . . . . 91

SGI Warranty Levels . . . . . . . . . . . . . . . . . . . . . 91

Optional SGI Remote Services (SGI RS) . . . . . . . . . . . . . . . . 92

7. Maintenance Procedures . . . . . . . . . . . . . . . . . . . . . 93

Maintenance Precautions and Procedures . . . . . . . . . . . . . . . . . 93

Preparing the System for Maintenance or Upgrade . . . . . . . . . . . . . . 94

Installing or Removing Internal Parts . . . . . . . . . . . . . . . . . . 94

Contents

007-5806-004 ix

Replacing ICE X System Components . . . . . . . . . . . . . . . . . . 95

Removing and Replacing a Blade Enclosure Power Supply . . . . . . . . . . . 95

Removing and Replacing Rear Fans (Blowers) . . . . . . . . . . . . . . 98

Removing or Replacing a Fan Enclosure Power Supply . . . . . . . . . . . .102

Removing a Fan Assembly Power Supply . . . . . . . . . . . . . . . .102

Replacing a Fan Power Supply . . . . . . . . . . . . . . . . . . .102

Overview of PCI Express Operation . . . . . . . . . . . . . . . . . .105

A. Technical Specifications and Pinouts . . . . . . . . . . . . . . . . . .107

System-level Specifications . . . . . . . . . . . . . . . . . . . . .107

D-Rack Physical and Power Specifications . . . . . . . . . . . . . . .108

D-Rack System Environmental Specifications . . . . . . . . . . . . . . .109

ICE X M-Rack Technical Specifications . . . . . . . . . . . . . . . .110

Ethernet Port Specification . . . . . . . . . . . . . . . . . . . .112

B. Safety Information and Regulatory Specifications . . . . . . . . . . . . . .113

Safety Information . . . . . . . . . . . . . . . . . . . . . . .113

Regulatory Specifications . . . . . . . . . . . . . . . . . . . . .115

CMN Number . . . . . . . . . . . . . . . . . . . . . . .115

CE Notice and Manufacturer’s Declaration of Conformity . . . . . . . . . . .115

Electromagnetic Emissions . . . . . . . . . . . . . . . . . . . .115

FCC Notice (USA Only) . . . . . . . . . . . . . . . . . . .116

Industry Canada Notice (Canada Only) . . . . . . . . . . . . . . .116

VCCI Notice (Japan Only). . . . . . . . . . . . . . . . . . .117

Chinese Class A Regulatory Notice . . . . . . . . . . . . . . . .117

Korean Class A Regulatory Notice . . . . . . . . . . . . . . . .117

Shielded Cables . . . . . . . . . . . . . . . . . . . . . . .118

Electrostatic Discharge and Laser Compliance. . . . . . . . . . . . . . .118

Lithium Battery Statements. . . . . . . . . . . . . . . . . . . .119

Index . . . . . . . . . . . . . . . . . . . . . . . . . .121

Table of contents

Other Silicon Graphics Server manuals

Silicon Graphics

Silicon Graphics InfiniteStorage 3500 User manual

Silicon Graphics

Silicon Graphics 1450 Operating manual

Silicon Graphics

Silicon Graphics 1450 User manual

Silicon Graphics

Silicon Graphics 1450 User manual

Silicon Graphics

Silicon Graphics POWER CHALLENGE User manual

Silicon Graphics

Silicon Graphics UV 300H User manual

Silicon Graphics

Silicon Graphics UV 30 User manual

Silicon Graphics

Silicon Graphics Rackable C2112-4KL1 User manual

Silicon Graphics

Silicon Graphics UV 2000 User manual

Silicon Graphics

Silicon Graphics InfiniteStorage 3104 User manual

Silicon Graphics

Silicon Graphics 2400 User manual

Silicon Graphics

Silicon Graphics Altix XE210 User manual

Silicon Graphics

Silicon Graphics Origin 350 User manual

Silicon Graphics

Silicon Graphics Origin 3900 User manual

Silicon Graphics

Silicon Graphics Altix XE340 User manual

Silicon Graphics

Silicon Graphics 1200 User manual

Silicon Graphics

Silicon Graphics 1400 User manual

Silicon Graphics

Silicon Graphics Rackable C2112-4G3 User manual

Silicon Graphics

Silicon Graphics Origin200 User manual

Silicon Graphics

Silicon Graphics Rackable C1104G-RP5 User manual

Silicon Graphics

Silicon Graphics Rackable C1104-GP2 User manual

Silicon Graphics

Silicon Graphics SGI 1200 Series User manual

Silicon Graphics

Silicon Graphics Altix XE320 User manual

Silicon Graphics

Silicon Graphics Rackable C2005 2U User manual

Popular Server manuals by other brands

HP AlphaServer ES47 Cli reference manual

Dell

Dell PowerEdge R410 Getting started

PowerNAS

PowerNAS CMA quick start guide

Supero

Supero FatTwin F617R3-FT user manual

MSI

MSI P1-109N-L70 user manual

Dell

Dell PowerEdge XE9640 Installation and service manual

Dell

Dell PS4100 Hardware

EtroVISION

EtroVISION EV3151 user manual

NEC

NEC Express5800/T110j System configuration guide

HP ProLiant Gen9 user guide

Compaq

Compaq ProLiant 3000 Maintenance and service guide

Dell

Dell PowerEdge 3YPMN Getting started guide

iRobo

iRobo IPC2U user manual

Nortel

Nortel 1000 Con?guration guide user manual

Asus

Asus AP7500 Hardware reference guide

Avid Technology

Avid Technology AirSpeed 5000 manual

HP Integrity rx2600 overview

Milestone

Milestone Husky IVO 350T Getting started