IDT 89HPES16NT2 User manual

April 2008

6024 Silver Creek Valley Road, San Jose, California 95138

Telephone: (800) 345-7015 • (408) 284-8200 • FAX: (408) 284-2775

Printed in U.S.A.

IDT™89HPES16NT2

PCI Express® Switch

User Manual

GENERAL DISCLAIMER

Integrated Device Technology, Inc. reserves the right to make changes to its products or specifications at any time, without notice, in order to improve design or performance

and to supply the best possible product. IDT does not assume any responsibility for use of any circuitry described other than the circuitryembodied in an IDT product. The

Company makes no representations that circuitry described herein is free from patent infringement or other rights of third parties which may result from its use. No license is

granted by implication or otherwise under any patent, patent rights or other rights, of Integrated Device Technology, Inc.

CODE DISCLAIMER

Code examples providedby IDT arefor illustrative purposesonly and should notbe relieduponfor developing applications.Anyuseof the codeexamples below iscompletely

at your own risk. IDT MAKES NOREPRESENTATIONSOR WARRANTIES OFANY KIND CONCERNING THE NONINFRINGEMENT,QUALITY, SAFETY OR SUITABILITY

OF THE CODE, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICU-

LAR PURPOSE, ORNON-INFRINGEMENT. FURTHER,IDT MAKES NO REPRESENTATIONS ORWARRANTIES AS TOTHE TRUTH,ACCURACYOR COMPLETENESS

OF ANY STATEMENTS, INFORMATION OR MATERIALS CONCERNING CODE EXAMPLES CONTAINED IN ANY IDT PUBLICATION OR PUBLIC DISCLOSURE OR

THAT IS CONTAINED ON ANY IDT INTERNET SITE. IN NO EVENT WILL IDT BELIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, INDIRECT, PUNITIVE OR

SPECIAL DAMAGES, HOWEVER THEY MAY ARISE, AND EVEN IF IDT HAS BEEN PREVIOUSLY ADVISED ABOUT THE POSSIBILITY OF SUCH DAMAGES. The code

examples also may be subject to UnitedStates export control lawsand maybe subject to theexportorimportlawsof other countries andit is your responsibility to comply with

any applicable laws or regulations.

LIFE SUPPORT POLICY

Integrated Device Technology's products are not authorized for use as critical components in lifesupport devices or systemsunless a specific written agreement pertaining to

such intended use is executed between the manufacturer and an officer of IDT.

1. Life support devices or systems are devices or systems which (a) are intended for surgical implant into the body or (b) support or sustain life and whose failure to perform,

when properly used in accordance with instructions for use provided in the labeling, can be reasonably expected to result in a significant injury to the user.

2. A critical component is anycomponents of a life supportdevice or system whosefailure to performcanbe reasonably expected tocause the failure of thelife support device

or system, or to affect its safety or effectiveness.

IDT, the IDT logo, and Integrated Device Technology are trademarks or registered trademarks of Integrated Device Technology, Inc.

Notes

PES16NT2 User Manual 1 April 15, 2008

About This Manual

Introduction

This user manual includes hardware and software information on the 89HPES16NT2, a member of

IDT’s PRECISE™ family of PCI Express® switching solutions offering the next-generation I/O interconnect

standard.

Finding Additional Information

Informationnot includedin thismanual suchas mechanicals,package pin-outs,and electricalcharacter-

istics can be found in the data sheet for this device, which is available from the IDT website (www.idt.com)

as well as through your local IDT sales representative.

Content Summary

Chapter 1, “PES16NT2 Device Overview,” provides a complete introduction to the performance capa-

bilities of the 89HPES16NT2. Included in this chapter is a summary of features for the device as well as a

system block diagram and pin description.

Chapter 2, “Clocking, Reset, and Initialization,” provides a description of the two differential refer-

ence clock inputs that are used internally to generate all of the clocks required by the internal switch logic

and the SerDes.

Chapter 3, “Link Operation,” describes the operation of the link feature including polarity inversion,

link width negotiation, and lane reversal.

Chapter 4, “Switch Operation,” discusses the procedure for forwarding PCIe® TLPs between switch

ports.

Chapter 5, “Power Management,” describes thepower managementcapability structurelocated inthe

configuration space of each PCI-PCI bridge in the PES16NT2.

Chapter 6, “SMBus Interfaces,” describes the operation of the 2 SMBus interfaces on the PES16NT2.

Chapter 7, “NTB Upstream Port Failover,” describes the NTB upstream port failover mechanism that

enables the construction of fault tolerant systems.

Chapter 8, “General Purpose I/O,” describes how the eight General Purpose I/O (GPIO) pins may be

individually configured as general purpose inputs, general purpose outputs, or alternate functions

Chapter 9, “Transparent Mode Operation,” describes how the PES16NT2 can be configured during a

fundamental reset to operate in transparent mode or transparent mode with serial EEPROM initialization.

Chapter 10, “Non-Transparent Mode Operation,” describes how the PES16NT2 can be configured

during a fundamental reset to operate in non-transparent mode or non-transparent mode with serial

EEPROM initialization.

Chapter 11, “JTAG Boundary Scan,” discusses an enhanced JTAG interface, including a system logic

TAP controller, signal definitions, a test dataregister, an instruction register, and usage considerations.

Signal Nomenclature

To avoid confusion when dealing with a mixture of “active-low” and “active-high” signals, the terms

assertion and negation are used. The term assert or assertion is used to indicate that a signal is active or

true, independent of whether thatlevel is representedby a high or low voltage. The term negate or negation

is used to indicate that a signal is inactive or false.

IDT

PES16NT2 User Manual 2 April 15, 2008

Notes To define the active polarity of a signal, a suffix will be used. Signals ending with an ‘N’ should be inter-

preted as being active, or asserted, when at a logic zero (low) level. All other signals (including clocks,

buses and select lines) will be interpreted as being active, or asserted when at a logic one (high) level. To

define buses, the most significant bit (MSB) will be on the left and least significant bit (LSB) will be on the

right. No leading zeros will be included.

Throughout this manual, when describing signal transitions, the following terminology is used. Rising

edgeindicates alow-to-high (0to 1)transition. Falling edge indicates a high-to-low (1 to0) transition.These

terms are illustrated in Figure 1.

Figure 1 Signal Transitions

Numeric Representations

To represent numerical values, either decimal, binary, or hexadecimal formats will be used. The binary

format is as follows: 0bDDD, where “D” represents either 0 or 1; the hexadecimal format is as follows:

0xDD, where “D” represents the hexadecimal digit(s); otherwise, it is decimal.

The compressed notation ABC[x|y|z]D refers to ABCxD, ABCyD, and ABCzD.

The compressed notation ABC[x..y]D refers to ABCxD, ABC(x+1)D, ABC(x+2)D,... ABCyD.

Data Units

The following data unit terminology is used in this document.

In quadwords, bit 63 is always the most significant bit and bit 0 is the least significant bit. In double-

words, bit 31 is always the most significant bit and bit 0 is the least significant bit. In words, bit 15is always

the most significant bit and bit 0 is the least significant bit. In bytes, bit 7 is always the most significant bit

and bit 0 is the least significant bit.

The ordering of bytes within words is referred to as either “big endian” or “little endian.” Big endian

systems label byte zero as the most significant (leftmost) byte of a word. Little endian systems label byte

zero as the least significant (rightmost) byte of a word. See Figure 2.

Term Words Bytes Bits

Byte 1/2 1 8

Word 1 2 16

Doubleword (Dword) 2 4 32

Quadword (Qword) 4 8 64

Table 1 Data Unit Terminology

1234

high-to-low

transition low-to-high

transition

single clock cycle

IDT

PES16NT2 User Manual 3 April 15, 2008

Notes

Figure 2 Example of Byte Ordering for “Big Endian” or “Little Endian” System Definition

Software in the context of this register terminology refers to modifications made by PCIe root configura-

tion writes, writes to registers made through the slave SMBus interface, or serial EEPROM register initial-

ization. See Table 2.

Type Abbreviation Description

Hardware Initialized HWINIT Register bits are initialized by firmware or hardware mechanisms

such as pin strapping or serial EEPROM. (System firmware hard-

ware initialization is only allowed for system integrated devices.)

Bits are read-only after initialization and can only be reset (for

write-once by firmware) with reset.

Read Only and Clear RC Software can read the register/bits with this attribute. Reading the

value will automatically cause the register/bit to be reset to zero.

Writing to a RC location has no effect.

Read Clear and Write RCW Software can read the register/bits with this attribute. Reading the

value will automatically cause the register/bits to be reset to zero.

Writes cause the register/bits to be modified.

Reserved Reserved The value read from a reserved register/bit is undefined. Thus,

software must deal correctly with fields that are reserved. On

reads,software mustuseappropriate maskstoextractthedefined

bits and not rely on reserved bits being any particular value. On

writes, software must ensure that the values of reserved bit posi-

tions are preserved. That is, the values of reserved bit positions

must first be read, merged with the new values for other bit posi-

tions and then written back.

Read Only RO Software can only read registers/bits with this attribute. Contents

are hardwired. Writing to a RO location has no effect.

Read Only and set by

Hardware ROS Software can only read registers/bits with this attribute. Contents

are set by hardware and may change. Writing to a ROS location

has no effect.

Read and Write RW Software can both read and write bits with this attribute.

Table 2 Register Terminology (Sheet 1 of 2)

0123

bit 0bit 31

Address of Bytes within Words: Big Endian

3210

bit 0bit 31

Address of Bytes within Words: Little Endian

IDT

PES16NT2 User Manual 4 April 15, 2008

Notes

Use of Hypertext

In Chapters 9 and 10 there are tables which contain register names and page numbers highlighted in

blue under the Register Definition column. In pdf files, users can jump from the source table directly to the

registers by clicking on the register name in the source table. Each register name in the table is linked

directly to the appropriate register in the register section of the chapter. To return to the source table after

having jumped to the register section, click on the same register name (in blue) in the register section.

Revision History

April 15, 2008: Initial publication of user manual.

Read and Write Clear RW1C Software can read and write to registers/bits with this attribute.

However, writing a value of zero to a bit with this attribute has no

effect. A RW1C bit can only be set to a value of 1 by a hardware

event. To clear a RW1C bit (i.e., change its value to zero) a value

of one must be written to the location. An RW1C bit is never

cleared by hardware.

Read and Write when

Unlocked RWL Software can read the register/bits with this attribute. Writing to

fied if the REGUNLOCK bit in the SWCNTL register is set. When

the REGUNLOCK bit is cleared, writes are ignored and the regis-

ter/bits are effectively read-only

Zero Zero A zero register or bit must be written with a value of zero and

returns a value of zero when read.

Type Abbreviation Description

Table 2 Register Terminology (Sheet 2 of 2)

Notes

PES16NT2 User Manual i April 15, 2008

Table of Contents

About This Manual

Introduction ....................................................................................................................................1

Content Summary ..........................................................................................................................1

Signal Nomenclature .....................................................................................................................1

Numeric Representations ..............................................................................................................2

Data Units ......................................................................................................................................2

Register Terminology .....................................................................................................................3

Use of Hypertext ............................................................................................................................4

Revision History .............................................................................................................................4

PES16NT2 Device Overview

Introduction.....................................................................................................................................1-1

List of Features...............................................................................................................................1-3

System Identification.......................................................................................................................1-4

Vendor ID................................................................................................................................1-4

Device ID................................................................................................................................1-4

Revision ID.............................................................................................................................1-5

JTAG ID..................................................................................................................................1-5

Logic Diagram.................................................................................................................................1-6

Pin Description................................................................................................................................1-7

Pin Characteristics........................................................................................................................1-10

Clocking, Reset, and Initialization

Introduction.....................................................................................................................................2-1

Initialization.....................................................................................................................................2-3

Reset...............................................................................................................................................2-5

Fundamental Reset................................................................................................................2-7

Hot Reset................................................................................................................................2-8

Non-Transparent Mode Reset................................................................................................2-9

Link Operation

Introduction.....................................................................................................................................3-1

Polarity Inversion............................................................................................................................3-1

Link Width Negotiation....................................................................................................................3-1

Lane Reversal.................................................................................................................................3-1

Link Retraining................................................................................................................................3-4

Link Down.......................................................................................................................................3-4

Slot Power Limit Support................................................................................................................3-4

Crosslink.........................................................................................................................................3-5

Link Status......................................................................................................................................3-5

IDT Table of Contents

PES16NT2 User Manual ii April 15, 2008

Notes Switch Operation

Introduction.....................................................................................................................................4-1

Routing ...........................................................................................................................................4-3

Data Integrity ..................................................................................................................................4-4

Switch Time-Outs ...........................................................................................................................4-5

Interrupts.........................................................................................................................................4-5

Switch Core Errors..........................................................................................................................4-6

Power Management

Introduction.....................................................................................................................................5-1

PME Messages ......................................................................................................................5-2

Link States......................................................................................................................................5-2

Active State Power Management ...................................................................................................5-3

SMBus Interfaces

Introduction.....................................................................................................................................6-1

SMBus Registers............................................................................................................................6-2

Master SMBus Interface.................................................................................................................6-4

Initialization.............................................................................................................................6-4

Serial EEPROM......................................................................................................................6-4

Slave SMBus Interface...................................................................................................................6-8

Initialization.............................................................................................................................6-9

SMBus Transactions ..............................................................................................................6-9

NTB Upstream Port Failover

Introduction.....................................................................................................................................7-1

Failover...........................................................................................................................................7-2

Static Upstream Port Failover.................................................................................................7-3

Dynamic Upstream Port Failover............................................................................................7-3

General Purpose I/O

Introduction.....................................................................................................................................8-1

GPIO Registers...............................................................................................................................8-1

GPIO Configuration ........................................................................................................................8-2

GPIO Pin Configured as an Input...........................................................................................8-2

GPIO Pin Configured as an Output........................................................................................8-2

GPIO Pin Configured as an Alternate Function......................................................................8-2

Transparent Mode Operation

Introduction.....................................................................................................................................9-1

End-to-End CRC.............................................................................................................................9-2

Interrupts.........................................................................................................................................9-2

Error Detection and Handling .........................................................................................................9-2

Configuration Requests..........................................................................................................9-5

Port Configuration Space Organization..........................................................................................9-5

Upstream Port A Configuration Space Registers ...........................................................................9-7

PCI Express Capability Structure.........................................................................................9-19

Power Management Capability Structure.............................................................................9-26

Switch Control and Status Registers....................................................................................9-30

Extended Configuration Space Access and INTx Status Registers.....................................9-36

PCI Express Virtual Channel Capability...............................................................................9-37

IDT Table of Contents

PES16NT2 User Manual iii April 15, 2008

Notes Physical Layer Control and Status Registers.......................................................................9-42

Non-Transparent Mode Operation

Introduction...................................................................................................................................10-1

Transaction Routing......................................................................................................................10-3

Address Routing...................................................................................................................10-3

ID Routing.............................................................................................................................10-6

Route to Root Implicit Routing..............................................................................................10-6

Broadcast from Root Implicit Routing...................................................................................10-6

Local Terminate at Receiver Implicit Routing.......................................................................10-6

Gather and Route to Root Implicit Routing...........................................................................10-7

Non-Transparent Bridge Interprocessor Communications............................................................10-7

Message Registers...............................................................................................................10-7

Doorbell Registers................................................................................................................10-8

Scratchpad Registers...........................................................................................................10-8

Interrupts.......................................................................................................................................10-8

MSI and INTx Message Generation.....................................................................................10-8

Non-Transparent Bridge TLP Processing.....................................................................................10-9

Configuration ................................................................................................................................10-9

Configuration Space...........................................................................................................10-10

Memory Mapped Configuration Space...............................................................................10-11

Configuration Requests......................................................................................................10-11

End-to-End CRC.........................................................................................................................10-12

Error Detection and Handling .....................................................................................................10-12

Power Management....................................................................................................................10-15

Initializing the Non-Transparent Bridge ......................................................................................10-16

Non-Transparent Port C Configuration Space Organization.......................................................10-18

Non-Transparent Mode Downstream Port C Configuration Space Organization Registers.......10-19

Port C Registers .........................................................................................................................10-21

PCI Express Capability Structure.......................................................................................10-31

Power Management Capability Structure...........................................................................10-38

Switch Control and Status Registers..................................................................................10-41

Extended Configuration Space Access and INTx Status Registers...................................10-47

PCI Express Virtual Channel Capability.............................................................................10-48

Physical Layer Control and Status Registers.....................................................................10-53

NTB Endpoint Configuration Space Organization ......................................................................10-55

NTB Internal Endpoint Configuration Space Registers...............................................................10-56

Non-Transparent Bridge Internal Endpoint Registers.................................................................10-59

PCI Express Capability Structure.......................................................................................10-68

Message Signaled Interrupt Capability Structure...............................................................10-72

Non-Transparent Bridge Configuration Capability Structure..............................................10-73

Non-Transparent Bridge Communications Capability Structure.........................................10-89

Power Management Capability Structure...........................................................................10-95

Extended Configuration Space Access Registers..............................................................10-97

PCI Express Extended Capability Header..........................................................................10-97

Non-Transparent Bridge Control and Status Registers......................................................10-98

NTB External Endpoint Configuration Space Registers ...........................................................10-104

Non-Transparent Bridge External Endpoint Registers..............................................................10-107

PCI Express Capability Structure.....................................................................................10-116

Message Signaled Interrupt Capability Structure.............................................................10-120

Non-Transparent Bridge Configuration Capability Structure............................................10-121

Non-Transparent Bridge Communications Capability Structure.......................................10-137

Power Management Capability Structure.........................................................................10-143

Extended Configuration Space Access Registers............................................................10-145

PCI Express Extended Capability Header........................................................................10-145

IDT Table of Contents

PES16NT2 User Manual iv April 15, 2008

Notes Non-Transparent Bridge Control and Status Registers....................................................10-146

JTAG Boundary Scan

Introduction...................................................................................................................................11-1

Test Access Point.........................................................................................................................11-1

Signal Definitions..........................................................................................................................11-1

Boundary Scan Chain...................................................................................................................11-3

Test Data Register (DR)...............................................................................................................11-3

Boundary Scan Registers.....................................................................................................11-4

Instruction Register (IR)................................................................................................................11-5

EXTEST................................................................................................................................11-6

SAMPLE/PRELOAD.............................................................................................................11-6

BYPASS...............................................................................................................................11-6

CLAMP.................................................................................................................................11-7

IDCODE................................................................................................................................11-7

VALIDATE............................................................................................................................11-7

RESERVED..........................................................................................................................11-7

Usage Considerations..........................................................................................................11-7

Table of contents

Other IDT Network Router manuals

IDT

IDT 89HPES12T3G2 User manual

Popular Network Router manuals by other brands

Allnet

Allnet ALL0333AU user guide

HomeSeer

HomeSeer HT-SEL Getting started guide

Sercomm

Sercomm Genesis OC3505D Quick installation guide

Billion

Billion BIPAC 5102 Series user manual

TP-Link

TP-Link TD-8817 QIG

Motorola

Motorola MR1900 user guide

Digi

Digi IX14 user guide

Cisco

Cisco NCS 540 Hardware installation guide

XAVi

XAVi X8821e user manual

BEC

BEC 6200WZL Quick installation guide

MikroTik

MikroTik hAP ac3 Series quick start

Korenix

Korenix JetNet 5428G Series user manual

NETGEAR

NETGEAR FS526T - Switch installation guide

Korenix

Korenix JetNet 5710G Series user manual

Automated Logic

Automated Logic ZN551 Technical instructions

Cisco

Cisco ASR 1000 Series Installation and configuration guide

EnGenius

EnGenius ESR-9710 quick start guide

Cisco

Cisco 805 Series Product overview