Cisco ONS 15454 User manual

CHAPTER

1-1

Cisco ONS 15454 Troubleshooting Guide, R4.6

November 2004

General Troubleshooting

This chapter provides procedures for troubleshooting the most common problems encountered when

operating a Cisco ONS 15454. To troubleshoot specific ONS 15454 alarms, see Chapter 2, “Alarm

Troubleshooting.” If you cannot find what you are looking for, contact the Cisco Technical Assistance

Center (1 800 553-2447).

Note The terms "UnidirectionalPath Switched Ring" and "UPSR" may appear in Cisco literature. These terms

do not refer to using Cisco ONS 15xxx products in a unidirectional path switched ring configuration.

Rather, these terms, as well as "Path Protected Mesh Network" and "PPMN," refer generally to Cisco's

path protection feature, which may be used in any topological network configuration. Cisco does not

recommend using its path protection feature in any particular topological network configuration.

This chapter includes the following sections on network problems:

•Network Troubleshooting Tests—Describes loopbacks and hairpin circuits, which you can use to

test circuit paths through the network or logically isolate faults.

Note For network acceptance tests, refer to the Cisco ONS 15454 Procedure Guide.

•Identify Points of Failure on a DS-N Circuit Path—Explains how to perform the tests described in

the “1.1 Network Troubleshooting Tests” section on a DS-N circuit.

•Using the DS3XM-6 Card FEAC (Loopback) Functions—Describes the far-end alarm and control

(FEAC) functions on the DS3XM-6 card.

•Identify Points of Failure on an Optical Circuit Path—Explains how to perform the tests described

in the “1.1 Network Troubleshooting Tests” section on an OC-N circuit.

The remaining sections describe symptoms, problems, and solutions that are categorized according to

the following topics:

•Restoring the Database and Default Settings—Provides procedures for restoring software data and

restoring the node to the default setup.

•PC Connectivity Troubleshooting—Provides troubleshooting procedures for PC and network

connectivity to the ONS 15454.

•CTC Operation Troubleshooting—Provides troubleshooting procedures for CTC login or operation

problems.

•Circuits and Timing—Provides troubleshooting procedures for circuit creation and error reporting

as well as timing reference errors and alarms.

1-2

Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004

Chapter 1 General Troubleshooting

1.1 Network Troubleshooting Tests

•Fiber and Cabling—Provides troubleshooting procedures for fiber and cabling connectivity errors.

•Power and LED Tests—Provides troubleshooting procedures for power supply and LED indicator

problems.

1.1 Network Troubleshooting Tests

Use loopbacks and hairpin circuits to test newly created SONET circuits before running live traffic or to

logically locate the source of a network failure. All ONS 15454 OC-N cards, transponder cards (TXP,

TXPP), muxponder (MXP) cards, and G-Series Ethernet cards allow loopbacks and hairpin test circuits.

Other cards do not allow loopbacks. These include ONS E-Series Ethernet, ML-Series Ethernet, and

DWDM cards such as OPT-BST, OPT-PRE, OSC-CSM, band add-drop cards, and channel add-drop

cards.

Caution Facility (line) or terminal loopbacks can be service-affecting. To protect traffic, apply a lockout or Force

switch to the target loopback port. For more information about these operations, refer to the

Cisco ONS 15454 Procedure Guide.

Caution On OC-N cards, a facility (line) loopback applies to the entire card and not an individual circuit. Exercise

caution when using loopbacks on an OC-N card carrying live traffic.

A facility (line) loopback tests the line interface unit (LIU) of a card, the EIA (electrical interface

assembly), and related cabling. After applying a facility loopback on a port, use a test set to run traffic

over the loopback. A successful facility loopback isolates the LIU, the EIA, or the cabling plant as the

potential cause of a network problem. Figure 1-1 shows a facility loopback on a DS-N card.

Figure 1-1 Facility (Line) Loopback Process on a DS-N Card

To test the LIU on an OC-N card, connect an optical test set to the OC-N port and perform a facility (line)

loopback or use a loopback or hairpin circuit on a card that is farther along the circuit path. Figure 1-2

shows a facility loopback on an OC-N card.

DS-N OC-NXC

5261

Test Set

1-3

Cisco ONS 15454 Troubleshooting Guide, R4.6

November 2004

Chapter 1 General Troubleshooting 1.1 Network Troubleshooting Tests

Figure 1-2 Facility (Line) Loopback Process on an OC-N Card

Caution Before performing a facility (line) loopback on an OC-N card, be sure the card contains at least two data

communications channel (DCC) paths to the node where the card is installed. A second DCC provides

a nonlooped path to log into the node after the loopback is applied, enabling you to remove the facility

loopback. Ensuring a second DCC is not necessary if you are directly connected to the ONS 15454

containing the loopback OC-N card.

A terminal loopback tests a circuit path as it passes through the cross-connect card (XC10G) and loops

back from the card with the loopback. Figure 1-3 shows a terminal loopback on an OC-N card. (Ethernet,

transponder, and muxponder loopbacks follow the same model.) The test-set traffic comes into the DS-N

card and goes through the cross-connect card to the OC-N card. The terminal loopback on the OC-N card

turns the signal around before it reaches the LIU and sends it back through the cross-connect card to the

DS-N card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does

not test the LIU on the OC-N card.

Setting a terminal loopback on the G-Series card might not stop the Tx Packets counter or the Rx Packet

counters on the CTC card-level view Performance > Statistics page from increasing. The counters can

increment even though the loopbacked port has temporarily disabled the transmit laser and is dropping

any received packets.

The Tx Packet statistic continues to increment because the statistic is not based on the packets

transmitted by the transmit (Tx) laser but on the Tx signal inside the G-Series card. In normal in-service

port operation, the Tx signal being recorded does result in the Tx laser transmitting packets, but in a

terminal loopback this signal is being looped back within the G-Series card and does not result in the Tx

laser transmitting packets.

The Rx Packet counter might also continue to increment when the G-Series card is in terminal loopback.

Receive (Rx) packets from any connected device are dropped and not recorded, but the internally looped

back packets follow the G-Series card’s normal receive path and register on the Rx Packet counter.

Figure 1-3 Terminal Loopback Process on an OC-N Card

DS-N OC-NXC

Test Set

78260

5266

OC-N

Test Set

XCDS-N

1-4

Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004

Chapter 1 General Troubleshooting

1.1 Network Troubleshooting Tests

Figure 1-4 shows a terminal loopback on a DS-N card. The test-set traffic comes in on the OC-N card

and goes through the cross-connect card to the DS-N card. The terminal loopback on the DS-N card turns

the signal around before it reaches the LIU and sends it back through the cross-connect card to the OC-N

card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does not test

the LIU on the DS-N card.

Figure 1-4 Terminal Loopback Process on a DS-N Card

ONS 15454 port loopbacks either terminate or bridge the loopback signal. In the ONS 15454 system, all

optical, electrical, Ethernet, and TXP/MXP facility loopbacks are terminated as shown in Table 1-1.

During terminal loopbacks, some ONS cards bridge the loopback signal while others terminate it.

If an optical, electrical, Ethernet, and TXP/MXP port terminates a terminal or facility loopback signal,

this means that the signal only loops back to the originating port and is not transmitted downstream. If

the port bridges a loopback signal, the signal loops back to the originating port and is also transmitted

downstream.

All ONS 15454 card bridging and terminating behaviors are listed in Table 1-1. When a port on a card

in the left column of this table originates a terminal or facility loopback, the signal behaves as listed in

the middle and right columns.

Note In Table 1-1, no AIS signal is injected if the signal is bridged. If the signal is terminated, an applicable

AIS is injected downstream for all cards except Ethernet cards.

DS-N

Test Set

XC OC-N

78259

Table 1-1 DS-N, OC-N, EC-N, and TXP/MXP Card Loopback Behavior

Card/Port Terminal loopback signal Facility loopback signal

DS-1 Terminated Terminated

DS-3 Bridged Terminated

DS-N transmux Bridged Terminated

All OC-N cards Bridged Terminated

EC-1 Bridged Terminated

G-Series Ethernet Terminated1

1. G-Series Ethernet terminal loopback is terminated and Ethernet transmission is disabled. No AIS is

inserted for Ethernet, but a TPTFAIL alarm is raised on the far-end Ethernet port.

Terminated2

2. G-Series facility loopback is terminated and no AIS is sent downstream. However, the Cisco Link

Integrity signal continues to be sent downstream.

TXP_MR_2.5G Bridged Terminated

TXP_MR_10G Bridged Terminated3

MXP_2.5G_10G Bridged Terminated4

1-5

Cisco ONS 15454 Troubleshooting Guide, R4.6

November 2004

Chapter 1 General Troubleshooting 1.1 Network Troubleshooting Tests

The loopback itself is listed in the Alarms window. For example, the window would list the

LPBKTERMINAL condition or LPBKFACILITY condition for a tested port.

In addition to the Alarms window listing, the following behaviors occur:

•If a DS-N, OC-N, or EC-1 port is placed in out of service (OOS) state, it injects an AIS signal

upstream and downstream.

•If a DS-N, OC-N, or EC-1 port is placed in out of service auto in-service (OOS_AINS) state or in

the out of service maintenance (OOS_MT) state before loopback testing, the port clears the AIS

signal upstream and downstream unless there is a service-affecting defect that would also cause an

AIS signal to be injected. For more information about placing ports into alternate states for testing,

refer to the Cisco ONS 15454 Procedure Guide.

•If a TXP_MR_2.5G port is placed in the out of service maintenance (OOS_MT) state before

loopback testing, the port raises an Alarms Suppressed for Maintenance (AS-MT) condition during

the loopback on the client side port.

•If a TXP_MR_10G port is placed in the out of service maintenance (OOS_MT) state before

loopback testing, the port raises a Loss of Signal (LOS) alarm on the DWDM-side port during the

loopback.

•If an MXP_2.5_10G port is placed in the out of service maintenance (OOS_MT) state before

loopback testing, the port raises a Loss of Signal (LOS) alarm on the DWDM-side port during the

loopback.

Bridged DS-N and OC-N terminal loopback examples are shown in Figure 1-5 and Figure 1-6.

Figure 1-5 Terminal Loopback on a DS-N Card with Bridged Signal

Figure 1-6 Terminal Loopback on an OC-N Card with Bridged Signal

3. A facility loopback on the client or trunk squelches the far end.

4. A facility loopback when the card is in transparent mode on the client or trunk squelches the far end.

Source

ONS Node

Destination

ONS Node

XCDS-N OC-N DS-NXCOC-N

Test Se

Test Set

15049

Source

ONS Node

Destination

ONS Node

XCOC-N OC-N OC-NXCOC-N

Test Se

Test Set

115050

1-6

Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004

Chapter 1 General Troubleshooting

1.2 Identify Points of Failure on a DS-N Circuit Path

A hairpin circuit brings traffic in and out on a DS-N port rather than sending the traffic onto the OC-N

card. A hairpin loops back only the specific STS or VT circuit and does not cause an entire OC-N port

to loop back, thus preventing a drop of all traffic on the OC-N port. The hairpin allows you to test a

specific STS or VT circuit on nodes running live traffic. Figure 1-9 shows the hairpin circuit process on

a DS-N card.

Figure 1-7 Hairpin Circuit Process on a DS-N Card

A cross-connect loopback tests a circuit path as it passes through the cross-connect card and loops back

to the port being tested. Testing and verifying circuit integrity often involves taking down the whole line;

however, a cross-connect loopback allows you to create a loopback on any embedded channel at

supported payloads at the STS-1 granularity and higher. For example, you can loop back a single STS-1,

STS-3c, STS-6c, etc. on an optical facility (line) without interrupting the other STS circuits.

The following restrictions apply to cross-connect loopbacks:

•You can create a cross-connect loopback on all working or protect optical ports unless the protect

port is used in a 1+1 protection group and is in working mode.

•If a terminal or facility loopback exists on a port, you cannot use the cross-connect loopback.

1.2 Identify Points of Failure on a DS-N Circuit Path

Facility (line) loopbacks, terminal (inward) loopbacks, and hairpin circuits are often used totest a circuit

path through the network or to logically isolate a fault. Performing a loopback test at each point along

the circuit path systematically isolates possible points of failure.

The example in this section tests a DS-N circuit on a two-node, bidirectional line switched ring (BLSR).

Using a series of facility loopbacks, terminal loopbacks, and hairpins, the path of the circuit is traced

and the possible points of failure are tested and eliminated. A logical progression of five network test

procedures apply to this sample scenario:

Note The test sequence for your circuits will differ according to the type of circuit and network topology.

1. A facility (line) loopback on the source node DS-N

2. A hairpin on the source node DS-N

3. A terminal (inward) loopback on the destination node DS-N

4. A hairpin on the destination node DS-N

5. A facility (line) loopback on the destination DS-N

Note All loopback tests require on-site personnel.

55485

OC-N

Test Set

XCDS-N

1-7

Cisco ONS 15454 Troubleshooting Guide, R4.6

November 2004

Chapter 1 General Troubleshooting 1.2.1 Perform a Facility (Line) Loopback on a Source DS-N Port

1.2.1 Perform a Facility (Line) Loopback on a Source DS-N Port

The facility (line) loopback test is performed on the node source port in the network circuit, in this

example, the DS-N port in the source node. Completing a successful facility (line) loopback on this port

isolates the cabling, the DS-N card, and the EIA as possible failure points. Figure 1-10 shows an example

of a facility loopback on a source DS-N port.

Figure 1-8 A Facility (Line) Loopback on a Circuit Source DS-N Port

A hairpin circuit brings traffic in and out on a DS-N port rather than sending the traffic onto the OC-N

card. A hairpin loops back only the specific STS or VT circuit and does not cause an entire OC-N port

to loop back, thus preventing a drop of all traffic on the OC-N port. The hairpin allows you to test a

specific STS or VT circuit on nodes running live traffic. Figure 1-9 shows the hairpin circuit process on

a DS-N card.

Figure 1-9 Hairpin Circuit Process on a DS-N Card