NetXen Intelligent NIC Quick start guide

NetXen, Inc.

Intelligent NIC™ Installation and Tuning Guide

Version 3.7

November 14, 2008

This document is applicable to releases 3.4.337

NetXen, Inc.

18922 Forge Drive

Cupertino, CA 95014-0701

Telephone: (408) 861-4500

Web Site: http://www.netxen.com

NetXen, Inc

Disclaimer

This is a preliminary document and is subject to change without prior notification and does

not represent a commitment on the part of NetXen, Inc. The information contained herein is

the proprietary and confidential information of NetXen or its licensors, and is supplied subject

to, and may be used only by NetXen’s customer in accordance with, a written agreement

between NetXen and its customer. Except as may be explicitly set forth in such agreement,

NetXen does not make, and expressly disclaims, any representations or warranties as to the

completeness, accuracy or usefulness of the information contained in this document.

Please contact NetXen, Inc. to obtain the latest revision of this document.

Trademarks

Intelligent NIC is a registered trademark of NetXen, Inc. NetXen and the NetXen logo are

trademarks of NetXen, Inc. Third Party brands and names are the property of their respective

owners.

Revision History

Version Date Editor Notes

3.7 11/14/08 SR Added Section 9, Teaming Driver.

Added Section 10, NIC VLAN in Windows.

Table of Contents

NetXen, Inc. i

1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 NIC Versions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.3 Currently Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Software Release Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 Hardware and Software Installation Requirements. . . . . . . . . . . . . . . . . . . 2

2.2.1 Cable Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2.2 XFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3

3 Quick-Start Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Pre-Installation Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.1 Guidelines on Firmware Tools and Processes . . . . . . . . . . . . . . . . . . . . . . 7

4.2 Linux Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2.1 Building the Linux Driver Source Code. . . . . . . . . . . . . . . . . . . . . 9

4.2.2 Linux NIC - GPL Inbox Kernel and Commercial Drivers. . . . . . . . 9

4.2.3 Linux LSA Driver Installation (NIC Commercial Driver Only) . . . 10

4.2.4 MSI-X Initialization and Support. . . . . . . . . . . . . . . . . . . . . . . . . 10

4.2.5 Auto-load Firmware Location . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.2.6 Host Memory Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.3 Windows Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.4 VMware Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5 Detailed Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.1 Installing and Connecting the Intelligent NIC Adapter. . . . . . . . . . . . . . . . 16

5.2 Windows Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5.2.1 NIC Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5.2.2 Configuring the Network Interface Card . . . . . . . . . . . . . . . . . . . 25

5.3 Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.3.1 Linux NIC Driver Installation (TAR format) . . . . . . . . . . . . . . . . . 28

5.3.2 Linux NIC Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.3.3 Configuring the NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.4 Uninstalling the Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.4.1 Uninstalling the NIC Driver in Windows . . . . . . . . . . . . . . . . . . . 30

5.4.2 Uninstalling the NetXen NIC Driver in Linux. . . . . . . . . . . . . . . . 30

5.4.3 Uninstalling the LSA Driver in Linux. . . . . . . . . . . . . . . . . . . . . . 30

6 Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6.1 Using the NetXen Flash Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6.2 Windows Firmware Update. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6.3 Linux Firmware Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.4 Updating the Flash on Multiple Standalone Cards in Windows. . . . . . . . . 33

7 Linux Sockets Acceleration (LSA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.1 LSA Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.1.1 LSA Driver Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . 35

7.1.2 TAR Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.2 LSA Offload Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.2.1 LSA Configuration Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.2.2 LSA Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

ii NetXen, Inc.

Table of Contents

7.2.3 nxoffload Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.3 Running LSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.4 LSA Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.4.1 Example Read Out. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8 Linux Configuration for Bonding Mode and Virtual LAN (VLAN). . . . . . . . . . . . . . 42

8.1 Linux Teaming/Bonding Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

8.1.1 Mode 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

8.1.2 Mode 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8.1.3 Bonding Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8.1.4 Bonding Driver Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

8.2 Configuring VLAN under Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

9 Windows Teaming Driver and Virtual LAN (VLAN). . . . . . . . . . . . . . . . . . . . . . . . 46

9.1 Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

9.2 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

9.3 Fault Tolerance Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

9.4 NIC VLAN in Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

9.4.1 Optional Configuration on the NIC and Switch Side. . . . . . . . . . 58

9.4.2 NIC VLAN as Send Side. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

9.4.3 NIC VLAN as Receive Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

10 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

10.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 61

10.2 Unable to ping the NIC Installed on the Host . . . . . . . . . . . . . . . . . . . . . . 61

10.3 Gather System Information and Check for Errors on Linux Host . . . . . . . 62

10.4 Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

10.4.1 MSI-X Interrupts for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

10.4.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

10.5 Additional Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

11 User Diagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

11.1 Windows Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

11.2 Linux Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Appendix A Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

A.1 Windows Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

A.1.1 NTttcp (Windows) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

A.1.2 nttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . . . . 69

A.2 Windows Optimization — Tcp1323Opts . . . . . . . . . . . . . . . . . . . . . . 70

A.3 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

A.3.1 IPERF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

A.4 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Appendix B Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

1. Overview

NetXen, Inc. Intelligent NIC Installation and Tuning Guide 1

1 Overview

This guide describes the system requirements, installation, and setup required for the

NetXen Intelligent NIC® network interface card product release.

1.1 Related Documents

Release notes included in the software release.

Linux ‘man’ pages:

- man nx_nic

- man nx_lsa

Check NetXen web site for updates.

1.2 NIC Versions

The NIC products support Windows, Linux, and other operating systems. See the release

notes for an updated list.

1.3 Currently Supported Features

Note: Check with NetXen periodically for software updates and additional features.

Product Number Product Definition

NXB-10GSXR Single 10-GE port (optical fiber)

NXB-10CX4 Single 10-GE port (copper)

Feature Windows Linux

TCP, UDP, and IP checksum offload Yes Yes

TCP segmentation offload (TSO / LSO) Yes Yes

Interrupt coalescing Yes Yes

Interrupt distribution No Yes

VLAN Yes Yes

SNMP Ethernet MIB Yes Yes

Jumbo frames Yes Yes

Teaming and Failover Yes Yes

Linux Sockets Acceleration (LSA) N/A Yes

Receive Side Scaling (RSS) Yes Yes

2Intelligent NIC Installation and Tuning Guide NetXen, Inc.

2. System Requirements

2 System Requirements

Before installing the product, verify the minimum system configuration requirements

outlined below.

2.1 Software Release Package Contents

2.2 Hardware and Software Installation Requirements

2.2.1 Cable Specifications

Contents Format Location

Documentation Release Notes NetXen web site

NIC Linux Driver [version].tar.gz [version]/driver/nx_nic.ko or

modprobe for nx_nic

NIC Windows Driver [version].zip [version]\win2k3\x64 or

[version]\win2k3\x86

NX Intercept Driver [version].-lsa.tar.gz [version]-lsa/driver/nx_intercept.ko

or modprobe for nx_intercept

LSA Driver [version].-lsa.tar.gz [version]-lsa/driver/nx_lsa.ko or

modprobe for nx_lsa

Firmware cx4_romimage or xfp_romimage [version]\win2k3\utils

or [version]/bin

Diagnostics nxudiag [version]\win2k3\utils

or [version]/bin

Tools nxflash [version]\win2k3\utils

or [version]/bin

Table 1 NXB-10GXSR Cable Specifications — Optical1

1. Check with NetXen for information on the NXB-10GXLR long range optical cable

specifications.

Parameter Optical Core Thickness

50 μm62.5 μm

Wavelength 850 nm 850 nm

Bandwidth (min) 500 MHz 500 MHz

Operating Range 300 m 300 m

Table 2 NXB-10CX4 Cable Specifications — Copper

802.3 Physical Layout Network Type Cable Length

10GBASE-CX4 LAN 15m (max)

2. System Requirements

NetXen, Inc. Intelligent NIC Installation and Tuning Guide 3

2.2.2 XFP Module

2.2.3 System and Software Requirements

Table 3 NXB-10GXSR XFP Module Specifications

Manufacturer Name Part Number

JDS Uniphase Multi-rate 10 Gb/s Short Wave

Optical XFP Transceiver 64P0194

PicoLight Multi-rate 10 Gb/s 850 nm

Optical XFP Transceiver Module PL-XXL-SC-S45-22

Table 4 System and Software Requirements

Hardware Systems Validated

Platforms Validated OS (32bit

and x64) Firmware1

1. NIC and LSA use the same firmware.

Driver

PCIe:

(x8) Host

Memory:

Minimum

2GB

AMD, Intel Windows Server:

2003 SP2, R2

Linux Distros:

RHEL4 U3,U4

SLES9 SP3,

SLES10, RHEL5

NIC

LSA Linux:2

NIC, LSA

Windows3

NIC

2. For Linux NIC and LSA drivers, install the kernel source and header files

3. For Windows NIC, SP2 is required.

4Intelligent NIC Installation and Tuning Guide NetXen, Inc.

3. Quick-Start Installation Instructions

3 Quick-Start Installation Instructions

This section is for users already familiar with the Windows or Linux operating systems.

For more information, refer to the pre-installation guidelines in Section 4. For a detailed

description of the installation and configuration procedure, refer to Section 5.

Table 5 Quick Start Instructions — New NIC Driver Installation

NetXen NIC Hardware Windows NIC Driver1

1. Refer to Section 4.3 of this document for more information on Windows pre-installation guidelines.

Linux NIC Commercial Version Driver2

2. Refer to Section 4.2 of this document for more information on Linux pre-installation guidelines.

Pre-Installation

1. Verify the system meets the requirements

described in Section 2.

2. Power down the system.

Installing the NetXen NIC Adapter

1. Insert the NetXen NIC adapter card into an

available PCI Express slot.

2. Connect the appropriate cables to the

board.

3. Power on the system.

Note that the NIC card comes with the

firmware already installed.

Verify NIC Adapter Installation

Once the NetXen NIC adapter has been

installed, verify the installation as follows.

For Linux-based systems, execute the

following command:

lspci -vvv -d 4040

For Windows-based systems, execute the

following sequence:

1. Right-click on My Computer

2. Select Manage;

3. Select Device Manager;

4. Check under Other devices for an

unknown device;

5. Right-click and select Properties for the

unknown device;

6. Select the Details tab and look for

VEN_4040 as part of the string.

Driver Installation

1. Unzip the provided NIC driver files to a

directory.

2. From the Device Manager, scan for new

hardware.

3. When prompted to connect to Windows

Update, Click No, not at this time, then

click Next.

4. In the next window, click Install from a

list of specific location and then click

Next.

5. Select Search for the best driver in these

locations. Select Include this location in

the search.

6. Provide the path to the directory where

the driver nxp2nic.sys3is located.

NIC Card Configuration

1. Open the Network Connections window

from the Control Panel.

2. Verify that NetXen NIC adapter icon says

Connected.

If not, right-click on the NIC Adapter

icon and select Properties, then assign

the IP and subnet addresses and click OK.

3. Use the x86 directory for 32-bit operating systems, and the x64 directory for 64-bit operating systems.

Driver Installation (tar.gz)

1. Untar the file, build, install, and load the

driver:

tar xzf [version].tar.gz

cd [version]

make

make install

modprobe nx_nic

NIC Card Configuration

1. Identify the NetXen Ethernet interface:

cat /proc/net/nx_nic/dev0 | grep eth

2. Assign an IP address:

ifconfig eth[n] [IP address]

3. Quick-Start Installation Instructions

NetXen, Inc. Intelligent NIC Installation and Tuning Guide 5

If a NetXen Intelligent NIC card is already installed on the system and a driver and/or

firmware update is required, refer to the procedure in Table 6 below.

Table 6 Quick Start Instructions — Updating from an Existing Driver and Firmware

Windows Driver/Firmware Update1

1. Refer to Section 4.1 for more information on firmware tools and guidelines.

Linux Driver/Firmware Update1VMware Firmware Update2

2. Refer to Section 4.4 for more information on VMware pre-installation guidelines

Verify Link Status

1. With the existing driver and firmware

installed, verify that the interface is up

before continuing with the firmware

update.

Update to New Firmware

1. Open a Command window.

2. Go to the utils directory in the driver

package provided.

3. The following binaries are required to be

in the same directory from where the

Flash command will be executed:

nxflash.exe

xfp_romimage

cx4_romimage

4. Run the following commands:

nxflash -i NX_NIC --all3

5. Verify the firmware has been updated

using the following command.

nxflash -i NX_NIC --info

Update to the New Driver

1. Install the new driver version.

2. Verify the new driver version.

3. Reboot the system after the firmware

update.

3. Writing of the Flash is a critical operation. Please do not interrupt or reboot during this time.

Verify Link Status

1. With the existing driver and firmware

installed, verify that the interface is up

using ifconfig before continuing with the

firmware update.

Update to New Firmware

1. Run the following command from the bin

directory of the driver:

./nxflash -i eth[n] --all3

Unload and Remove the Existing

Driver

1. Unload the existing driver.

ifconfig eth[n] down

rmmod nx_nic

2. Remove the existing driver module.

rm -f /lib/modules/[kernel version]/

kernel/

drivers/net/nx_nic.ko4

Reboot the System

1. After the new firmware has been flashed

and the old firmware has been removed,

reboot the system.

Update to the New Driver

1. Update to the new driver as follows:

tar xzf [version].tar.gz

cd [version]

make

make install

modprobe nx_nic

2. Ensure that the Ethernet interface is up.

ifconfig eth[n] <ip addr> up

4. For Linux kernel 2.4, the driver will appear as a nx_nic.o file.

Firmware Update Methods

There are three ways to perform a firmware

update in VMware.

1. Using a non-virtualized Linux OS host.

2. Using a non-virtualized Windows OS

host.

3. Using the NetXen LiveCD.

For a VMware firmware update using a non-

virtualized Linux OS, first perform the steps

listed in the rightmost column of Table 5

above, “Linux NIC Driver Installation”.

Next perform the steps listed in the middle

column of Table 6, “Linux Driver/Firmware

Update”.

For a VMware firmware update using a non-

virtualized Windows OS, first perform the

steps listed in the middle column of Table 5

above, “Windows NIC Driver Installation”.

Next perform the steps listed in the leftmost

column of Table 6, “Windows Driver/

Firmware Update”.

For a firmware update using the NetXen

LiveCD, download the NetXen LiveCD5

ISO image and burn that image onto a CD

and boot the system with the CD. Use option

2 from the LiveCD menu shown below to

update the flash on the card.

NetXen LiveCD Firmware Update

Menu Options:

1. Show NetXen board information

2. Update to latest firmware on NetXen

board

3. Reboot system if firmware has been

updated

Enter here:

5. Refer to the support page on the on NetXen web site for more information on downloading the LiveCD.

6Intelligent NIC Installation and Tuning Guide NetXen, Inc.

3. Quick-Start Installation Instructions

Once the WIndows or Linux NIC installation is complete as described in the previous tables, the following

procedures in Table 7 can be used to enable the Linux LSA driver.

Table 7 Quick Start Instructions — Enabling Linux LSA Driver

Enabling Linux LSA Driver

Note: the Linux NIC driver must be installed before executing the procedure below.

1. Untar the file, build, and load the drivers.

tar xzf [version]-lsa.tar.gz

cd [version]-lsa

make

cd driver

insmod ./nx_intercept.ko

insmod ./nx_lsa.ko

2. To verify the LSA driver is operating, execute the following command:

cat /proc/net/nx_nic/lsa_1/stats

3. Example for offloading iperf.

cd ../nxoffload

./nxoffload –a –n iperf

4. To display the current offload rules execute the following:

./nxoffload -s

This manual suits for next models

Table of contents

Other NetXen Network Card manuals

NetXen

NetXen NXB-10GSXR User manual

Popular Network Card manuals by other brands

FirstComm

FirstComm OBiWiFi5G Setup guide

Sun Microsystems

Sun Microsystems Interface Adapter user guide

TRENDnet

TRENDnet TU2-ETG user guide

YASKAWA

YASKAWA SI-EN3 Technical manual

Paradyne

Paradyne BitStorm 2600 IP DSLAM installation guide

Belkin

Belkin F5D7050 user manual

3Com

3Com EtherLink 3C509B quick guide

TRENDnet

TRENDnet TEG-PCITX user guide

Cisco

Cisco MC16E installation guide

Delta Electronics

Delta Electronics Braking Modules VFDB Series instruction sheet

TwinMOS

TwinMOS Booming user manual

Dialogic

Dialogic Media Board DM/V1200BTEPEQ installation guide

Buffalo

Buffalo AirStation WLI-PCM-L11GP user manual

National Instruments

National Instruments NI 9234 operating instructions

Hama

Hama 49276 Operating instruction

Linksys

Linksys WCF54G - Wireless-G Compact Flash Card Product data

Compaq

Compaq Wireless LAN 100 specification

Bose

Bose PowerMatch Dante quick start guide