Juniper NETWORKS STRM - TECHNICAL NOTE REV 6-2008 User manual

Release 2008.2

JUNIPER NETWORKS STRM

TECHNICAL NOTE

USING EXTENSION DOCUMENTS

JUNE 2008

Device extensions allow you to modify how a DSM parses logs, which is useful for

resolving parsing issues. However, before you define a device extension, you must

build an extension document. This document provides information on defining an

extension document including:

•About Extension Documents

•Understanding Extension Document Elements

•Creating Extension Documents

•Device Type IDs

This document assumes an advanced knowledge of XML coding.

About Extension

Documents The extension document is specified in Extensible Markup Language (XML)

format. You can create and edit the document using any common word processing

application. You can create multiple extension documents and associate an

extension document to various device types. Using an extension document, you

can resolve parsing issues, such as:

•Fixing an event that has missing or incorrect fields (for example, if the

username is not being parsed).

•Completing the parsing of an event when the DSM to which it is attached fails to

produce a result. Any new events produced by the device extension are

associated to the device that failed to parse the original payload. This action

prevents these events from appearing as un-parsed in the STRM interface.

XML format allows for a simplification of the extension document’s organization,

and verifies the extension document’s contents. Using this format requires that all

regular expressions be contained in character data (CDATA) sections to prevent

the special characters that are needed by regular expressions from interfering with

the markup format. For example:

<![CDATA[(tcp|udp|icmp|gre)]]></pattern>

Where (tcp|udp|icmp|gre) is the actual regular expression pattern.

Release 2008.2

The extension document allows you to parse a device’s payload. Within the

extension document you can include statements of varying degrees of complexity,

as required to parse the desired information.

Understanding

Extension

Document

Elements

This section explains the two main divisions of the extension document:

•Patterns

•Match Groups

Patterns Rather than associating a regular expression directly with a particular field name,

patterns (patterns) are declared separately at the top of the extension document

and can be subsequently referenced multiple times within the file.

Match Groups A match group (match-group) is a set of patterns used for parsing or modifying

one or more types of events. A matcher is an entity within a match group that is

parsed (for example, EventName) and is paired with the appropriate pattern and

group for parsing. Any number of match groups may appear in the extension

document.

Table 1 Pattern Parameters

Parameter Description

(Required) Specify a regular string that is unique within the extension

document.

case-insensitive

(Optional) Specify if you wish the pattern to ignore character case when

doing a match, for example abc is the same as ABC.

If not specified, this parameter defaults to false.

trim-whitespace

(Optional) Specify if you wish the pattern to ignore white space and

carriage returns. If the CDATA sections are split onto different

lines, this parameter ensures that any extra spaces and

carriage returns are not interpreted as part of the pattern.

If not specified, this parameter defaults to false.

Table 2 Match Group Parameters

Parameter Description

order

(Required) Specify an integer greater than zero to specify the order in which

the match groups are executed. It must be unique within the

extension document.

description

(Optional) Specify a description for the match group, which can be any string.

This information may appear in the logs.

If not specified, this parameter defaults to empty.

Release 2008.2

Understanding Extension Document Elements 3

Match groups can have up to three different types of entities:

•Matcher (matcher)

•Single-Event Modifier (event-match-single)

•Multi-Event Modifier (event-match-multiple)

Matcher (matcher)

A matcher entity is a field that is parsed (for example, EventName) and is paired

with the appropriate pattern and group for parsing. Matchers have an associated

order, so if multiple matchers are specified for the same field name, the matchers

are executed in the order presented until a successful parse is found or a failure

occurs.

device-type-

id-override

(Optional)

Specify a different device’s QID. Allows the particular match group

to search in the specified device for the event type. It must be a

valid device type ID, represented as an integer. A list of device type

IDs is presented in Table 6.

If not specified, this parameter defaults to the device type of the

device to which the extension is attached.

Table 3 Matcher Entity Parameters

Parameter Description

field

(Required) Specify the field to which you wish the pattern to apply,

for example, EventName, or SourceIp. See Table 4 for a

list of valid field names.

pattern-id

(Required) Specify the pattern you wish to use when parsing the

field out of the payload. This value must match (including

case) the ID parameter of the pattern previously defined

in a pattern ID parameter (Table 1).

order

(Required) Specify the order that you wish this pattern to attempt

among matchers assigned to the same field. If there are

two matchers assigned to the EventName field, the one

with the lowest order is attempted first.

Table 2 Match Group Parameters (continued)

Parameter Description

Release 2008.2

capture-group

(Optional) Specify capture group(s), as denoted in the regular

expression inside parenthesis ( ). These captures are

indexed starting at one and processed from left to right

in the pattern. The capture-group field must be a

positive integer less than or equal to the number of

capture groups contained in the pattern. The default

value is zero, which will be the entire match.

For example, you can define a single pattern for a

source IP address andport; wherethe SourceIpmatcher

may use a capture group of 1, and the SourcePort

matcher may use a capture group of 2, but only one

pattern needs to be defined.

This field has a dual purpose when combined with the

enable-substitutions parameter.

enable-substitutions

(Optional) Specify this Boolean parameter as true when a field

cannot be adequately represented with a straight group

capture.

You can combine multiple groups together with extra

text to form a value. This parameter enables that

behavior.

This parameter changes the meaning of the

capture-group parameter. The capture-group

parameter creates the new value, and group

substitutions are specified using \x where xis a group

number from 1 to 9. You may use groups multiple times,

and any free-form text can also be inserted into the

value. For example, if you need to form a value out of

group 1, followedby an underscore, followedby group 2,

an @, and then group 1 again, the appropriate

capture-group syntax is:

capture-group=”\1_\2@\1”

In another example, a MAC address is separated by

colons, but STRM assumes that MAC addresses are

hyphen separated. The syntax to parse and capture the

individual portions is:

capture-group=”\1:\2:\3:\4:\5:\6”

If no groups are specified in the capture-group when

substitutions are enabled, a direct text replacement

occurs.

Default is false.

Table 3 Matcher Entity Parameters (continued)

Parameter Description

Release 2008.2

Understanding Extension Document Elements 5

Table 4 provides a list of valid field names for use in the matcher field parameter (see

Table 3).

Table 4 Matcher Field Names

Field Name Description

EventName

(Required) Specify the event name to be retrieved from the QID to

identify the event.

EventCategory Specify an event category for any event with a category

not handled by an event-match-single entity or an

event-match-multiple entity.

Combined with EventName, EventCategory is used to

search for the event in the QID.

SourceIp Specify the source IP address for the message.

SourcePort Specify the source port for the message.

SourceIpPreNAT Specify the source IP address for the message before

Network Address Translation (NAT) occurs.

SourceIpPostNAT Specify the source IP address for the message after NAT

occurs.

SourceMAC Specify the source MAC address for the message.

SourcePortPreNAT Specify the source port for the message before NAT

occurs.

SourcePortPostNAT Specify the source port for the message after NAT occurs.

DestinationIp Specify the destination IP address for the message.

DestinationPort Specify the destination port for the message.

DestinationIpPreNAT Specify the destination IP address for the message before

NAT occurs.

DestinationIpPostNAT Specify the destination IP address for the message after

NAT occurs.

DestinationPortPreNAT Specify the destination port for the message before NAT

occurs.

DestinationPortPostNAT Specify the destination port for the message after NAT

occurs.

DestinationMAC Specify the destination MAC address for the message.

DeviceTime Specify the time that the event was sent, according to the

device (this is NOT the time that the event arrived).

STRM detects timestamps in the following formats:

•Valid syslog timestamp in the form of mm dd hh:mm:ss,

for example:

Jan 13 12:33:10

•Current locale timestamp

Any other formats will not properly convert.

Release 2008.2

Single-Event Modifier (event-match-single)

Single-event modifier (event-match-single) matches (and subsequently

modifies) exactly one type of event, as specified by the required, case-sensitive

EventName parameter. This entity allows mutation of successful events by

changing the device event category, severity, or the method for sending identity

events.

When events matching this event name are parsed, the device category, severity,

and identity properties are imposed upon the resulting event. An

event-match-single entity consists of three optional properties:

Protocol Specify the protocol associated with the event; for

example, TCP, UDP, or ICMP.

If a protocol is not properly parsed out of a message, ports

that were parsed may not appear in STRM (it only displays

ports for port-based protocols).

UserName Specify the user name associated with the event.

HostName Specify the host name associated with the event. This field

is usually only associated with identity events.

GroupName Specify the group name associated with the event. This

field is usually only associated with identity events.

NetBIOSName Specify the NetBIOS name associated with the event. This

field is usually only associated with identity events.

Table 5 Single-Event Modifier Parameters

Parameter Description

device-event-category Specify a new category for searching in the QIDfor the

event. This is an optimizing parameter, since some

devices have the same category for all events.

severity Specify the severity of the event. This parameter must

be an integer value between 1 and 10.

If a severity of less than 1 or greater than 10 is

specified, the system defaults to 5.

If not specified, the default is whatever is found in the

QID.

Table 4 Matcher Field Names (continued)

Field Name Description

Release 2008.2

Creating Extension Documents 7

Multi-Event Modifier (event-match-multiple)

The multi-event modifier (event-match-multiple) matches a range of event

types (and subsequently modifies) as specified by the pattern-id parameter

and the capture-group-index parameter.

Note: This match is not run against the payload, but is run against the results of

the EventName matcher previously parsed out of the payload.

This entity allows mutation of successful events by changing the device event

category, severity, or the method the event uses to send identity events. The

capture-group-index must be an integer value (substitutions are not

supported) and pattern-ID must reference an existing pattern entity. All other

properties are identical to their counterparts in the single-event modifier

Creating Extension

Documents This section provides you with information on creating extension documents

including:

•Writing a Complete Extension Document

•Uploading Extension Documents

•Solving Specific Parsing Issues

send-identity Specifies the sending of identity change information

from the event. Choose one of the following options:

•UseDSMResults – If the DSM returns an identity

event, the event is passed on. If the DSM does not

return an identity event, the DSM does not create or

modify the identity information.

This is the default value if no value is specified.

•SendIfAbsent – If the DSM creates identity

information, the identity event is passed through

unaffected. If no identity event is produced by the

DSM, but there is enough information in the event

to create an identity event, an event is generated

with all the relevant fields set.

•OverrideAndAlwaysSend – Ignores any identity

event returned by the DSM and creates a new

identity event, if there is enough information.

•OverrideAndNeverSend – Suppress any identity

information returned by the DSM.

Table 5 Single-Event Modifier Parameters (continued)

Parameter Description

Release 2008.2

Writing a Complete

Extension Document The example of an extension document included in this section provides

information on how to parse one particular type of Cisco FWSM so that events are

not sent with an incorrect event name. For example, if you wish to resolve the word

session, which is embedded in the middle of the event name:

Nov 17 09:28:26 129.15.126.6 %FWSM-session-0-302015: Built UDP

connection for faddr 38.116.157.195/80 gaddr

129.15.127.254/31696 laddr 10.194.2.196/2157 duration 0:00:00

bytes 57498 (TCP FINs)

This condition causes the DSM to not recognize any events and all the events are

un-parsed and associated with the generic logger.

Although only a portion of the text string (302015) is used for the QID search, the

entire text string (%FWSM-session-0-302015) identifies the event as coming from

a Cisco FWSM. Since the entire text string is invalid, the DSM assumes that the

event is invalid.

A FWSM device has a large number of event types, many with unique formats.

The following extension document example, indicates how to parse one event

type.

The pattern IDs do not have to match the field names that they are parsing. Even

though the following example duplicates the pattern, the SourceIp field and the

SourceIpPreNAT field could use the exact same pattern in this case (this may not

be true in all FWSM events).

<?xml version="1.0" encoding="UTF-8"?>

<device-extension xmlns="event_parsing/device_extension">

<pattern id="SourceIp" xmlns=""><![CDATA[gaddr

(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>

<pattern id="SourceIpPreNAT" xmlns=""><![CDATA[gaddr

(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>

<pattern id="SourceIpPostNAT" xmlns=""><![CDATA[laddr

(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>

<pattern id="DestinationIp" xmlns=""><![CDATA[faddr

(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>

<pattern id="Protocol" case-insensitive="true"

xmlns=""><![CDATA[(tcp|udp|icmp|gre)]]></pattern>

<match-group order="1" description="FWSM Test" device-type-id-override="6" xmlns="">

Release 2008.2

Creating Extension Documents 9

<matcher field="Protocol" order="2" pattern-id="Protocol_6" capture-group="TCP"

enable-substitutions=”true”/>

<event-match-multiple pattern-id="EventNameId" capture-group-index="1"

device-event-category="Cisco Firewall"/>

</match-group>

</device-extension>

The above extension document example demonstrates some of the basic aspects

of parsing:

•IP addresses

•Ports

•Protocol

•Multiple fields using the same pattern with different groups

This example parses all FWSM events that follow the specified pattern, although

the fields that are parsed may not be present in those events (if the events include

different content).

The information that was necessary to create this configuration that was not

available from the event:

•The event name is only the last six digits (302015) of the

%FWSM-session-0-302015 portion of the event.

•The FWSM has a hard-coded device type category of Cisco Firewall.

•The FWSM uses the Cisco Pix QID and therefore includes the

device-type-id-override="6" parameter in the match group (the Pix

firewall’s device type ID is 6, see Table 6).

If the QID information is not specified or is unavailable, you can modify the event

mapping using the Event Viewer. For more information, see the Modifying Event

Mapping section in the STRM Users Guide.

An event name and a device event category is required when looking for the event

in the QID. This device event category is a grouping parameter within the database

that helps define like events within a device. The event-match-multiple at the

end of the match group includes hard-coding of the category. The

event-match-multiple uses the EventNameId pattern on the parsed event

name to match up to six digits. This pattern is not run against the full payload, just

that portion parsed as the EventName field.

The EventName pattern references the %FWSM portion of the events; all Cisco

FWSM events contain the %FWSM portion. The pattern in the example matches

%FWSM followed by any number (zero or more) of letters and dashes. This pattern

match resolves the word session that is embedded in the middle of the event

name that needs to removed. The event severity (according to Cisco), followed by

Release 2008.2

a dash and then the true event name as expected by STRM. The only string with a

capture group (that is, bounded by parenthesis) is this pattern of digits

(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).

The IP addresses and ports for the event all follow the same basic pattern: an IP

address followed by a slash followed by the numeric port number. This pattern

parses two pieces of data (the IP address and the port), and specifies different

capture groups in the matcher section.

Thus, the IP address/port patterns are four sets of one to three digits, separated by

periods followed by a slash and the port number. The IP address section is in a

group, as is the port number (but not the slash). As was previously mentioned, the

matcher sections for these fields reference the same pattern name, but a different

capture group (the IP address is group 1 and the port is group 2).

The protocol is a common pattern that searches the payload for the first instance of

TCP, UDP, ICMP, or GRE (the pattern is marked with the case-insensitive

parameter so that any occurrence matches).

Note: You must search for the protocol when writing extension documents, as

STRM may not display port numbers if the event is not based on a port-based

protocol. See Converting a Protocol for an example of how to search for a protocol.

Although a second protocol pattern does not occur in the event being used as an

example, there is a second protocol pattern defined with an order of two. If the

lowest-ordered protocol pattern does not match, the next one is attempted (and so

on). The second protocol pattern also demonstrates the concept of a direct

substitution; there are no match groups in the pattern, but with the

enable-substitutions parameter enabled, the text TCP can be used in place of

protocol=6.

Uploading Extension

Documents Multiple extension documentscan be created, uploaded, and associated to various

device types. Extension documents can be stored anywhere prior to uploading to

STRM. When you select an extension document for uploading, STRM validates

the document against the internal XSD. STRM also verifies the validity of the

document before uploading to the system. For more information about device

extensions, see Managing Sensor Devices Guide.

This manual suits for next models

Other Juniper Software manuals

Juniper

Juniper JUNOSE SOFTWARE FOR E SERIES 11.3.X - COMMAND REFERENCE A TO M... Installation manual

Juniper

Juniper JUNIPER CARE SERVICES User manual

Juniper

Juniper NETWORK AND SECURITY MANAGER 2010.4 - NSMXPRESS SERIES II REV... User manual

Juniper

Juniper JUNOSE SOFTWARE FOR E SERIES 11.3.X - PHYSICAL LAYER CONFIGURATION GUIDE... User manual

Juniper

Juniper JUNOSE SOFTWARE FOR E SERIES 11.3.X - IP-IPV6-IGP CONFIGURATION GUIDE... User manual

Juniper

Juniper SECURITY THREAT RESPONSE MANAGER 2008.2 - MANAGING VULNERABILITY ASSESSMENT... User manual

Juniper

Juniper NETWORK AND SECURITY MANAGER - NSM CONFIGURATION GUIDE FOR EX SERIES DEVICES REV... User manual

Juniper

Juniper STRM 2008-2 - TECHNICAL NOTE CHANGING NETWORK SETTING... User manual

Juniper

Juniper JUNOS PULSE 2.0 RELEASE NOTES Instruction Manual

Juniper

Juniper NETWORK AND SECURITY MANAGER 2010.4 - REV1 User manual

Juniper

Juniper CARE PLUS User manual

Juniper

Juniper NETWORK AND SECURITY MANAGER 2010.2 - ADMINISTRATION GUIDE... Instruction Manual

Juniper

Juniper JUNOSE 11.1.0 - REV 4-29-2010 User manual

Juniper

Juniper STRM 2008-2 - TECHNICAL NOTE CHANGING NETWORK SETTING... Installation and operation manual

Juniper

Juniper JUNOSE SOFTWARE 11.0.X - COMMAND REFERENCE A TO M... Installation manual

Juniper

Juniper SYSTEM BASICS - CONFIGURATION GUIDE V 11.1.X User manual

Juniper

Juniper RESIDENT ENGINEER User manual

Juniper

Juniper NETWORK AND SECURITY MANAGER 2010.4 - CONFIGURING INTRUSION DETECTION PREVENTION DEVICES GUIDE REV... User manual

Juniper

Juniper STRM 2008.2 RELEASE NOTES Instruction Manual

Juniper

Juniper JUNOSE SOFTWARE 11.2.X - LINK LAYER CONFIGURATION GUIDE... User manual

Juniper

Juniper MEDIA FLOW MANAGER 2.0.2 - ADMINISTRATOR S GUIDE AND... Service manual

Juniper

Juniper MWS1000 Installation instructions

Juniper

Juniper JUNOSE 11.0.X PHYSICAL LAYER User manual

Juniper

Juniper SECURITY THREAT RESPONSE MANAGER 2008.2 R2 - INSTALLATION... User manual

Popular Software manuals by other brands

COMPRO

COMPRO COMPROFM manual

Muratec

Muratec OFFICEBRIDGE ONLINE user guide

Oracle

Oracle Contact Center Anywhere 8.1 installation guide

Adobe

Adobe 65007312 - Photoshop Lightroom Programmer's guide

Avaya

Avaya NULL One-X for RIM Blackberry user guide

HP ProLiant BL420c user guide

PS Audio

PS Audio PowerPlay Programming manual

Brady

Brady LOCKOUT PRO 3.0 Administrator's guide

Avaya

Avaya Interaction Center user guide

Texas Instruments

Texas Instruments TI-83 Plus Silver Edition Guide book

Novell

Novell GROUPWISE 8 - INTERNET AGENT manual

Oracle

Oracle Application 9i Configuration guide

Acer

Acer RDM user guide

Canon

Canon Vixia HF21 instruction manual

Canon

Canon ZR950 instruction manual

Samsung

Samsung Auto Backup user manual

Polycom

Polycom Vortex EF2201 Application note

Brocade Communications Systems

Brocade Communications Systems Brocade 8/12c user manual