Custom Parsers

To start creating a custom parser for device logs, you should begin by reviewing the Event Parser XML Specification. Writing the XML specification is the primary task in creating a custom parser.

Event Parser XML Specification

Custom Parser XML Specification Template

Parser Name Specification

Device or Application Type Specification

Format Recognizer Specification

Pattern Definition Specification

Parsing Instructions Specification

Creating a Custom Parser

Deleting or Disabling a Parser

Exporting a Custom Parser

Importing a Custom Parser

Parser Examples

Cisco IOS Syslog Parser

 

Event Parser XML Specification

FortiSIEM uses an XML-based parser framework to parse events. These topics describe the parser syntax and include examples of XML parser specifications.

Custom Parser XML Specification Template

Parser Name Specification

Device or Application Type Specification

Format Recognizer Specification

Pattern Definition Specification

Parsing Instructions Specification

Custom Parser XML Specification Template

The basic template for a custom parser XML specification includes five sections. Click on the name of any section for more information.

Section	Description
Parser Name Specification	Name of the parser file
Device Type	The type of device or application associated with the parser
Format Recognizer Specification	Patterns that determine whether an event will be parsed by this parser
Pattern Definition Specification	Defines the parsing patterns that are iterated over by the parsing instructions
Parsing Instructions Specification	Instructions on how to parse events that match the format recognizer patterns

Custom Parser XML Specification Template

Parser Name Specification

This section specifies the name of the parser, which is used only for readability and identifying the device type associated with the parser.

Device or Application Type Specification

This section specifies the device or the application to which this parser applies. The device and application definitions enable FortiSIEM to detect the device and application type for a host from the received events. This is called log-based discovery in FortiSIEM. Once a received event is successfully parsed by this file, a CMDB entry is created with the device and application set from this file. FortiSIEM discovery may further refine the device.

There are two separate subsections for device and application. In each section, vendor, model and version can be specified, but version is not typically needed.

Examples of Specifications for Types of Device and Applications

Hardware Appliances

In this case, the type of event being parsed specifies the device type, for example Cisco IOS, Cisco ASA, etc.

Software Operating Systems that Specify the Device Type

In this case, the type of events being parsed specifies the device type, for example Microsoft Windows etc. In this case the device type section looks like

Applications that Specify Both Device Type and Application

In this case, the  events being parsed specify the device and application types because Microsoft SQL Server can only run on Microsoft Windows OS.

Applications that Specify the Application Type but Not the Device Type

Consider the example of an Oracle database server, which can run on both Windows and Linux operating systems. In this case, the device type is set to Generic but the application is specific. FortiSIEM depends on discovery to identify the device type.

Format Recognizer Specification

In many cases, events associated with a device or application will contain a unique pattern. You can enter a regular expression in the Format Recognizer section of the parser XML file to search for this pattern, which, if found, will then parse the events according to the parser instructions. After the first match, the event source IP to parser file map is cached, and only that parser file is used for all events from that source IP. A notable exception is when events from disparate sources are received via a syslog server, but that case is handled differently.

While not a required part of the parser specification, a format recognizer can speed up event parsing, especially when one parsing pattern file among many pattern files must be chosen. Only one pattern check can determine whether the parsing file must be used or not. The other less efficient option would be to examine patterns in every file. At the same time, the format recognizer must be carefully chosen so that it is not so broad to misclassify events into wrong files, and at the same time, not so narrow that it fails at classifying the right file.

Format Recognizer Syntax

The specification for the format recognizer section is:

In the regexpattern block, a pattern can be directly specified using regex or a previously defined pattern (in the pattern definition section in this file or in the GeneralPatternDefinitions.xml file) can be referenced.

Example Format Recognizers

Cisco IOS

All Cisco IOS events have a %module name pattern.

Cisco ASA

All Cisco ASA events have the pattern ASA-severity-id pattern, for example ASA-5-12345.

Palo Alto Networks Log Parser

In this case, there is no unique keyword, so the entire message structure from the beginning to a specific point in the log must be considered.

Event

<14>May 6 15:51:04 1,2010/05/06 15:51:04,0006C101167,TRAFFIC,start,1,2010/05/06

15:50:58,192.168.28.21,172.16.255.78,::172.16.255.78,172.16.255.78,rule3,,,icmp,vsys1,untrust,untrust,ether net1/1,ethernet1/1,syslog-172.16.20.152,2010/05/06

15:51:04,600,2,0,0,0,0,0×40,icmp,allow,196,196,196,2,2010/05/06 15:50:58,0,any,0

Pattern Definition Specification

In this section of the parser XML specification, you set the regular expression patterns that that FortiSIEM will iterate through to parse the device logs.

You can also write a long pattern definition in multiple lines and indicate their order as shown in this example. The value of the list attribute should be begin in first line and end in last line. If there are more than two lines, the attribute should be set to continue for the other lines.

Parsing Instructions Specification

This section is the heart of the parser, which attempts to recognize patterns in a log message and populate parsed event attributes.

In most cases, parsing involves applying a regular expression to the log, picking up values, and setting them to event attributes. Sometimes the processing is more involved, for example when attributes need to be stored as local variables and compared before populating the event attributes. There are three key components that are used in parsing instructions: Event attributes and variables, inbuilt functions that perform operations on event attributes and variables, and switch and choose branching constructs for logical operations. Values can be collected from both unstructured and structured strings in log messages.

Event Attributes and Variables

Setting an Event Attribute to a Constant

Setting an Event Attribute from Another Variable

Inbuilt Functions

Combining Two or More Strings to Produce a Final String

Normalize MAC Address

Compare Interface Security Level

Convert Hex Number to Decimal Number

Convert TCP/UDP Protocol String to Port Number

Convert Protocol String to Number

Convert Decimal IP to String

Convert Host Name to IP

Add Two Numbers

Divide Two Numbers

Scale Function

Extract Host from Fully Qualified Domain Name

Replace a String Using a Regular Expression

Replace String in String

Resolve DNS Name

Convert to UNIX Time

Trim Attribute

Branching Constructs

Choose Construct

Switch Construct

Collecting Values from Unstructured Strings

Collecting Fields from Structured Strings

Key=Value Structured Data

Value List Structured Data

Event Attributes and Variables

The dictionary of event attributes are defined in FortiSIEM database and any member not belonging to that list is considered a local variable. For readability, local variables should begin with an _, although this is not enforced.

Setting an Event Attribute to a Constant

Setting an Event Attribute from Another Variable

The $ symbol is used to specify the content of a variable. In the example below, attribute hostMACAddr gets the value stored in the local variable

Combining Two or More Strings to Produce a Final String

This is accomplished by using the combineMsgId function. Here _evIdPrefix is the prefix, _evIdSuffix is the suffix, and the output will be s tring1-_evIdPrefix-_evIdSuffix.

Normalize MAC Address

This is accomplished by using the normalizeMAC function. The output will be six groups of two nibbles separated by a colon, for example AA:BB

This is accomplished by using the compIntfSecVal function. This primarily applies to Cisco ASA and PIX firewalls. The results returned are:

This is accomplished by using the convertHexStrToInt function.

Convert TCP/UDP Protocol String to Port Number

This is accomplished by using the convertStrToIntIpPort function.

Convert Protocol String to Number

This is accomplished by the using the convertStrToIntIpProto function.

Convert Decimal IP to String

This is accomplished by using the converIpDecimalToStr function.

Convert Host Name to IP

This is accomplished by using the convertHostNameToIp function.

Add Two Numbers

This is accomplished by using the add function.

Divide Two Numbers

This is accomplished by using the divide function.

Scale Function

This is accomplished by using the scale function.

Extract Host from Fully Qualified Domain Name

This is accomplished by using the extractHostFromFQDN function. If _fqdn` contains a . , get the string before the first .,  otherwise, get the whole string.

Replace a String Using a Regular Expression

This is accomplished by using the replaceStringByRegex function.

Replace String in String

This is accomplished by using the replaceStrInStr function.

Resolve DNS Name

This is accomplished by using the resolveDNSName function, which converts DNS name to IP address.

Convert to UNIX Time

This is accomplished by using the toDateTime function.

Trim Attribute

This is accomplished by using the trimAttribute function. In the example below, it is used to trim the leading and trailing dots in destName.

Branching Constructs

Choose Construct

The format is:

Switch Construct The format is:

Collecting Values from Unstructured Strings

From a string input source, a regex match is applied and variables are set. The variables can be event attributes or local variables. The input will be a local variable or the default raw message variable. The syntax is:

The regexpattern is specified by a list of variables and sub-patterns embedded within a larger pattern. Each variable and sub-pattern pair are enclosed within <>.

Consider an example in which the local variable _body is set to list 130 permitted eigrp 172.16.34.4(Serial1 ) > 172.16.34.3, 1 packet. From this sting we need to set the values to local variables and event attributes.

Value	Set To	Type
130	_aclName	Local Variable
permitted	_action	Local Variable
eigrp	_proto	Local Variable
172.16.34.4	srcIpAddr	Event Attribute
Serial1	srcIntfName	Event Attribute
172.16.34.3	destIpAddr	Event Attribute
1	totPkts	Event Attribute

This is achieved by using this XML. Note that you can use both the collectAndSetAttrByRegex and collectFieldsByRegex functions to collect values from fields.

Collecting Fields from Structured Strings

The are usually two types of structured strings in device logs:

Key=value structured

Value list structured

In each case, two simpler specialized parsing constructs than are provided

Key=Value Structured Data

Certain logs, such as SNMP traps, are structured as Key1 = value1 <separator> Key2 = value2,…. These can be parsed using the col lectAndSetAttrByKeyValuePair XML attribute tag with this syntax.

When a key1 match is found, then the entire string following key1 up to the separatorString is parsed out and stored in the attribute variab leOrEventAttribute1.

As an example, consider this log fragment.

_body =

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.60 = Hex-STRING: 07 D8 06 0B

13 15 00 00 2D 07 00    SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.11.0

= Hex-STRING: 00 16 B6 DB 12 22

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.12.0 = Hex-STRING: 00 21 55

4D 66 B0  SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.13.0 = INTEGER: 36

SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.1.0 = Hex-STRING: 00 1A 1E C0

60 7A  SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.56.0 = INTEGER: 2   SNMPv2-SMI::enterprises.14823.2.3.1.11.1.1.17.0 = STRING:

“00:1a:1e:c0:60:7a”

The corresponding parser fragment is:

After parsing, the attribute values are set:

Value	Attribute
00 16 B6 DB 12 22	srcMACAddr
00 21 55 4D 66 B0	destMacAddr
2	wlanRadioId
00:1a:1e:c0:60:7a	apMac

Value List Structured Data

Certain application logs, such as those from Microsoft IIS, are structured as a list of values with a separator. These can be parsed using the coll ectAndSetAttrByPos XML attribute tag following this syntax.

When the position offset1 is encountered, the subsequent values up to the separatorString is stored in variableOrEventAttribute1.

As an example, consider this log fragment.

The parser fragment is:

<collectAndSetAttrByPos src=”$_body” sep=’  ‘>

<attrPosMap attr=”srvInstName” pos=’1’/>

<attrPosMap attr=”destName” pos=’2’/>

<attrPosMap attr=”relayDevIpAddr” pos=’2’>

<attrPosMap attr=”destIpAddr” pos=’3’/>

<attrPosMap attr=”httpMethod” pos=’4’/>

<attrPosMap attr=”uriStem” pos=’5’/>

<attrPosMap attr=”uriQuery” pos=’6’/>

<attrPosMap attr=”destIpPort” pos=’7’/>

<attrPosMap attr=”user” pos=’8’/>

<attrPosMap attr=”srcIpAddr” pos=’9’/>

<attrPosMap attr=”httpVersion” pos=’10’/>

<attrPosMap attr=”httpUserAgent” pos=’11’/>

<attrPosMap attr=”httpReferrer” pos=’13’/>

<attrPosMap attr=”httpStatusCode” pos=’15’/>

<attrPosMap attr=”httpSubStatusCode” pos=’16’/>

<attrPosMap attr=”httpWin32Status” pos=’17’/>

<attrPosMap attr=”recvBytes” pos=’18’/>

<attrPosMap attr=”sentBytes” pos=’19’/>

<attrPosMap attr=”durationMSec” pos=’20’/>

</collectAndSetAttrByPos>

For structured strings, techniques in this section are more efficient than in the previous section since, the expression is simpler and ONE tag can be used to parse regardless of the order in which the keys or values appear in the string.

 

 

Creating a Custom Parser

Prerequisites

You should have examples of the logs that you want to parse

You should have created any new device/application types, event attribute types, or event types that you want to use in your XML specification

You should already have written the XML specification for your parser

You should have prepared a test event that you can use to validate the parser

Parsers Applied in Order

Parsers are applied in the order they are listed in Admin > Device Support > Parsers, so it is important to add your custom parser to the list in relation to any other parsers that may be applied to your device logs. If you click Fix Order, this will arrange the parsers with system-defined parsers at the top of the list in their original order, and user-defined parsers at the bottom. By sure to click Apply to make sure the change in order is picked up by the back-end module.

Procedure

1. Go to Admin > Device Support > Parsers.
2. Select a parser that is above the location in the list where you want to add your parser, and then click New.
3. Enter a Name for the parser.
4. Select a Device Type to which the parser should apply.

If the device type doesn’t appear in the menu, you should create a new device type

5. Enter a Test Event containing an example of an event that you want to use to validate the parser.
6. Enter the Parser XML.
7. Click Validate.

This will validate the XML.

8. Click Test.

This will send the test event to the parser to make sure it is parsed correctly, and will also test the parsers above and below yours in the list to make sure they continue to parse logs correctly.

9. If the XML for your parser validates and the test event is correctly parsed, select Enable.

If you need to continue working on your parser, you can Save it without selecting Enable.

10. Click Save.
11. Click Apply to have the backend module pick up your parser and begin applying it to device logs.

You should now validate that events are being parsed by creating some activity that will cause a log to be generated, and then run a query against the new device IP address and validate the parsed results.

 

 

 

Deleting or Disabling a Parser

1. Go to Admin > Device Support > Parsers.
2. Select the parser you want to delete or disable.
3. Click Delete or Disable.
4. Click Yes to confirm that you want to delete or disable the parser.

Exporting a Custom Parser

To export a parser, you must also export XML files for the device/app types, event attribute types, event types, and then the parser specification file used by your parser.

1. Go to Admin > Device Support > Device/App Types.
2. Select the device/application types used in your parser, and then click Export.
3. Go to Admin > Device Support > Event Attribute Types.
4. Select the event attribute types used in your parser, and then click Export.
5. Go to Admin > Device Support > Event Types.
6. Select the event types used in your parser, and then click Export.
7. Go to Admin > Device Support > Parsers.
8. Select the parser specification for your parser, and then click Export.

Importing a Custom Parser

Importing a custom parser involves importing four XML files: the XML files containing any device/app types, event attribute types, or event types that you have created for this parser, followed by the parser specification XML file.

1. For each device/app type, event attribute type, or event type XML file that is required for your parser, go to the appropriate tab in Admin > Device Support, and then click Import.
2. Browse to the location of your XML file, and then click Upload.
3. Go to Admin > Device Support > Parsers, and then click Import.
4. Browse to the location of your parser specification XML file, and then click Upload.
5. Follow the instruction in Creating a Custom Parser to validate your XML and test the parser, and to make sure it appears in the correct position in the list of parsers.

Parser Examples

Cisco IOS Syslog Parser

Cisco IOS Syslog Parser

Add Device Type

Create a file CiscoIOSParser.xml with this content.

Create the Parser Specification and Add Local Patterns

Create the parser XML file with this content, and add the pattern definition patCiscoIOSMod for detecting IOS modules such as SEC.

 

Define the Format Recognizer

Add this format recognizer for detecting %SEC-6-IPACCESSLOGP, which is a signature of Cisco IOS syslog messages.

Parse the Syslog Header

A syslog message consists of a syslog header, and a body. For better organization, we first parse the syslog header and event type. Subsequent code will include event type specific parsing, which is why event type is extracted in this step. In this example, the header is in boldface.

<190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP: list testlog permitted tcp 192.168.20.33(3438) -> 69.147.86.184(80), 1 packet

The XML code for parsing the header does the following:

1. Matches the pattern <190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP:
2. Sets the eventType attribute to IOS-SEC- IPACCESSLOGP.
3. Sets deviceTime.
4. Sets event severity (1-7 scale in Cisco IOS, 1=> most severe, to normalized 1-10 scale in FortiSIEM where 10=>most severe)
5. Saves the event list testlog permitted tcp 192.168.20.33(3438) -> 69.147.86.184(80), 1 packet in a temporary variable _body.

Note that the patterns gPatSyslogPRI, gPatMon, gPatDay, gPatTime, gPatInt, gPatmesgBody are global patterns that are defined in the GeneralPatternDefinitions.xml file:

This parser file XML fragment for parsing the example syslog message looks like this:

 

Parse the Syslog Body

The parsing is done on an eventType by eventType basis, since the formats are eventType specific. Parsing the syslog body involves three steps:

1. Parsing the action string. Based on the action staring value (permit or denied), modify the eventType by appending the action string value at the end, and also modify the eventSeverity
2. Parsing the protocol, source and destination IP, port, and totalPackets.
3. Converting the protocol string to a protocol integer.

 

Final Parser

</patternDefinitions>

<parsingInstructions>

<!—parse header –>

<collectFieldsByRegex src=”$_rawmsg”>

 

<regex><![CDATA[<:gPatSyslogPRI>?<:gPatMon>\s+<:gPatDay>\s+<:gPatTime>

%<_evIdPrefix:patCiscoIOSMod>-<_severity:gPatInt>-<_evIdSuffix:patStrEnd

Colon>: <_body:gPatMesgBody>]]></regex>

</collectFieldsByRegex>

<setEventAttribute attr=”eventType”>combineMsgId(“IOS-“,

$_evIdPrefix, “-“, $_evIdSuffix)</setEventAttribute>

<choose>

<when test=’$_severity IN “6, 7″‘>             <setEventAttribute attr=”eventSeverity”>1</setEventAttribute>         </when>

<when test=’$_severity = “1”‘>            <setEventAttribute attr=”eventSeverity”>10</setEventAttribute>         </when>

<when test=’$_severity = “2”‘>

<setEventAttribute attr=”eventSeverity”>8</setEventAttribute>

</when>

<when test=’$_severity IN “3, 4″‘>

<setEventAttribute attr=”eventSeverity”>5</setEventAttribute>

</when>

<when test=’$_severity = “5”‘>

<setEventAttribute attr=”eventSeverity”>2</setEventAttribute>

</when>

</choose>

<!—parse body –>

<choose>

<when test=’$eventType IN “IOS-SEC-IPACCESSLOGP,

IOS-SEC-IPACCESSLOGDP, IOS-SEC-IPACCESSLOGRP”‘>

<collectAndSetAttrByRegex src=”$_body”>

<regex><![CDATA[list

<_aclName:gPatStr>\s+<_action:gPatWord>\s+<_proto:gPatWord>\s+<srcIpAddr

:gPatIpV4Dot>\(<srcIpPort:gPatInt>\)<:gPatMesgBody>->\s+<destIpAddr:gPat

IpV4Dot>\(<destIpPort:gPatInt>\),\s+<totPkts:gPatInt> <:gPatMesgBody>]]>

</regex>           </collectAndSetAttrByRegex>

<choose>

<when test=’$_action = “permitted”‘>                  <setEventAttribute

attr=”eventType”>combineMsgId(“IOS-“, $_evIdPrefix, “-“, $_evIdSuffix, “-PERMITTED”)</setEventAttribute>           <setEventAttribute attr=”eventSeverity”>1</setEventAttribute>

</when>

<when test=’$_action = “denied”‘>

<setEventAttribute attr=”eventType”>combineMsgId(“IOS-“, $_evIdPrefix, “-“, $_evIdSuffix, “-DENIED”)</setEventAttribute>                  <setEventAttribute attr=”eventSeverity”>3</setEventAttribute>               </when>

</choose>           <setEventAttribute attr=”ipProto”>convertStrToIntIpProto($_proto)</setEventAttribute>

Parsed Output Input syslog:

<190>91809: Jan 9 02:38:47.872: %SEC-6-IPACCESSLOGP: list testlog permitted tcp 192.168.20.33(3438) ->

69.147.86.184(80), 1 packet

Parsed fields:

1. phRecvTime: the time at which the event was received by FortiSIEM
2. phDeviceTime: Jan 9 02:38:47 2010
3. eventType: SEC-IPACCESSLOGP-PERMITTED
4. eventSeverity: 3
5. eventSeverityCategory: LOW
6. aclName: testlog
7. ipProto: 6
8. srcIpAddr: 192.168.20.33
9. destIpAddr: 69.147.86.184
10. srcIpPort: 3438
11. destIpPort: 80
12. totPkts: 1

The master list of event attributes supported by FortiSIEM is here