Viewing Incident Notification History

There are two ways you can view the notification history for an incident.

1. In the Incident Notification Status column of the Incident Dashboard.
2. Click on an incident in the Incident Name column of the Incident Dashboard, and then select View Notification History from the Option s

Incident XML File Format

This topic includes an example of the XML file that is generated for incidents, and descriptions of its contents.

Example Incident XML File

XML Tag and Attribute Definitions

Example Incident XML File

<?xml version=”1.0″ encoding=”UTF-8″ ?> <incident incidentId=”5672″ ruleType=”PH_RULE_AUTO_SRVC_DOWN” severity=”10″ repeatCount=”1″ organization=”Super” status=”Cleared”>   <name>Auto Service Stopped</name>   <description>Detects that an automatically running service stopped.

Currently this works for windows servers and is detected via

WMI.</description>

<displayTime>Fri Jun 29 15:51:10 PDT 2012</displayTime>

<incidentSource>

</incidentSource>

<incidentTarget>

<entry attribute=”hostIpAddr” name=”Host IP”>172.16.10.15</entry>

<entry attribute=”hostName” name=”Host Name”>QA-V-WIN03-ADS</entry>

</incidentTarget>

<incidentDetails>

<entry attribute=”serviceName” name=”OS Service

Name”>Spooler</entry>

<entry attribute=”servicePath” name=”OS Service

Path”>C:\WINDOWS\system32\spoolsv.exe</entry>

</incidentDetails>

<affectedBizSrvc>Auth Service</affectedBizSrvc>

<identityLocation>

</identityLocation>  <rawEvents>

[SrvcDown]

[PH_DEV_MON_AUTO_SVC_START_TO_STOP]:[eventSeverity]=PHL_INFO,[fileName]= phPerfJob.cpp,[lineNumber]=6005,[hostName]=QA-V-WIN03-ADS,[hostIpAddr]=1 72.16.10.15,[serviceName]=Spooler,[servicePath]=C:\WINDOWS\system32\spoo lsv.exe,[serviceDesc]=Manages all local and network print queues and controls all printing jobs. If this service is stopped, printing on the local machine will be unavailable. If this service is disabled, any services that explicitly depend on it will fail to start.,[phLogDetail]=  </rawEvents>

</incident>

XML Tag and Attribute Definitions

Setting Scripts as Notification Actions

One of the actions you can specify for an incident notification is to execute a script. For example, suppose you are monitoring Windows services that are in Auto mode, and you have rules that will trigger an incident if one of those services is stopped. The notification action for that incident can include the running of a script by AccelOps that will re-start the service, as shown in the example scripts in this topic.

How Script Notification Actions are Processed

1. When you specify the notification action as a script, you must provide the full path to the script in the notification policy settings, for example /tmp/Myscript.py.
2. You must write the script so it expects the incident XML file to be located in the same directory as the script, for example /tmp if the script location is /tmp/Myscript.py.
3. When a notification policy is triggered by an incident, the policy actions are handled in sequential order, so if there are multiple script actions, the first one will be processed before the second one.
4. When the script action is processed, the AccelOps notification module will first generate an incident XML file and put it in the same directory as the script. AccelOps will then call the script with the XML file name as an argument.
5. When the script returns, the incident XML file that was created by AccelOps is deleted, so there is no confusion with the next script action which involves a new incident XML file and is processed only after the previous script action is complete.

Setting a Script Notification Action

1. Log in to your Supervisor node.
2. Go to Analytics > Incident Notification Policy.
3. Select the notification policy where you want to add the script action.
4. Under Actions, next to the Methods table, click … .
5. Under Run Script, click Add.
6. For Script Name, enter the name of the script and the absolute directory path to it.
7. Click OK.

Example of a Windows Restart Script as a Notification Action

This topic provides an example of a script that could be used as a notification action, following the example of re-starting a Windows service that has stopped an triggered an incident as described in Setting Scripts as Notification Actions.

This example requires two scripts: one located on the Windows server that hosts the service, and a script on the AccelOps Supervisor host machine that will be triggered by the incident notification and will execute the Windows server script.

Windows Script

AccelOps Script

Windows Script

1. Create a script named installWinexeSvc.bat for starting the remote winexe provider service.

This script, restartWinService.py, reads the incident XML file, parses out the target IP and stopped service, and issues a winexe command to restart the service.

#!/usr/bin/python importos, re, sys, time importxml.dom.minidom iflen(sys.argv) != 2:

print “Usage: parseTargetIP.py incident.xml”    exit() else:

fileName = sys.argv[1] print “parsing incident xml file : “, fileName #os.system(“cp “+ fileName + ” “+ fileName + “.txt”) # /incident/incidentTarget/entry[@attribute=’hostIpAddr’] doc = xml.dom.minidom.parse(fileName) nodes = doc.getElementsByTagName(‘incidentTarget’) ifnodes.length < 1:

print “no incident Target found!” else:

targeNode = nodes[0] targetIP = “” fornode in targeNode.childNodes :    ifnode.nodeType == node.ELEMENT_NODE:       ifnode.getAttribute(“attribute”) == “hostIpAddr”:

targetIP = node.firstChild.data iftargetIP == “”:

print “no incident target found!” # trim IP, e.g. 10.1.20.189(SH-Quidway-SW1) targetIP = re.sub(r’\(.+\)’, “”, targetIP) print “restart service for target IP: “, targetIP # parse process name nodes = doc.getElementsByTagName(‘incidentDetails’) ifnodes.length < 1:

print “no incidentDetails found!” else:

targeNode = nodes[0] fornode in targeNode.childNodes :    ifnode.nodeType == node.ELEMENT_NODE:       ifnode.getAttribute(“attribute”) == “serviceName”:

targetService = node.firstChild.data ########################################################################

######################## # NOTE:  You need to replace the user and password with an account on your Windows server that # #        has permissions to run thiswindows command.

# ########################################################################

######################## # stop the service stopCmd = “winexe –user Administrator –password ProspectHills! //”+ targetIP + ” ‘sc stop “+ targetService + “‘” ret = os.system(stopCmd)

print “stop service with return code ,”, ret print “waiting service stop” time.sleep(10) ########################################################################

######################## # NOTE:  You need to replace the user and password with an account on your Windows server that # #        has permissions to run thiswindows command.

#

########################################################################

######################## ## start the service startCmd = “winexe –user Administrator –password ProspectHills! //”+ targetIP + ” ‘sc start “+ targetService + “‘”

Sending Email and SMS Notifications for Incidents

When you set actions for an incident notification, one option is to send an email or SMS message to groups or individuals, and you also have an option to specify a template that should be used in the email.

Prerequisites

Procedure

Related Links

Prerequisites

Make sure the email gateway has been configured for your deployment.

You should also have set up any email templates that you want to use for notifications.

Procedure

1. Log in to your Supervisor node.
2. Go to Analytics > Incident Notification Policy.
3. Select the policy that you want to set up the email or SMS notification for.
4. Under Actions, next to the email/sms notification table, click … .
5. For multi-tenant deployments, select the Organization that contains the individuals or groups you want notified.

Under Folders, you will see the user groups for that organization listed.

6. In the Folders pane, select a group.

In the Items pane, you will see a list of users for that group.

7. Select a group and click Folder >> to add a group to the Notification Actions list, or select individual users and click Items >>.
8. Under Notification Actions, select the Method, Email or SMS, that you want to use sending the notification.
9. Select an Email Template if you are sending an email notification. If you leave this blank, the default email template will be used.

Related Links

Setting Up the Email Gateway

Setting Scripts as Notification Actions

Customizing Email Templates for Notifications

Email templates for incident notifications are based on incident variables that you put into the subject and body of the template, which are then populated with the actual attribute values in the incident.

Incident Attribute Variables

Example Email Template

Template

Generated Email

Creating an Email Template

Incident Attribute Variables

These are the incident attribute variables you can use for your email template.

$organization

$status

$hostName

$incidentId

$incidentTime

$firstSeenTime

$lastSeenTime

$incident_severityCat

$incident_severity

$incident_incidentCount

$ruleName

$ruleDescription

$incident_source

$incident_target

$incident_detail

$affectedBizService

Example Email Template

This example first shows a template with the incident attribute variables, and then an email based on this template with the variables populated from an incident.

Template

Email Subject:

$ruleName was triggered at $incidentTime

Email Body:

The host, $incident_target, was being scanned by $incident_source starting at $firstSeenTime and ending at $lastSeenTime. There were $incident_incidentCount hits.

Please investigate and report as necessary.

Generated Email

Subject: Server Memory Warning was triggered at Jan 10 22:43 UTC

Body: The host, Host IP: 192.168.1.23 Host Name: QA-V-WIN03-ORCL, was being scanned by 10.1.1.1 starting at Jan 10 22:05 UTC and ending at Jan 10 22:11 UTC. There were 2 hits.

Please investigate and report as necessary.

Creating an Email Template

1. Log in to your Supervisor node.
2. Go to Admin > General Settings > Incident Email Templates.
3. Click Add.
4. For multi-tenant deployments, select the organization for which you are creating the email template.
5. Enter a Name for the template.
6. Enter the Email Subject and Email Body.

You can select attribute variables from the Insert Content menu to enter into your template, rather than having to type them out by hand.

7. Click OK.

be used. To set an email template as default, select the template in the list on the Incident Email Templates page, and then click Set as Default. For multi-tenant deployments, to select a template as default for an organization, first select the organization, then set the default email template for that organization.

Creating an Incident Notification Policy

Prerequisites

Make sure you have enabled the settings for sending email or other notification actions as described in Setting Up Routing Information for Reports and Incident Notifications.

You should read the introductory topic on incident notifications to understand how policy conditions are processed..

Procedure

1. Log in to your Supervisor node.
2. Go to Analytics > Incident Notification Policy.
3. Click New.
4. Select the Incident Severity.

Only incidents matching the severity level you select will trigger a notification.

5. For Rules, click … and select the rule or rules you want to trigger this notification.
6. Set a Time Range during which this notification will be in effect.

Notifications will be sent only if an incident occurs during the time range you set here.

7. For Affected Items, click … and use the CMDB Browser to select the devices or applications for which this policy should apply.

Instead of individual devices or groups, you can apply the notification policy to an IP address or range by clicking Add under IP/Range. You can also select a group, and then select the Not option to explicitly exclude that group of applications or devices from the notification policy.

8. For multi-tenant deployments, select the Organizations to which the notification policy should apply.

Notifications will be sent only if the triggering incidents affect the selected organization.

9. Select the Actions to take when the notification is triggered.

See the topics under Sending Email and SMS Notifications for Incidents, Creating Tickets In FortiSIEM In-built Ticketing System, Creatin g Inbound Policies for Updating Ticket Status from External Ticketing Systems, and Setting Scripts as Notification Actions for more information about notification actions.

10. Enter any Comments about the policy.
11. When you are finished creating the notification policy, select Enabled to make it active in your deployment.
12. Click Save.

Device Risk View of Incidents

Viewing Devices Sorted By Risk

1. Go to Incident tab
2. Set Group By to Host Risk Score.
3. Left pane shows Devices Sorted By Risk
4. Right pane shows incidents for the device selected in left panel

Calendar View of Incidents

The calendar view of incidents provides a summary view of the number of incidents that have occurred on a calendar day, grouped by severity. Clicking a group loads a summary of those incidents.

This screenshot shows the calendar view of incidents for the month of February 2015.

Fishbone View of Incidents

The fishbone view of incidents presents a view of networks and devices in those networks, along with the incidents triggered for those devices over the last 24. This view is derived from the Network Segments in the CMDB, with the devices associated with those segments overlaid. The numbers and colors for each device indicate the number and severity of incidents associated with that device.

Clicking on an incident  number will show you a summary of those incidents. Clicking on Last Seen, First Seen, Incident Name, or Incid ent Details in that summary will let you select Incident Details to view more information. Clicking on any IP addresses associated with the device will open a contextual menu that will let you find out more information about that device.

Clicking on an IP number or hostname in the fishbone view will let you view the Quick Info for that device, or you can select Topology to view it within the context of your network topology.

Hovering your mouse cursor over a device or incident number will show you the IP address and host name for that device, as well as the type of device.

This screenshot shows an example fishbone view of network segments, devices, and associated incidents.

Incident Notifications

The sending of notifications when an incident occurs is handled by Notification Policies, which you can see listed in the Analytics > Incident Notification Policies page. Instead of having notifications set for each rule, you can create a policy and have it apply to multiple rules.

When viewing the notification policies, think of  the columns on the page as representing a series of “If … and … then” statements that lead to the notification action. For example, you could read the table columns as a sentence:

“IF Incident Severity is X1 AND Rule is X2 AND Time Range is X3 AND Affected Items includes X4 AND Affected Organizations is X5, THEN take the actions specified in the ACTION column.”

When AccelOps evaluates whether a notification action should be triggered based on the notification conditions, it evaluates all notification policies, and will trigger the actions of all policies that meet the condition, instead of just the first policy that meets the conditions. This means that the order of policies in the list doesn’t matter, and that you can write policies with overlapping conditions that could also, for example, include different actions.

See also the topics under Incident Notification for more information about the methods that are available for sending notifications from AccelOps, including the AccelOps API.

Creating an Incident Notification Policy

Sending Email and SMS Notifications for Incidents

Customizing Email Templates for Notifications

Setting Scripts as Notification Actions

Example of a Windows Restart Script as a Notification Action Incident XML File Format

Viewing Incident Notification History

Using Group By Attributes to View Incidents

The Incident Dashboard presents a view of all incidents based on the filter conditions you select. However, there may be situations in which you want to view incidents grouped on incident attributes like Incident Source, Incident Target, Severity, or Incident Name. Once incidents are grouped by their attributes, you can view Incident Details for the entire group.

1. Log in to your Supervisor node.
2. Go to Incidents.
3. In the Group By menu, select the attributes you want to use to group the incidents, and then click Refresh.

The Incident Dashboard will refresh and display incidents grouped according to the attributes you selected, with a COUNT(Matched Events) column that indicates how many incidents are in each group.

4. Select a group and then click on it to open the Options
5. In the Options menu, select Show Incident Details for This Group.

The Incident Dashboard will refresh to show all incidents in the selected incident group, and you can use the Contextual Menus to find out more information about them.

Viewing and Searching Incidents

The Incident Dashboard displays incident information for your IT infrastructure based on the filter conditions you set. You can also view incidents grouped by incident attributes, use values in incident attributes to refine your searches, view information about rules that triggered incidents, and use incident information to create rule exceptions and event dropping rules.

List View of Incidents

Searching for Incidents by Incident Attributes

Using Group By Attributes to View Incidents

Device Risk View of Incidents

Calendar View of Incidents

Fishbone View of Incidents

List View of Incidents

There are two ways you can view the incidents that are occurring in your IT infrastructure.

The Incidents tab, shown in the screenshot for this topic, where you can view incidents and incident details

Dashboard > Incident Dashboard, which includes the same incident summary and user interface controls found in the Incidents tab, but which also provides other views of incidents, including a fishbone view of incidents in your infrastructure, a topology view with the number and severity of incidents overlaid on devices, a calendar view, and a location view that includes both a summary view of incident source and target IP locations and a map view, along with the number and severity of incidents for that location overlaid on the map.

In both locations you can filter the incidents in the dashboard, find out more information about sources and targets of incidents, customize the dashboard layout, and manage the rules associated with incidents.

Incident Attributes

Incident Dashboard User Interface Controls

Incident Dashboard Filter Controls

Incident Management Controls

Contextual Menus

Incident Details

Incident Details

Triggered Events

Related Incidents

Incident Attributes

An Incident has the following attributes.

Incident Dashboard User Interface Controls

This screenshot shows the Incidents tab with the major user interface controls outlined in red.

Incident Dashboard Filter Controls

The filter controls let you control which incidents are shown in the dashboard.

Incident Management Controls

Contextual Menus

Clicking on an item within a column of the incident summary will open a contextual menu, with options depending on whether the incident attribute you selected includes an IP address (Source IP or Target IP, for example), or some other kind of incident attribute. Shared between both menus are an Add to Filter option, which enables you to select a result attribute and add it to the Filter By conditions. Both menus also include most of the same options available in the Incident Management controls to edit and add exceptions to rules. The IP address contextual menu provides options to view more information about the associated device, with many of the same options you would find in the Analysis menu used in search summary dashboards.

This screenshot shows the IP contextual menu open after selecting an IP address in the Incident Source column of the Incidents tab.

Incident Details

The Incident Details pane at the bottom of the Incidents Dashboard provides you with information about a selected incident in three areas: Incide nt Details, Triggered Events, and Related Incidents.

Incident Details

The Incident Details include the ID of the incident, specific details about the event that triggered the incident, and the definition of the rule associated with the incident.

Triggered Events

The list of events that triggered the incident. For columns containing an event type, or host or  IP information, click on an item to open a contextual menu and view more information about it.

Related Incidents

Use this menu to view related incidents based on the Source, Target, Rule Name, or Reporting IP associated with the selected incident.

Searching for Incidents by Incident Attributes

As your review incidents in your dashboard, you may want to build searches based on attributes from selected incidents. For example, you may want to use the value for the Incident Target attribute in an incident as a filter condition to find similar or related incidents, and then add more conditions based on the results of that search.

1. Log in to your Supervisor node.
2. Go to Incidents.
3. In the Incident Dashboard, select an incident.
4. Click on the attribute value for the selected incident that you want to add to the Filter By condition to open the Options menu, and then select Add to Filter.

The type of search will change to Advanced, and the attribute value you selected will be added to the Filter By conditions.

5. Click in the Filter By Conditions field to open the Conditions Builder and add other incident attributes.
6. Click Refresh when you’re done creating filter conditions to see the results.