System Performance Estimates and Recommendations for Large Scale Deployments

This topic includes estimates and recommendations for storage capacity, disk performance, and network throughput for optimum performance of FortiSIEM deployments processing over 10,000 EPS.

In general, event ingestion at high EPS requires lower storage IOPS than for queries simply because queries need to scan higher volumes of data that has accumulated over time. For example, at 20,000 EPS, you have 86,400 times more data in a day than in one second, so a query such as ‘Top Event types by count for the past 1 day’ will need to scan 20,000 x 86,400 = ~ 1.72 billion events. Therefore, it is important to size your FortiSIEM cluster to handle your query and report requirements first, which will also handle event ingestion very well. These are the top 3 things to do for acceptable FortiSIEM query performance:

1. Add more worker nodes, higher than what is required for event ingestion alone
2. 10Gbps network on NFS server is a must, and if feasible on Supervisor and Worker nodes as well
3. SSD Caching on NFS server – The size of the SSD should be as close to the size required to cache hot data. In typical customer scenarios, the last 1 month data can be considered hot data because monthly reports are quite commonly run

 

Schedule frequently run reports into the dashboard

If you have frequently run ranking reports that have group-by criteria (as opposed to raw message based reports), you can add such reports into a custom dashboard so that FortiSIEM schedules to run these reports in inline mode. Such reports compute their results in streaming manner as event data is processed in real-time. Such reports do not put any burden on the storage IOPS because they read very little data from the EventDB. Note that raw message reports (no group-by) are always computed directly from EventDB

 

An example scenario is presented at the end of this guide.

System Performance Component	Estimates and Recommendations
Event Storage Capacity	Storage capacity estimates are based on an average event size of 64 compressed bytes x EPS (events per section). Browser Support and Hardware Requirements includes a table with storage capacity requirements for up to 10,000 EPS.
Root Disk IOPS	Standard hard disk IOPS
CMDB Disk IOPS	1000 IOPS or more. Lab testing for EC2 scalability used 2000 IOPS.
SVN Disk IOPS	1000 IOPS
EventDB IOPS for Event Ingestion	1000 IOPS for 100K EPS (minimum)
EventDB Read IOPS for Queries	As high as feasible to improve query performance (use SSD caching on NFS server when feasible). In EC2 scalability testing, 2000 read IOPS while ingesting 100K EPS using one supervisor and two workers produced these results: Index Query – No filter, display COUNT(Matched Events), group-by event type for 24 hours 1.  Total Events processed = 2,594,816,711 (2.59 billion events) 2.  Average events per second scanned by Query (QEPS) = 1.02 million QEPS 3.  Average Query Runtime = 2543 seconds (~ 42 minutes) Raw Event Log Query – Same as Index Query with filter Raw Event Log contains ‘e’ 1.           Total Events processed = 350,914,385 (350 million events) 2.           Average events per second scanned by Query (QEPS) = 179,909 EPS (179k QEPS) 3.  Average Query Runtime = 1950 seconds (~ 33 minutes)

Network Throughput	Recommend 10Gbps network between Supervisor, Workers, and NFS server. Using VMXNet3 Adapter for VMware To achieve the best network throughput in VMware environments, delete the E1000 network adapter and add one that uses VMXNet3 for theeth0/eth1 network configuration. VMXNet3 adapter supports 10Gbps networking between VMs on the same host as well as across hosts, though you must also have a 10Gbps physical network adapter to achieve that level of throughput across hosts. You may need to upgrade the virtual hardware version (VM Ware KB 1003746) in order to have the ability to use VMXNet3. More details on different types of VMWare network adapters is available in VMWare KB 1001805 Achieving 10Gbps on AWS EC2 To achieve 10Gbps in the AWS EC2 environment, you will need to: 1.  Deploy FortiSIEM Super, Workers, and NFS server on 8xlarge class of instances (for example, c3.8xlarge ). Refer to EC2 Instance Types for available types, and look for instance types with 10 Gigabit noted next to them. 2.  You will need to use the HVM image for both the FortiSIEM image and NFS server image that supports enhanc ed networking. 3.  Supervisor, Workers, and NFS Server must be placed under the same AWS EC2 placement group within an AWS VPC.
Network Interfaces	FortiSIEM recommends the use of separate network interfaces for event ingestion/GUI access and storage data to NFS
Number of Workers	6000 EPS per worker for event ingestion. More worker nodes for query performance. See example below.

 

Example:

An MSP customer has 12,000 EPS across all their customers. Each event takes up 64 bytes on average in compressed form in the EventDB.

These calculations are just extrapolations based on a test on EC2. Actual results may vary from this because of differences in hardware, event data, types of queries. Therefore, it is recommended that customers do a pilot evaluation using production data either on-premise or on AWS before arriving at an exact number of worker nodes

Installation

Additional Information in the Help Center

You can find additional information about installation, upgrades, and license management for your AccelOps deployment in the Installati on, Upgrades, and Licenses section of the Help Center maintained by AccelOps Support.

The topics in this section are intended to guide you through the basic process of setting up and configuring your AccelOps deployment. This includes downloading and installing the AccelOps OVA image, using your hypervisor virtual machine manager to configure the hardware settings for your AccelOps node, setting up basic configurations on your Supervisor node, and registering your Supervisor and other nodes. Setting up IT infrastructure monitoring, including device discovery, monitoring configuration, setting up business services, is covered in under the section Confi guring Your AccelOps Platform.

What You Need to Know before You Begin Installation What Kind of Deployment Will You Set Up?

Who Will Install and Configure AccelOps?

What Information Do You Need to Get Started? The Basic Installation Process

What You Need to Know before You Begin Installation

What Kind of Deployment Will You Set Up?

Before beginning installation you should have determined the exact deployment configuration you will follow, as described in the topics under Dep loyment Options. Note that many deployment options have particular hardware requirements. For example, if you intend to use an NFS server for a cluster deployment, or if want to use Visual Analytics, you will need to make sure that you have the necessary hardware and network components in place. We strongly recommend that you read through all the installation topics for your deployment configuration before you begin.

Who Will Install and Configure AccelOps?

These topics assume that you have the basic system administration skills required to install AccelOps, and that you are already familiar with the use of hypervisors such as VMware ESX or, if you are setting up a Cloud deployment, that you are already familiar with Cloud environments such as Amazon Web Services.

What Information Do You Need to Get Started?

You will need to have administrator-level permissions on the host where you will download and install AccelOps, and you will also need to have username and password associated with your AccelOps license. If you intend to use NFS storage for event data, you will also need to have set up an NFS server prior to installation.

The Basic Installation Process

The installation process for any AccelOps deployment consists of a few steps:

Import the AccelOps virtual appliance into a hypervisor or Amazon Web Services environment

Edit the virtual appliance hardware settings

Start and configure the virtual appliance from the hypervisor console

Register the virtual appliance

Topics in this section will take you through the specific installation and configuration instructions for the most popular hypervisors and deployment configurations.

System Performance Estimates and Recommendations for Large Scale Deployments

Browser Support and Hardware Requirements

Information Prerequisites for All FortiSIEM Installations

Hypervisor Installations

Installing in Amazon Web Services (AWS)

Determining the Storage Type for EventDB in AWS

Configuring Local Storage in AWS for EventDB

Setting Up Supervisor, Worker and Collector Nodes in AWS

Setting Up AWS Instances

Creating VPC-based Elastic IPs for Supervisor and Worker Nodes in AWS Configuring the Supervisor and Worker Nodes in AWS

Registering the Collector to the Supervisor in AWS

Setting up a Network Bridge for Installing AccelOps in KVM

Importing the Supervisor, Collector, or Worker Image into KVM Configuring Supervisor Hardware Settings in KVM

Importing a Supervisor, Collector, or Worker Image into Microsoft Hyper-V

Setting the Network Time Protocol (NTP) for ESX

Installing a Supervisor, Worker, or Collector Node in ESX

Importing the Supervisor, Collector, or Worker Image into the ESX Server

Editing the Supervisor, Collector, or Worker Hardware Settings

Setting Local Storage for the Supervisor

Troubleshooting Tips for Supervisor Installations

Configuring the Supervisor, Worker, or Collector from the VM Console

ISO Installation

Installing a Collector on Bare Metal Hardware

General Installation

Configuring Worker Settings

Registering the Supervisor

Registering the Worker

Registering the Collector to the Supervisor

Using NFS Storage with AccelOps

Configuring NFS Storage for VMware ESX Server

Using NFS Storage with Amazon Web Services

Setting Up NFS Storage in AWS

Setting Up Snapshots of EBS Volumes that Host EventDB and CMDB in AWS

Moving CMDB to a separate Database Host

FortiSIEM Windows Agent and Agent Manager Install

FortiSIEM Windows Agent Pre-installation Notes

Installing FortiSIEM Windows Agent Manager

Installing FortiSIEM Windows Agent

 

Matrix of Multi-Tenancy Deployment Configuration Options

This matrix shows the components required for the each multi-tenancy deployment option.

Deployment Option	Supervisor Node	Worker Node	Collector Node	NFS Server	Report Server	Visual Analytics Server	Description
Single Multi-Tenant Supervisor Node	x						This is the most basic single site multi-tenant deployment, primarily suitable for hosting providers. Organizations are created by splitting up the IP address space.
Multi-Tenant Supervisor Node Collectors with	x		x				This is a service provider deployment covering multiple sites. Data collection is simplified by deploying a collector for the satellite sites. You can add organizations by assigning a collector to an organization, or by splitting up the IP address space.
Multi-Tenant Cluster	x	x		x			This is a scalable service provider deployment suitable for deployments with large compute and storage needs. An NFS Server is required in the data sharing architecture between Supervisor and Worker nodes. Organi zations are created by splitting up the IP address space.
Multi-Tenant Cluster with Collectors	x	x	x	x			This deployment adds collectors to the configuration and is the most comprehensive service provider deployment. You can add organizations by assigning a collector to an organization, or by splitting up the IP address space.
Multi-Tenant Supervisor Node with Tableau Visual Analytics	x				x	x	This is the most basic single site multi-tenant deployment, with added capability for Visual Analytics with Tableau.
Multi-Tenant Supervisor Node with Collectors and T ableau Visual Analytics	x		x		x	x	This is a service provider deployment covering multiple sites, with added capability for Visual Analytics with Tableau. Data collection is simplified by deploying a collector for the satellite sites.
Multi-Tenant Cluster with T ableau Visual Analytics	x	x		x	x	x	This is a scalable service provider deployment ,with added capability for Visual Analytics with Tableau. An NFS Server is required in the data sharing architecture between Supervisor and Worker nodes.
Multi-Tenant Cluster with Collectors and Tableau Visual Analytics	x	x	x	x	x	x	This deployment adds collectors to the configuration and is the most comprehensive service provider deployment, with added capability for Visual Analytics with Tableau.

 

 

Multi-Tenant Deployment Options for Managed Service Providers or Multiple Organizations

While a common use case for FortiSIEM is the monitoring of IT infrastructure for a single enterprise, Managed Service Providers (MSPs) and large enterprises with multiple organizations can also use FortiSIEM to monitor IT infrastructure at the customer or organization level, either by splitting IP addresses to correspond to the customer or organization, or by deploying Collectors for each customer or organization and managing the monitoring and analysis of their data from a centralized Supervisor.

Standalone Supervisor Deployment for Multi-Tenancy

Supervisor and Worker Cluster Deployment for Multi-Tenancy

Supervisor with Collectors Deployment for Multi-Tenancy

Matrix of Multi-Tenancy Deployment Configuration Options

Standalone Supervisor Deployment for Multi-Tenancy

FortiSIEM allows users to create organizations, to and manage the entire IT infrastructure monitoring life cycle from data collection, storage, analytics and alerting for an organization that organization as a separate entity from other organizations. There are several use cases for this this multi-tenant model.

Hosting service providers that host multiple customers in their own data center

Managed service providers that manage a customer’s data centers from their own data center

Large enterprises that want to manage separate parts of the organization as individual customers

The simplest multi-tenancy deployment involves a single Supervisor, with organizations defined through the splitting of IP address ranges. For example:

Page

Supervisor and Worker Cluster Deployment for Multi-Tenancy

As the number of monitored devices, or the analyzed event rate, grows, one Supervisor may not be able to handle the load. In that case, you can deploy a cluster of Supervisor and Worker virtual appliances that share data over NFS. In a cluster deployment, the Supervisor and Worker nodes have specific functions:

10.1.2.0/24 = Customer 2

During the discovery process, the AccelOps Supervisor node will tag a device with the correct customer ID based on the IP address definition.

 

Supervisor with Collectors Deployment for Enterprises

There are two cases where a single Supervisor may not be enough for your deployment.

There are monitored devices behind a firewall that will not allow monitoring protocols like Windows Management Instrumention (WMI) to be used from the Supervisor

The Supervisor can only reach the monitored devices through a high latency network like a Wide Area Network (WAN), in which case monitoring like protocols like Simple Network Management Protocol (SNMP) or WMI do not work well

In these cases you can deploy Collectors to monitor the devices, and they will communicate to the Supervisor over HTTP(S). The Collectors communicate with the devices, collect and parse events and logs, compress them, and then send them to the Supervisor for monitoring and analysis. Collectors also can buffer the events, in case transmission to the Supervisor is interrupted. As shown in the diagrams, you can use Collectors in a deployment with a single Supervisor, or in a deployment that also includes Workers.

An AccelOps deployment with a single Supervisor and Collectors

An AccelOps deployment using a Single Supervisor + 2 Workers + 2 Collectors

Matrix of Enterprise Deployment Configuration Options

This matrix shows the components required for each enterprise deployment option.

Deployment Option	Supervisor Node	Worker Node	Collector Node	NFS Server	Report Server	Visual Analytics Server	Description
Single Supervisor Node	x						This is the most basic single site enterprise deployment.
Supervisor Node with Collectors	x		x				This is also an enterprise deployment covering multiple sites. Data collection is simplified by deploying a collector for the satellite sites.
Enterprise Cluster	x	x		x			This is the scalable enterprise deployment. An NFS Server is required in the data sharing architecture between Supervisor and Worker nodes.
Enterprise Cluster with Collectors	x	x	x	x			This deployment adds collectors to the mix and is the most comprehensive enterprise deployment.
Supervisor Node with Tableau Visual Analytics	x				x	x	This is the most basic single node enterprise deployment, with added capability for Visual Analytics with Tableau
Supervisor Node with Collectors and Tableau Visual Analytics	x		x		x	x	This is also an enterprise deployment covering multiple sites with added capability for Visual Analytics with Tableau. Data collection is simplified by deploying a collector for the satellite sites.
Enterprise Cluster with Ta bleau Visual Analytics	x	x		x	x	x	This is the scalable enterprise deployment with added capability for with added capability for Visual Analytics with Tableau. An NFS Server is required in the data sharing architecture between Supervisor and Worker nodes.
Enterprise Cluster with Collectors and Tableau Visual Analytics	x	x	x	x	x	x	This deployment adds collectors to the mix and is the most comprehensive enterprise deployment, with added capability for Visual Analytics with Tableau.

 

Deployment Options

FortiSIEM architecture of workers, collectors, and supervisors offers a number deployment options for enterprises at any level of scale, as well as deployment options for managed service providers who need multi-tenant solutions. Topics in this section describe these deployment options in detail, including use cases for each deployment type as well as node and server configurations for each deployment type.

Enterprise Deployment Options

Standalone Supervisor Deployment for Enterprises

Supervisor and Worker Cluster Deployment for Enterprises

Supervisor with Collectors Deployment for Enterprises

Matrix of Enterprise Deployment Configuration Options

Multi-Tenant Deployment Options for Managed Service Providers or Multiple Organizations

Standalone Supervisor Deployment for Multi-Tenancy

Supervisor and Worker Cluster Deployment for Multi-Tenancy

Supervisor with Collectors Deployment for Multi-Tenancy

Matrix of Multi-Tenancy Deployment Configuration Options

Enterprise Deployment Options

For FortiSIEM, an Enterprise deployment is one in which there is a single organization for which data is gathered and analyzed, and the virtual appliances are located entirely on-premises for that organization.

Standalone Supervisor Deployment for Enterprises

Supervisor and Worker Cluster Deployment for Enterprises

Supervisor with Collectors Deployment for Enterprises

Matrix of Enterprise Deployment Configuration Options

Standalone Supervisor Deployment for Enterprises

This is the simplest possible deployment option, in which a single Supervisor handles all the work of monitoring, processing, and analyzing data.

You can configure the Supervisor to use local or NFS storage, depending on your event data storage requirements, as described in Using NFS

Storage with AccelOps

Supervisor and Worker Cluster Deployment for Enterprises

As the number of monitored devices, or the analyzed event rate, grows, one Supervisor may not be able to handle the load. In that case, you can deploy a cluster of Supervisor and Worker virtual appliances that share data over NFS. In a cluster deployment, the Supervisor and Worker nodes have specific functions:

Supervisors, Workers, Collectors, and Organizations

An FortiSIEM deployment can be configured using either a single virtual appliance, or with multiple virtual appliances that play different roles within the deployment. The Supervisor virtual appliance is the primary component in both standalone and cluster deployments, and all deployments begin with the set up and configuration of the Supervisor. As described in Supervisor and Worker Cluster Deployment for

Enterprises, there may be situations in which the single appliance cannot monitor all the data and devices in your infrastructure, and so you can deploy Worker virtual appliances to take up the extra load. Finally, you may encounter situations in which you need to deploy Collectors  for the purpose of gathering data that will be processed by Supervisors and Workers. As described in Supervisor with Collectors Deployment for Enterprises and Supervisor and Worker Cluster Deployment for Multi-Tenancy, these are most likely situations where you need to monitor IT infrastructure for different sites, as in the case of a large or distributed enterprise, or for different organizations, as in the case of multi-tenant installations for Managed Service providers (MSPs). For these situations each Organization is defined separately within FortiSIEM, so you can tailor your monitoring, analytics, and reports to meet the specific needs of that organization.

 

Features and Architecture

FortiSIEM provides an all-in-one, seamlessly integrated and service-oriented IT infrastructure monitoring solution that covers performance, availability, change, and security monitoring aspects of network devices, servers, and applications. It is offered in two versions:

A VMware based virtual appliance, which you can deploy as a single appliance or a cluster of virtual appliances in a highly available, scaled-out grid architecture. This is what we refer to as FortiSIEM Enterprise.

Software-as-a-Service (SaaS), where you deploy a Collector virtual on-premises for a customer, and all of the customer data is transmitted to an FortiSIEM data center. This is what we refer to as FortiSIEM Multi-Tenant, since collector deployments are commonly used by organizations such as Managed Service Providers to monitor the services of their customers.

Some of the features of the FortiSIEM monitoring solution include:

Intelligent Device Discovery

Analytics

Business Services

Architecture

Intelligent Device Discovery

The first step in the monitoring process is IT infrastructure discovery. FortiSIEM has a fast and intelligent discovery engine that can automatically crawl an IT infrastructure and discover network devices, servers, and applications in depth. The user needs to provide appropriate credentials, a discovery IP address range, and optionally a starting router IP address for faster discovery.

A wide range of information is discovered including hardware information, serial numbers and licenses, installed software, running applications and services, and router configuration. The discovered devices are automatically categorized into detailed functional groups, such as Routers/Switches, Firewalls, and Network IPS, and this information is maintained within an integrated configuration management database (CMDB). Some special relationships are also discovered, for example WLAN Access Points to WLAN Controllers, VMware guests to physical hosts, etc. The CMDB is kept up to date through user-defined scheduled discoveries and FortiSIEM listening to changes as part of performance monitoring.

A novel aspect of FortiSIEM discovery is that those aspects of a device that can be monitored are also discovered at the same time. For example, given SNMP, WMI, and JDBC credentials for a Windows server, FortiSIEM might discover the following:

System performance metrics that can be collected by SNMP, for example CPU, memory utilization, and disk space utilization

System performance metrics that can be collected by WMI, for example Disk I/O utilization, memory swap rates, and process utilization Application specific metrics that can be collected by WMI, for example IIS, DNS, DHCP, and Exchange metrics Event logs that can be collected by WMI

Database logs that can be pulled from the server by JDBC

You simply approve the discovered results and monitoring begins. This approach reduces human error, since FortiSIEM learns from the true device configuration state.

Analytics

FortiSIEM uses a unified event-based framework to analyze all data including logs, performance monitoring data. Logs can either be sent to FortiSIEM via Syslog, SNMP traps, or other common log shipping methods, or FortiSIEM can periodically access the system and collect the logs. Performance monitoring data is collected by periodically probing the system. The data is parsed, indexed, and stored in a proprietary flat-file based database. In contrast, the CMDB information is stored in a PostgreSQL relational database. FortiSIEM unified data management architecture combines the two databases and presents a single view to the user.

FortiSIEM provides a broad range of metrics. First, it is possible to search all data based on keywords or in a structured way using the attributes parsed by AcceOps. The search can be done in real time, in which the data streaming in from devices is displayed, or the search can be based on historical data. Historical data is referred to as Reports in FortiSIEM, and can be scheduled to run at intervals you set. A large number of reports are provided in a categorized fashion, based on device type, and also based on functionality such as availability, performance, change and security. Two novel aspects of FortiSIEM metrics include unification and drill-down capabilities. With unification, all the data is analyzed and presented the same way, whether it be real time search, reports, rules or performance, availability, or change or security data. By using drill-down you can start from a specific context, such as Top Authentication Failed Users, and iteratively select attributes to further analyze data and get to the root cause of a problem. As an example, the investigation of Top Authentication Failed users could follow a drill-down of pick user and time range -> Top Destination IP, Ports for specific user and time range -> pick destination IP and port -> Query all raw messages.

FortiSIEM also uses rules for real-time alerting – a real-time event correlation engine analyzes all data and triggers alerts based on these rules. FortiSIEM ships with 500+ broad rules that cover a broad range of inter-related performance, availability, change and security scenarios. Rules can vary from simple text search and threshold conditions, to comprehensive logic supporting full Boolean operators and nested sub-patterns referencing multiple elements including thresholds and defined services. Thresholds can be static or dynamically derived from profiled network, system resource and user activity. You can add new rules, and customize existing ones, as described in Creating Rules using GUI. Business Services

A business service lets you view FortiSIEM metrics and prioritize alerts from a business service perspective. A business service is defined within FortiSIEM as a smart container of relevant devices and applications serving a business purpose. Once defined, all monitoring and analysis can be presented from a business service perspective. It is possible to track service level metrics, efficiently respond to incidents on a prioritized basis, record business impact, and provide business intelligence on IT best practices, compliance reporting, and IT service improvement. What is also novel about FortiSIEM is how easily a business service can be defined and maintained. Because FortiSIEM automatically discovers the applications running on the servers as well as the network connectivity and the traffic flow, you can simply choose the applications and respective servers and be intelligently guided to choose the rest of components of the business service. This business service discovery and definition capability in FortiSIEM completely automates a process that would normally take many people and considerable effort to complete and maintain.

Architecture

The FortiSIEM virtual appliance solution operates as a turnkey, guest host application running within the most popular hypervisors with the option of using NFS or local storage. The implementation process is flexible and can be accomplished in phases to support a variety of distributed and hybrid-cloud implementations The FortiSIEM virtual appliance is placed on a network where it can obtain operational data, as well as establish sessions with the infrastructure. Remote sites can use the FortiSIEM Collector client to locally discover, collect, compress and securely transmit of operation data back to the FortiSIEM virtual appliance. FortiSIEM’ scale-out architecture allows for virtual appliance clustering to increase processing capacity and availability. Additional virtual appliances can be added on-the-fly with nominal configuration, which will automatically distribute workload across cluster members to extend event analysis throughput and to reduce query response time.

 

 

Page 91