SIEM Planning - Reference Architecture for Midsize Deployments

After going through several websites and documents, I sadly discovered, like many of you had before, that HP haven’t yet published any reference architecture or certified design documents for different needs.

I decided to write a series blog articles to create reference architectures for SIEM deployments, basically for HP ArcSight, but the fact that solution components are more or less similar in different vendors, I believe they will be applicable to all SIEM environments.

Gartner defines a small deployment as one with around 300 log sources and 1500 EPS. A midsize deployment is considered to have up to 1000 log sources and 7000 EPS. Finally a large deployment generally covers more than 1000 log sources with approximately 15000 EPS. There can of course be larger deployments with over 15000 EPS but architecture-wise they can be considered as very “large” deployments.

In this article, I will give the details of a midsize deployment, covering components both for a primary datacenter and a disaster recovery center, working in an active-passive setup.

The reference architecture for midsize deployment is for a scenario where the company needs both a long term log storage solution (ArcSight Logger) and Security Event Management and SOC capabilities (ArcSight ESM).

The scheme below shows how different components of the architecture are set up.

• In this setup software SmartConnectors are used to collect the logs. Up to 8 software connectors can be configured on a server and 1 GB of memory should be allocated on the server for each connector instance other than what the server needs for its operation.
• In case appliances are not used, do pay attention to use built-for-purpose hardware servers where resources are not shared because like other big data solutions, these systems are greedy in terms of resources (CPU, Memory, IOPS rate) and do not perform well on virtual environments.
• Sources send logs to one SmartConnector only. SmartConnector level redundancy is only possible only for Syslog connectors and that when connectors are put behind a load balancer. This also provides load sharing and scalability and is a best practice. DB and File connectors do not have such options as they pull the logs from sources.
• When a DB or File collector is down, no log is lost until collector comes back as the logs continue to be written on local resources at the source.
• For log storage and searching, SmartConnectors in each datacenter send their logs to their respective Logger appliance hosted in the same location, providing  important bandwidth savings. Each logger appliance back up the other one using the failover destinations option configured on the SmartConnectors. Thanks to the peering configuration between loggers, logs can be queried through any of the logger appliances without having to connect on each device.
• DC ESM is the primary ESM for both datacenters. DRC ESM is only used in case of DR
• Logs and Alerts are archived daily both on ESM and Logger.
• In DR case, there is no RPO. Configurations for ESM and Logger are planned to be synchronized manually. ESM and Logger are expected to be operational instantly.
• Configuration backups for SmartConnectors and Loggers are collected using Arcsight Management Center (Arc MC).
• SmartConnector statistics and status can be easily followed using Arc MC as well. Realizing SmartConnector updates are also recommended to be done over Arc MC using the GUI.
• SmartConnector level configuration options (aggregation, filter out, batching etc.) are easier to be configured using Arc MC.
• Finally it is strongly recommended to use a Test ESM system to test all filters, rules, active lists and other configuration objects before applying them on production systems as a misconfguration in these settings may crash your ESM and make you lose very valuable data.