SIEM Planning - Capacity Planning and Sizing

SIEM projects are well-known to be demanding and greedy when it comes to the resources and your CIOs/CISOs would like to hear about your direct (software licensing, server investment, etc.) and indirect ( archiving storage, support) costs for at least following 3 years other than the benefits the Project will provide.

In this article, I will of course give the basic formulas about sizing. However the most important information to be provided is the real life experience of a live operation concerning the values and comparison to those values to the benchmarks.

First unit, which constitutes the base for all our calculations, is the Event per Second (EPS) value that each source system generates. EPS value greatly depends on 2 factors, audit policy or rules applied on the source system and business of the system. A server with “Object Access” audit rules enabled and Web Server functionality configured of course would not generate the same number of logs with a standard server. Windows family of servers also tends to generate much greater number of logs than Linux and UNIX servers, all with standard default configurations.

EPS estimations can be reached using links 1, 2 and 3.

Having calculated the number of EPS for each source asset group the next step to do is calculating the Event per Day (EPD) value.

EPD = ∑ EPS X 86400

Once EPD value is calculated, we have to decide an average log message size to know how much storage we will need each day. Log sources generate logs starting from 200 bytes range on network and infrastructure devices to 10 kilobytes or more on application and database side.  Syslog standard (RFC 5424) sets the maximum size of the content field of a log message to 2 kilobytes. In light of this information, it is wise and advisable to assume a raw log message size as 500 bytes.

Average raw log message size being set to 500 bytes, the amount of Daily log messages in GB is calculated as follows:

Daily Raw Log Size = EPD * 500 / (1024)³

Log management appliances do some changes on the log messages to make them understandable and meaningful. This operation is called “Normalization” and it increases the log size depending on the solution you use. In my personal experience with HP ArcSight, normalization increased the log size about 90% to 100%. Some other people have seen up to 200% of increase in their experiences.  As a result we obtain the below given formula for daily normalized log size:

Daily Normalized Log Size = Daily Raw Log Size * 2

The calculated value does not really represent the daily storage value for log management systems. Many vendors came up with proprietary compression solutions and claim they compress logs 10 times (10:1) which is quite idealistic. It is however, safe to consider a ratio of 8:1 for calculations. So the formula becomes:

Daily Storage Requirement = Daily Normalized Log Size / 8

The annual storage need would basically be 365 times the Daily Storage Requirement, if you want your calculations to be on the safe side. Nevertheless, EPS numbers seriously fall during weekends and vacations. Watch how much your average EPS numbers decrease in such periods and do your own calculations for your annual needs.

Annual Storage Requirement = Daily Storage Requirement * 365

The last important point is the retention period when you plan your storage investments for future. 2 factors are decisive in the definition of retention period, Compliance Requirements and Security Requirements.

Compliance Requirements only concern Log Management systems, in HP’s case it is Logger and there is not much to decide really, whatever the legislation obliges, you have to configure.

For security needs which are addressed by HP’s ESM system, the decision is yours. I have seen many decision makers trying to keep themselves on the very safe side and take retention periods unnecessarily long.  According to Mandiant, the median number of days attackers were present on a victim network before they were discovered was 205 days in 2014, down from 229 days in 2013 and 243 days in 2012. This brings me to the conclusion that retention period for security alert creation, monitoring, trending and forensics should be at least 1 year and not longer than 3 years. According to the same study of Mandiant, “The longest time an attacker was present before being detected in 2013 was six years and three months.”. Last but not the least, the retention period of course depends on the sector of activity, defense being the longest and the strictest followed by financial institutions.

A rough estimation about Storage IOPS values can be calculated with the following formulas:

Storage IOPS Needed (Direct Attached Storage) = EPS * 1.2

Storage IOPS Needed (SAN) = EPS * 2.5