Sunday, August 2, 2015

SIEM Deployment - Configuring Filtering on SmartConnectors

One of the big obstacles security analysts face when deploying SIEM solutions is that, once you ask system owners to send their "security related" logs to your log collectors, they misunderstand you and save everything.

There are also times that system owners really cannot eliminate junk logs (security wise of course).

Because of the fact that more logs mean more money spent (resources, licensing, storage, etc.) we have to eliminate those logs at same point. In this article, I will detail filtering out logs on SmartConnectors in this article, which is the best place to filter logs because it is the closest to the log source.

The first step of filtering should be deciding which logs you are going to filter. It is never bad to say it again, when collecting logs, do set your scenarios before and know what logs you need to fire your rules. All the rest is garbage, which you can eliminate at some level (source or SmartConnector).

In this example, I have chosen filtering out Microsoft Windows logs and the criterion, I use for it is the deviceEventClassId. You can filter your logs according to any criterion you want.

In order to start, we should run runagentsetup script under your <Connector_Home>\current\bin directory. For the sake of simplicity, I used a windows based SmartConnector for this demo.

In the next menu we choose "Modify Connector" option.

Then "Add, modify, or remove destinations" option should be chosen.

Step 4 is an important step to be well understood. Filtering operation, just like aggregation and other SmartConnector level modifications is made per destination, which means that filtering settings you made are only valid for the Logger or ESM that you choose at this step. If you want to do the filtering for a second destination, you should start over once more. This however and fortunately does not apply for failover destinations.

At step 5, "Modify destination settings" option is chosen.

The next menu is where we actually choose the operation we want to configure.

In the final configuration screen, we enter the parameters according to which we are going to filter the incoming logs. For this example, we are filtering out logs in which deviceEventClassId field contains "Microsoft-Windows-Security-Auditing:4674" or "Microsoft-Windows-Security-Auditing:5447".

If you want to learn more about Microsoft Windows Audit Events, I'd suggest you to visit this website and read this blog article.

Once this step is done, we click next and reach the final configuration screen.

Do not forget to restart your SmartConnector service in order to apply the filtering settings.

SIEM Deployment - Installing ArcSight Systems on Amazon EC2

SIEM systems can be easily considered as “Big Data” systems as the resources they use and the amount of information they process fit into Big Data systems scale. Running SIEM components require serious computing resources not only RAM, CPU and hard disk space but also a high IOPS rate.

For those reasons, if you do not have a chance to have a test environment at work or at home (my home server with i7 CPU (8 thread) and 16GB Ram with 256 GB SSD did not really satisfy me honestly) you should look for another solution. After considering my options about buying a better home server, I found out that cloud solutions such as Amazon’s EC2 offer a way better TCO and ROI comparing to owning a home server.

In this article, I’ll guide you through setting up your own Red Hat Enterprise Linux 6.5 server on which you can either install ArcSight ESM, ArcSight Logger or ArcSight Management Center.

For those who are new to Amazon’s Elastic Cloud 2 (EC2) service, it is basically an Infrastructure as a Service (IaaS) offering where you can build a server with the resources you like and pay as you go. Nothing is charged when you keep your system shut. What’s even better is that, Amazon offers entry level machines for free if you create an account an share your credit card information. As long as you use free tier servers, not even a dime is charged.

Actually these free tier servers come installed with the OS you want and they are the best option to use as SmartConnector servers or as log sources. They are ready to be used only in 10 minutes or less.

First we should log into EC2 console from link. (Registration required)

From the next screen, we choose EC2 option under Compute options.

Once successfully logged into EC2 Management Console, we should launch an instance, either using the “Launch Instance” link on the home page or under Instance option on the left-side menu.

On the next screen, we start configuring our server. The first step is to choose the OS (RHEL 6.5), among the Community AMIs proposed by Amazon.

In Step 2, we should choose an Instance Type among many pre-set instances. This is a very important step as how much we are going to pay and the performance we are going to get largely depends on the instance we choose. For ArcSight Logger, an m4.large instance is sufficient for test purposes as well as for ESM starting with m4.2xlarge or higher is a wise choice.

In Step 3, almost all other details not mentioned in Step 2 are chosen. At this step, I highly recommend choosing a /28 subnet for your environment and enabling “Auto-assign Public IP” option. You may choose to use dedicated hardware resources, if you have some bucks to spend, that would provide you better performance for sure (I never used dedicated hardware on EC2). Finally, you may create more than 1 instances if you plan to install more than one component (ESM, Logger,  ArcMC), that would definitely save you some time.

In Step 4, we set up the storage options. For both ESM and Logger, we need at least 50 GB space for /opt/arcsight directory and at least 5 GB free space for /tmp directory. Because of that we add 2 new volumes having slightly more space than the bare minimum demanded, which we will configure later.

After Step 4 we jump to Step 6 and set access rules for accessing our system. Giving inbound permission for tcp 9000, tcp 8443 and tcp 443 ports (in addition to initial tcp 22 for SSH) in a security group will allow us to apply this security group both to ESM and Logger. You should of course limit access to your system only to your public IP, in case you a have static IP.

After Step 6 our configuration is ready to launch. Accessing Linux devices on EC2 may be a new thing, which I will mention in a short article.

Once we log in to our newly installed server with ec2-user account, we should format and mount partitions following the instructions below (after applying sudo su command of course).

1. First we check our partitions with lsblk command and see some output like below:
    $ lsblk
    xvda        202:0       0     10G  0   disk
   └─xvda1 202:1       0       6G   0   part   /
    xvdb        202:16    0      52G  0  disk    
    xvdc        202:17    0         8G  0  disk    

2. Then we format xvdb and xvdc drives in ext4 filesystem as below
   $ mkfs –t ext4  /dev/xvdb
   $ mkfs –t ext4  /dev/xvdc

3. In the third step, we should mount the partitions to the directories we need as follows:
   $ mount  /dev/xvdb  /opt/arcsight
   $ mount  /dev/xvdb  /tmp

4. Basically after this step, we are ready to follow the same steps we do at work or home to install ArcSight systems. However if we miss the following step, the configuration we did will not persist and we may not see the additional partitions mounted after the next reboot. For that reason, we apply following commands:

   $ cp /etc/fstab  /etc/fstab.orig
   $ vi /etc/fstab

Insert the lines below at the bottom of /etc/fstab file and save the file.

/dev/xvdb          /opt/arcsight     ext4       defaults,nofail  0 2
/dev/xvdc          /tmp                  ext4       defaults,nofail  0 2

You can follow AWS documentation link for more detail about this configuration.

Once you finished all these steps, you have a consistent environment for your ArcSight systems.