SIEM Deployment - Configuring Filtering on SmartConnectors

One of the big obstacles security analysts face when deploying SIEM solutions is that, once you ask system owners to send their "security related" logs to your log collectors, they misunderstand you and save everything.

There are also times that system owners really cannot eliminate junk logs (security wise of course).

Because of the fact that more logs mean more money spent (resources, licensing, storage, etc.) we have to eliminate those logs at same point. In this article, I will detail filtering out logs on SmartConnectors in this article, which is the best place to filter logs because it is the closest to the log source.

The first step of filtering should be deciding which logs you are going to filter. It is never bad to say it again, when collecting logs, do set your scenarios before and know what logs you need to fire your rules. All the rest is garbage, which you can eliminate at some level (source or SmartConnector).

In this example, I have chosen filtering out Microsoft Windows logs and the criterion, I use for it is the deviceEventClassId. You can filter your logs according to any criterion you want.

In order to start, we should run runagentsetup script under your <Connector_Home>\current\bin directory. For the sake of simplicity, I used a windows based SmartConnector for this demo.

In the next menu we choose "Modify Connector" option.

Then "Add, modify, or remove destinations" option should be chosen.

Step 4 is an important step to be well understood. Filtering operation, just like aggregation and other SmartConnector level modifications is made per destination, which means that filtering settings you made are only valid for the Logger or ESM that you choose at this step. If you want to do the filtering for a second destination, you should start over once more. This however and fortunately does not apply for failover destinations.

At step 5, "Modify destination settings" option is chosen.

The next menu is where we actually choose the operation we want to configure.

In the final configuration screen, we enter the parameters according to which we are going to filter the incoming logs. For this example, we are filtering out logs in which deviceEventClassId field contains "Microsoft-Windows-Security-Auditing:4674" or "Microsoft-Windows-Security-Auditing:5447".

If you want to learn more about Microsoft Windows Audit Events, I'd suggest you to visit this website and read this blog article.

Once this step is done, we click next and reach the final configuration screen.

Do not forget to restart your SmartConnector service in order to apply the filtering settings.

SIEM Deployment - Installing HP Arcsight SmartConnector on Windows Servers

SIEM system administrators mostly come from Linux world and they prefer using Linux OS for HP ArcSight component installations. I also agree on that decision as the performance and security they provide is superior comparing to Windows Servers.

However there are some situations where you have to use Windows Servers for SmartConnector installations such as when you want to use WinRM application for Windows log collection. Your company may be lacking experienced Linux admins and that could be the second reason to use Windows based SmartConnectors. Last but not the least, as you can see from the installation video, it is much easier to install Windows based SmartConnectors then Linux based ones.

Below, you can find details about the basic installation of an HP ArcSight SmartConnector on a Windows Server for collecting log messages.

REQUIREMENTS / PREREQUISITES

1. A Windows 2008 Server installed.
2. A user with sufficient rights to install the software.
3. A user with added to "Event Log Readers" group to read the logs on the server. (OPTIONAL)
4. Connector binaries downloaded. (Download the correct version for your OS, x86 or x64!!)
5. Connector destinations ( ArcSight Logger and/or ArcSight ESM) installed and working.
6. Create a receiver on the logger to connect the connector.
7. Create a subscription on Event Viewer to get logs.
8. Check the configuration of log receiving folders and increase size.
9. Define the protocol and port on which you will listen the incoming logs.
10. Firewall permissions given for incoming log collection.

INSTALLATION

1. Create installation directory preferably under your second partition E:\SmartConnectors\Microsoft.
2. Run the setup file.
3. Install the connector to run standalone or as a service.
4. Start the connector service from services.msc.
5. Check events on the logger.
6. Set agent.properties parameters.(OPTIONAL)
7. Set agent.wrapper.conf parameters.(OPTIONAL)

SIEM Deployment - Windows Local Security Policy (Audit Policy) Configuration

In my previous blog article on collecting logs from Windows Servers, I missed a control point which can be very important while getting results from your SEM installation.

Let me make it more clear for you. While trying to write a correlation rule for an event, in which Windows Server Log Configuration is changed, we first found the event id for such action. After finding the event id, we noticed that although the main audit categories are configured correctly to log both successful and failed attempts, the source system actually did not generate any logs. You will find the reason why below.

The way auditing rules are defined has drastically changed from Windows Server 2003 to Windows Server 2008 and later. In Windows Server 2008, you have the 8 main audit policy categories, each with success and failure options as the scheme below shows.

In addition to main categories, in the same screen at the bottom of the menu, another setting called "Advanced Audit Policy Configuration" exists. Under that option the 8 main categories are detailed under 53 subcategories which provides advanced granularity (not comparing with Linux of course).

The important thing is to know which policy items have precedence over the other, the general policy or the more detailed policy? By default, the main categories have precedence over subcategories, even if you configured both. The way how the default behavior is changed and how it should be configured to avoid complications can be found here. The operation is different for domain computers and non-domain computers.

The best and safe approach to deal with this is to configure the audit policy using the detailed options and force the priority on it over the general policy.

Once it is decided that subcategories are going to be used for auditing, it is important to know what subcategories should be chosen with what actions in order to get the right amount of events. Audit categories concerning object access may generate too many logs to be stored and processed and they can produce important numbers when you have a considerable number of servers to monitor. Once again, the top down approach in design of SEM systems show the correct way, only generate the logs that you are going to need or use for correlation rules or compliance.

After a careful analysis of events and documentation, I ended up creating the policy below for my installations. I believe that this policy covers most of the needs but if you consider using it you'd better spend some time to adjust it according to your own needs.

Finally, the policy slightly differs for Active Directory Domain Controllers and the other servers as DS Access category only concerns Domain Controllers. For a non-DC server all the subcategories under DS Access category should be set to "No auditing".

SYSTEM AUDIT POLICY SETTINGS
Category/Subcategory	Suggested Settings
System
Security System Extension	Success and Failure
System Integrity	Success and Failure
IPsec Driver	No Auditing
Other System Events	Failure
Security State Change	Success and Failure
Logon/Logoff
Logon	Success and Failure
Logoff	Success and Failure
Account Lockout	Success and Failure
IPsec Main Mode	No Auditing
IPsec Quick Mode	No Auditing
IPsec Extended Mode	No Auditing
Special Logon	Success and Failure
Other Logon/Logoff Events	Success and Failure
Network Policy Server	Success and Failure
Object Access
File System	Success and Failure
Registry	Success and Failure
Kernel Object	Success and Failure
SAM	No Auditing
Certification Services	Success and Failure
Application Generated	Success and Failure
Handle Manipulation	No Auditing
File Share	Success and Failure
Filtering Platform Packet Drop	No Auditing
Filtering Platform Connection	No Auditing
Other Object Access Events	No Auditing
Detailed File Share	No Auditing
Privilege Use
Sensitive Privilege Use	No Auditing
Non Sensitive Privilege Use	No Auditing
Other Privilege Use Events	No Auditing
Detailed Tracking
Process Termination	Success and Failure
DPAPI Activity	No Auditing
RPC Events	Success and Failure
Process Creation	Success and Failure
Policy Change
Audit Policy Change	Success and Failure
Authentication Policy Change	Success and Failure
Authorization Policy Change	Success and Failure
MPSSVC Rule-Level Policy Change	No Auditing
Filtering Platform Policy Change	No Auditing
Other Policy Change Events	Failure
Account Management
User Account Management	Success and Failure
Computer Account Management	Success and Failure
Security Group Management	Success and Failure
Distribution Group Management	Success and Failure
Application Group Management	Success and Failure
Other Account Management Events	Success and Failure
DS Access
Directory Service Changes	Success and Failure
Directory Service Replication	No Auditing
Detailed Directory Service Replication	No Auditing
Directory Service Access	Success and Failure
Account Logon
Kerberos Service Ticket Operations	Success and Failure
Other Account Logon Events	Success and Failure
Kerberos Authentication Service	Success and Failure
Credential Validation	Success and Failure

Considerations About a Successful SIEM and Log Management Project

You may spend a lot of effort to build up an infrastructure of SIEM and Log Management composed of source systems, connectors, loggers, rule creation and correlation engines and management systems and then see very little of valuable output….if you do not pay attention to the first thing you actually do in the first place: Define a proper auditing policy.

A recent and very successful approach for building Log Management and SIEM capabilities actually consists of an inverse installation process. Some Security Service Providers come and collaborate with your teams to apply defined risk scenarii on your infrastructure to see if your infrastructure components generate the log messages which should alert you that something unusual and odd is going on. It is at this very step that your SIEM team learns which type of events they should be collecting among a big pile of others, which in many cases constitute most of your log storage without actually providing a value.

Such an approach may create huge differences in outputs and may trigger changes in your infrastructure. I know companies which changed some of their components just because they do not provide essential log information which would allow security alert generation.

A very important thing to keep in mind when deploying SIEM solutions is the involvement of all infrastructure and application teams. No matter how qualified IT Security guys responsible for SIEM deployment, they do not master Operating Systems as much as Windows, Linux and UNIX administrators do, also considering different versions of OSes that can be in place. To my experience, I know that 3 generations of Windows Servers coexist in majority of companies without counting R2 versions. It goes the same way for databases and applications. A SIEM project would probably fail or underperform if all IT teams do not collaborate with SIEM project team and stay isolated in their silos.

Another way of dealing with this issue would be to create a security team composed of security masters in each domain. First difficulty in that approach is to bring together such talents which is very costly and the second challenge is to keep them in the company and provide consistency because such people are highly in demand. This option seems applicable only in very large structures such as multinationals, especially in finance sector where there really are some things to be at stake, money and also reputation.

There are of course lots and lots to say about other aspects of SIEM and Log Management projects. But maybe the most important things to know about them are to set the expectations correctly (Benefits, Aim: “Security?, Compliance?, Both?”, Scope, Schedule and Budget), be patient, provide continuous support and monitor the output closely. The technology in this market is rapidly evolving and it still has much more to offer in years to come.

SIEM Deployment - Collecting Logs from Windows Servers

An important step in SIEM deployments is to collect logs from servers on which your infrastructure, applications and middleware run.

The event collection from servers is to be seen in the 2nd step in a 3 step approach in SIEM deployment, first step being network and network security devices and last step being the applications.

Microsoft proposes a well-designed log collection infrastructure with 2 options left to the administrators. Both methods use WinRM protocol and are safer than collecting logs using RPC.

In the first method, the collector server connects to event sources and pulls the logs with the help of a domain user which is added to the Event Log Readers group. This method is known as “Collector Initiated Event Forwarding” in Microsoft world and can be better remembered as PULL method. This is the easier method but does not scale well in large environments and may be inefficient as the collector server should regularly contact event sources whether they have logs to transmit or not.

In the second method, which I will detail more in this article, event sources push logs to a collector server over http or https. This method is known as “Source Initiated Subscription” and can be better remembered as PUSH method. This method scales much better to larger environments and is definitely more secure when https option is used as transmission protocol. This method requires less configuration when configured through Group Policy. However special attention must be paid for the collector server when logs are sent over https, especially about listener configuration as it is not that well documented on Microsoft’s documentation and needs good system administration skills.

EVENT SOURCE CONFIGURATION

First of all, the winrm service should be activated from the command line with winrm quickconfig command.

In the Local Group Policy Editor, under Computer Configuration -> Administrative Templates ->  Windows Components -> Event Forwarding menu, “Configure the server address, refresh interval, and issuer certificate authority of a target Subscription Manager” setting should be configured as below:

            

At this point you can prefer to use http or https protocols to transmit your logs to the collector server. The only difference at the event source is the port numbers used for http and https protocols which are 5985 and 5986 respectively as shown below:

Server=http://Servername.Domain:5985/wsman/Subscription Manager/WEC,Refresh=60

Server=https://Servername.Domain:5986/wsman/Subscription Manager/WEC,Refresh=60

Log sending period is chosen as 60 seconds in these examples.

 Once event forwarding configuration is done, the group policy must be updated with  gpupdate /force command.

 In Local Security Policy screen, audit policy settings under Security Setttings -> Local Policies ->  Audit Policy should be configured. It is recommended to enable logging for both successful and failed attempts for all policy items except “Audit object access”, as this setting may generate too many logs if both successful and failed attempts are logged. I recommend configuring this setting only for failed attempts for the beginning and deciding later according to the criticality of the server.

EVENT COLLECTOR CONFIGURATION

First of all, Windows Event Collector service and WinRM service should be enabled from the command line by using wecutil qc and winrm qc commands respectively.

 After this step, Event Viewer is launched and a new Subscription should be created by right clicking on Subscriptions menu item and selecting Create Subscription or selecting from the menu on the right.

Once the Create Subscription window opens, for source initiated subscriptions settings are configured as below.  These settings largely depend on your own needs however there are real life experiences to follow such as collecting logs in “Hardware Events” destination as logs from this folder are correctly parsed at Arcsight and need no further efforts.

Subscription name: Event collection

Destination log: Hardware Events

Subscription type: Source computer initiated

  

Configuration ends after this step and if there are no problems, like firewall rules blocking the traffic, you should be seeing transferred logs in your target log collection folder. The best way to check if there is a problem or not is to use "Runtime Status" option from the menu on the right.

For more details you can read the Technet article below:

https://msdn.microsoft.com/en-us/library/windows/desktop/bb870973(v=vs.85).aspx