SIEM Deployment - Installing HP Arcsight Software Connectors on Linux

In HP ArcSight solution architecture one of the most value adding components is the Smart Connectors. With the several functions they provide, Smart Connectors really help differentiating HP ArcSight’s SIEM solution from other.

So what exactly are ArcSight Smart Connectors? In a 3 layered SIEM Architecture, Arcsight Smart Connectors constitute the second layer between log processing systems ( Arcsight Logger or Arcsight ESM) and source systems generating logs according to defined audit policies.

From technical SIEM perspective, ArcSight Smart Connectors are Java applications which allow receiving or fetching logs from one defined log source, which can be several devices sending their logs in syslog format over the same protocol and port number (e.g. UDP 514) or an application writing its logs to a flat file. ArcSight Smart Connectors come with 256 MB minimum memory size and that memory is adjustable up to 1024 MB by configuration agent.properties file, among other connector properties to be changed according to your specific needs.

One physical server can host up to 8 connector processes, meaning that you can collect logs from 8 different source groups as long as your server support that much capacity.

Below, you can find details about the basic installation of an HP ArcSight Smart Connector on a CentOS Linux server for collecting syslog messages.

REQUIREMENTS / PREREQUISITES

1. A RHEL or CentOS Linux 6.X Server installed.

2. Root or sudo rights for connector user.

3. Connector binaries downloaded. (Download the correct version for your OS, x86 or x64 !!)

4. Connector destinations (Logger and/or ESM) installed and working.

5. Define the protocol and port on which you will listen the incoming logs.

   Choose port numbers over 1024 if you are installing with a non-root user as non-root users are not allowed to listen ports below 1024.

INSTALLATION

1. Create installation directory under /opt path. In this example it is /opt/arcsight/connectors .

2. Create a receiver on the logger to connect the connector.

3. Run the connector binary you previously downloaded. (From /home/arcsight directory in my installation).

# ./ArcSight-7.1.1.7348.0-Connector-Linux64.bin -i console

4. Install the connector to run standalone or as a service.

INSTALLATION_PATH\current\bin\arcsight connectors  à Run Standalone

INSTALLATION_PATH \current\bin\arcsight agentsvc -i -u arcsight -sn syslog_unix à Run with arcsight user as a service with arc_syslog_unix service name

5. Check events on the logger.

6. Set agent.properties parameters (Optional)

7. Set agent.wrapper.conf parameters (Optional)

SIEM Deployment - HP Arcsight Logger Installation

HP Arcsight Logger product constitutes the log management part of HP's Security Event Management and Log Management product portfolio, ESM being the security event management part.

Before getting this much into SEM and Log Management, they both meant the same thing for me, as most of the products available on the market were trying to do. Architecture-wise, Arcsight managed to distinguish its offerings  for different needs and markets. This issue is the topic of another blog entry however if you are looking for a product which will allow you to store all your logs in a stable way and query specific patterns very quickly then Arcsight Logger is the solution you are looking for.

As of mid-2015, the latest version of HP Arcsight Logger is the 6.0 SP1 version with no known security bugs. Arcsight 6.0 SP1  :

• Distributes latest version of OpenSSL, 0.9.8zc, which addresses multiple vulnerabilities including CVE-2014-0224.
• Resolves the Bourne-Again Shell (Bash) Code Injection Vulnerability, including CVE-2014-6271, CVE-2014-7169, CVE-2014-7186, CVE-2014-7187, CVE-2014-6277, and CVE-2014-6278.
• Disables support for SSL v3.0 encryption, to address the Padding Oracle On Downgraded Legacy Encryption (POODLE) vulnerability (CVE-2014-3566).

Version 6.0 SP1 also brings support for doubled local storage size. Each instance of logger now can support up to 8TB of logs before sending logs to archive.

In this article, I tried to resume how you can get your Logger up and running in a short amount of time. This is also the first time for me to include a video which obviously makes the article more interesting.

The installation of Arcsight Logger is a 2 step process, Preparation and Installation itself.

For the preparation you should of course have your server equipped with necessary resources (just like all other Log Management products, Logger also is greedy in resources), Logger software, license,user accounts (root privileges are required for the installation) and ports.
PREPARATION
Server Requirements
OS

• Red Hat Enterprise Linux (RHEL) versions 6.2 and 6.5 (64-bit),
• CentOS versions 5.5 and 6.5 (64-bit)

Hardware

For the Trial Logger and VM Instances:
CPU: 1 or 2 x Intel Xeon Quad Core or equivalent
Memory: 4 - 12 GB (12 GB recommended)
Disk Space: 10 GB (minimum) in the Logger installation directory (/opt/...)
Temp directory: 1 GB

For the Enterprise Version of Software Logger:
CPU: 2 x Intel Xeon Quad Core or equivalent
Memory: 12 - 24 GB (24 GB recommended)
Disk Space: 65 GB (minimum) in the Software Logger installation directory. (/opt/...)
Root partition: 400 GB
Temp directory: 1 GB

For performance reasons, it is preferable to use dedicated hardware for Logger rather than using virtual machines. For faster searchs archive connections should be over direct fiber channel rather over NFS.

Logger interface can be reached through all known browsers with recent versions.

Logger can be installed using root and non-root accounts but following points should be taken into consideration:

• For root installs, allow access to port 443 as well as the ports for any protocol that thelogger receivers need, such as port 514 for the UDP receiver and port 515 for the TCP receiver.
• For non-root installs, allow access to port 9000 as well as the ports for any protocol that the Logger receivers need, such as port 8514 for the UDP receiver and port 8515 for the TCP receiver.

INSTALLATION
You can follow instructions given below. The video also follows the same steps.

1. Install Linux Server (Minimal Server with GUI for trial installations). Do not "Easy Install" when using Vmware and manually set partitions

2. Adjust partitions as below as a minimum:

/ 10240 MB
/home 10240 MB
swap 4096  MB (Typically half of your RAM but do not exagerrate)
/opt 70000 MB (Give Minimum 65 GB, more is better)
/tmp 2048  MB

3. Create arcsight user

groupadd arcsight
useradd -c "arcsight_software_owner" -g arcsight -d /home/arcsight -m -s /bin/bash arcsight

4. Copy sources and license to /home/arcsight

5. Set hostname in /etc/hosts

#vim /etc/hosts

192.168.X.Y logger.mycompany.com

6. Make sure system time is correct

7. Create /opt/arcsight with arcsight user
chown arcsight:arcsight /opt/arcsight

8. Disable selinux and iptables for performance (Use Network Firewall instead !!)

Selinux can be an important performance drawback!

#chkconfig iptables off
#chkconfig ip6tables off
# vim cat /etc/sysconfig/selinux

SELINUX=disabled

9. Change release file if not using recommended versions
vim /etc/redhat-release
CentOS release 6.5 (Final)

10. Make OS changes specific to Logger
chmod +x /sbin
chmod +x /sbin/ifconfig
chmod +x /sbin/lspci
chmod +x /usr/sbin

#vim /etc/security/limits.d/90-nproc.conf
* soft nproc 10240
* hard nproc 10240
* soft nofile 65536
* hard nofile 65536

#reboot

11.Install Logger
#cd /home/arcsight

#./ArcSight-logger-6.0.0.7307.1.bin

SIEM Deployment - Collecting Logs from Linux Servers

Most companies run their business critical systems on Linux servers, which are famous for their stability, performance, security among other capabilities. Collecting logs from Linux servers thus becomes an important step in realizing Log Management projects.

Log management problem and need for Linux and UNIX servers is thought and taken care of long before it is finally taken seriously by Microsoft; therefore configuration is more straightforward and works more stable in my experience.

There is however a list of items to be followed carefully in order to be sure that everything works fine. The list may be hard to keep in mind comparing to steps in Windows, so I list them down below.

1. Check auditd daemon configuration to see if auditing service works fine (/etc/audit/auditd.conf)
2. Check audit event dispatcher configuration (/etc/audisp/audispd.conf)
3. Configure audit event dispatcher syslog configuration (/etc/audisp/plugins.d/syslog.conf)
4.  Create audit rules by editing /etc/audit/audit.rules file (More detail below)
5. Configure the syslog daemon to redirect log messages to a collector server.
6. Restart the daemons to activate the configurations.

I took RedHat family of Linux systems (RHEL, CentOS, Fedora,etc.) for configuration example in this article and configuration steps and commands apply to almost all Linux distributions with small changes.

There is not much to say about first and second steps, as they are routine controls to see if daemons are enabled and fine tunings may be done if necessary.

At the third step, under the syslog.conf file we should configure the args parameter to say what facilities we want to send to the syslog. To be coherent with the below configuration I set it as below:

args = LOG_INFO LOG_LOCAL5 LOG_LOCAL4 LOG_LOCAL3

Then comes a very important step, configuring the audit.rules file which actually is your audit policy for the server. If you are up to this point, most probably your company should already have one and your audit.rules file should not be empty. But in case you started being interested with linux servers just for the sake of log management (like me), I would suggest you to first read and edit, and then copy the usr/share/doc/audit-x.y.z/stig.rules document as your audit.rules file. stig.rules file is a really well prepared document to guide you to write your own rules and it is very good for a starter honestly. In my case the configuration applied was like below:

[root@localhost etc]# vi /usr/share/doc/audit-2.3.7/stig.rules

[root@localhost etc]# cp /usr/share/doc/audit-2.3.7/stig.rules /etc/audit/audit.rules

cp: overwrite `/etc/audit/audit.rules'? y

I can also suggest you to add below lines in your audit.rules file as a best practice. (For 64bit systems in this example)

-a always,exit -F arch=b64 -S sethostname -S setdomainname -k HOSTNAME_CHANGED

-a always,exit -F arch=b64 -S kill -F a1=9 -k KILL9

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S settimeofday -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S adjtimex -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S clock_settime –k SYSTEM_TIME_CHANGED

-w /etc/localtime -p wa -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -S mount -k DEVICE_MOUNTED

-a always,exit -F dir=/boot -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/root -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/etc -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/bin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sbin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib64 -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/usr -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/net -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sys -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/cgroup -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/selinux -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/adm -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/cron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/at -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/anacron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F path=/var/log/messages -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/dmesg -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/secure -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

In the fifth step, we should configure the syslog daemon. In Linux systems, rsyslog service is responsible from reading the events and writing them to specific log files. To decide which actions are going to be logged /etc/rsyslog.conf file should be edited with a text editor.

More specifically RULES section in rsyslog.conf file should be edited like below:

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                                                          /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none                           /var/log/messages

# The authpriv file has restricted access.

authpriv.*                                                                                    /var/log/secure

# Log all the mail messages in one place.

mail.*                                                                                          /var/log/maillog

# Log cron stuff

cron.*                                                                                         /var/log/cron

# Everybody gets emergency messages

*.emerg                                                     

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                                                        /var/log/spooler

# Save boot messages also to boot.log

local7.*                                                                                   /var/log/boot.log

#Log Management

syslog.info; auth.info; daemon.info;                               @@CollectorServerIP

authpriv.info; cron.info; kern.info

# System log information

local5.info, local4.info, local3.info                                 @@CollectorServerIP

In the above configuration @ sign symbolizes log sending over UDP 514 port and @@ symbolizes TCP 514 port. In order to not to lose any logs I configured it over tcp. I have been told by a colleague recently that this may add a significant load on systems where number of logs are important, but I still believe that tcp method should be given a chance before switching to udp, if it is deemed inevitable.

If you want to complicate things you may choose to send your logs you may send them encrypted but that configuration is not a part of this article.

As a final step, we restart the auditd and rsyslog services. At this point we must be able to see on the collector server log messages arriving to the syslog server software installed.