SIEM Deployment - Collecting Logs from Linux Servers

Most companies run their business critical systems on Linux servers, which are famous for their stability, performance, security among other capabilities. Collecting logs from Linux servers thus becomes an important step in realizing Log Management projects.

Log management problem and need for Linux and UNIX servers is thought and taken care of long before it is finally taken seriously by Microsoft; therefore configuration is more straightforward and works more stable in my experience.

There is however a list of items to be followed carefully in order to be sure that everything works fine. The list may be hard to keep in mind comparing to steps in Windows, so I list them down below.

1. Check auditd daemon configuration to see if auditing service works fine (/etc/audit/auditd.conf)
2. Check audit event dispatcher configuration (/etc/audisp/audispd.conf)
3. Configure audit event dispatcher syslog configuration (/etc/audisp/plugins.d/syslog.conf)
4.  Create audit rules by editing /etc/audit/audit.rules file (More detail below)
5. Configure the syslog daemon to redirect log messages to a collector server.
6. Restart the daemons to activate the configurations.

I took RedHat family of Linux systems (RHEL, CentOS, Fedora,etc.) for configuration example in this article and configuration steps and commands apply to almost all Linux distributions with small changes.

There is not much to say about first and second steps, as they are routine controls to see if daemons are enabled and fine tunings may be done if necessary.

At the third step, under the syslog.conf file we should configure the args parameter to say what facilities we want to send to the syslog. To be coherent with the below configuration I set it as below:

args = LOG_INFO LOG_LOCAL5 LOG_LOCAL4 LOG_LOCAL3

Then comes a very important step, configuring the audit.rules file which actually is your audit policy for the server. If you are up to this point, most probably your company should already have one and your audit.rules file should not be empty. But in case you started being interested with linux servers just for the sake of log management (like me), I would suggest you to first read and edit, and then copy the usr/share/doc/audit-x.y.z/stig.rules document as your audit.rules file. stig.rules file is a really well prepared document to guide you to write your own rules and it is very good for a starter honestly. In my case the configuration applied was like below:

[root@localhost etc]# vi /usr/share/doc/audit-2.3.7/stig.rules

[root@localhost etc]# cp /usr/share/doc/audit-2.3.7/stig.rules /etc/audit/audit.rules

cp: overwrite `/etc/audit/audit.rules'? y

I can also suggest you to add below lines in your audit.rules file as a best practice. (For 64bit systems in this example)

-a always,exit -F arch=b64 -S sethostname -S setdomainname -k HOSTNAME_CHANGED

-a always,exit -F arch=b64 -S kill -F a1=9 -k KILL9

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S settimeofday -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S adjtimex -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S clock_settime –k SYSTEM_TIME_CHANGED

-w /etc/localtime -p wa -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -S mount -k DEVICE_MOUNTED

-a always,exit -F dir=/boot -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/root -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/etc -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/bin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sbin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib64 -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/usr -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/net -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sys -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/cgroup -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/selinux -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/adm -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/cron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/at -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/anacron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F path=/var/log/messages -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/dmesg -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/secure -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

In the fifth step, we should configure the syslog daemon. In Linux systems, rsyslog service is responsible from reading the events and writing them to specific log files. To decide which actions are going to be logged /etc/rsyslog.conf file should be edited with a text editor.

More specifically RULES section in rsyslog.conf file should be edited like below:

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                                                          /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none                           /var/log/messages

# The authpriv file has restricted access.

authpriv.*                                                                                    /var/log/secure

# Log all the mail messages in one place.

mail.*                                                                                          /var/log/maillog

# Log cron stuff

cron.*                                                                                         /var/log/cron

# Everybody gets emergency messages

*.emerg                                                     

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                                                        /var/log/spooler

# Save boot messages also to boot.log

local7.*                                                                                   /var/log/boot.log

#Log Management

syslog.info; auth.info; daemon.info;                               @@CollectorServerIP

authpriv.info; cron.info; kern.info

# System log information

local5.info, local4.info, local3.info                                 @@CollectorServerIP

In the above configuration @ sign symbolizes log sending over UDP 514 port and @@ symbolizes TCP 514 port. In order to not to lose any logs I configured it over tcp. I have been told by a colleague recently that this may add a significant load on systems where number of logs are important, but I still believe that tcp method should be given a chance before switching to udp, if it is deemed inevitable.

If you want to complicate things you may choose to send your logs you may send them encrypted but that configuration is not a part of this article.

As a final step, we restart the auditd and rsyslog services. At this point we must be able to see on the collector server log messages arriving to the syslog server software installed.