SIEM Deployment - Creating Logs on Linux Servers with audit.rules

As I mentioned in several other blog articles, your Security Event Management infrastructure is only as effective as your source auditing capabilities. If you are not generating the necessary logs, containing useful information and in an understandable and meaningful structure, no matter how correctly you deploy your log management product, you end up failing.

That is actually why I am spending this much time (and you also should) examining each and every source system and write about them here in several articles. My first articles were on the mere basics you should follow to see your logs on the log management platform. In these new series of articles, I will mention about what events, as a minimum, we should care and how to log them. Let’s get started!

In Linux systems, what events are going to be logged is managed by the audit.rules file which is most of the times located under /etc/audit/ folder. In the scheme below you can see the way the audit mechanism works in Linux.

The good point concerning auditing features of Linux comparing to Windows Servers is that you can pretty much audit everything and customize the logs according to your needs. You can write audit rules for any process or command you want, you can specifically audit the actions of a special user or audit only a specific action (write, read, execute, append). The name tags that you can add to your log messages may greatly simplify your job when you will be writing your correlation rules on your SEM engine.

The way how an audit rule is written is explained in auditctl man page. Below I am giving a nice template which covers most of the general situations. You should make sure that you cover all your critical processes by adding audit rules for them.

An important thing to know about using this file is to adapt it to your own systems. One of the first things you should do is to use arch= parameter accordingly whether you are using a 32 bit system (b32) or a 64 bit system (b64). Some files and commands change from older versions to newer ones, eg. faillog file which keeps the failed log attempts in Red Hat Enterprise Linux 5 does not exist anymore in later versions and you should configure for pam_tally process and files. Also, please change the comments in rules (text after –k parameter) according to your needs.

Finally there are 2 parameters that need to be used with caution. Do use “e -2” parameter in the end of the file to prevent tampering without logging and reboot. The second parameter “h -2” is more likely to be used in military/defense environments, causes system to halt if logging is crashed, so it should be used with caution.

# First rule - delete all -D

# Increase the buffers to survive stress events.

# Make this bigger for busy systems

-b 8096

# Feel free to add below this line. See auditctl man page

#Capture all failures to access on critical elements

-a exit,always -F arch=b64 -S open -F dir=/etc -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/bin -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/sbin -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/usr/bin -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/usr/sbin -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/var -F success=0 -k CriticalElementFailures

-a exit,always -F arch=b64 -S open -F dir=/home -F success=0 -k CriticalElementFailures

#Capture all successful deletions on critical elements

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/etc -k CriticalElementDeletions

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/bin -k CriticalElementDeletions

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/sbin -k CriticalElementDeletions

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/usr/bin -k CriticalElementDeletions

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/usr/sbin -k CriticalElementDeletions

-a exit,always -F arch=b64 -S unlinkat -F success=1 -F dir=/var -k CriticalElementDeletions

#Capture all successful modification of owner or permissions on critical elements

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/etc -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/bin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/sbin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/usr/bin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/usr/sbin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/var -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S fchmodat -S fchownat -F dir=/home -F success=1 -k CriticalElementModifications

#Capture all successful modifications of content

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/etc -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/bin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/sbin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/usr/bin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/usr/sbin -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/var -F success=1 -k CriticalElementModifications

-a exit,always -F arch=b64 -S pwrite64 -S write -S writev -S pwritev -F dir=/home -F success=1 -k CriticalElementModifications

#Capture all successful creations

-a exit,always -F arch=b64 -S creat -F dir=/etc -F success=1 -k CriticalElementCreations

-a exit,always -F arch=b64 -S creat -F dir=/bin -F success=1 -k CriticalElementCreations

-a exit,always -F arch=b64 -S creat -F dir=/sbin -F success=1 -k CriticalElementCreations

-a exit,always -F arch=b64 -S creat -F dir=/usr/bin -F success=1 -k CriticalElementCreations

-a exit,always -F arch=b64 -S creat -F dir=/usr/sbin -F success=1 -k CriticalElementCreations

-a exit,always -F arch=b64 -S creat -F dir=/var -F success=1 -k CriticalElementCreations

#Capture all successful reads (only for High-Impact Systems)

-a exit,always -F arch=b64 -S open -F dir=/etc -F success=1 -k CriticalElementReads

-a exit,always -F arch=b64 -S open -F dir=/bin -F success=1 -k CriticalElementReads

-a exit,always -F arch=b64 -S open -F dir=/sbin -F success=1 -k CriticalElementReads

-a exit,always -F arch=b64 -S open -F dir=/usr/bin -F success=1 -k CriticalElementReads

-a exit,always -F arch=b64 -S open -F dir=/usr/sbin -F success=1 -k CriticalElementReads

-a exit,always -F arch=b64 -S open -F dir=/var -F success=1 -k CriticalElementReads

#Monitor for changes to shadow file (use of passwd command)

-w /usr/bin/passwd -p x

-w /etc/passwd -p ra

-w /etc/shadow -p ra

#Monitor for use of process ID change (switching accounts) applications

-w /bin/su -p x -k PrivilegeEscalation

-w /usr/bin/sudo -p x -k PrivilegeEscalation

-w /etc/sudoers -p rw -k PrivilegeEscalation

#Monitor for use of tools to change group identifiers

-w /usr/sbin/groupadd -p x -k GroupModification

-w /usr/sbin/groupmod -p x -k GroupModification

-w /usr/sbin/useradd -p x -k UserModification

-w /usr/sbin/usermod -p x -k UserModification

#Ensure audit log file modifications are logged.

-a exit,always -F arch=b64 -S unlink -S unlinkat -F dir=/var/log/audit -k AuditLogRemoval

# Monitor for use of audit management tools

-w /sbin/auditctl -p x -k AuditModification

-w /sbin/auditd -p x -k AuditModification

# Ensure critical apps are monitored.  List will vary by mission.

-a exit,always -F arch=b64 -F path=/sbin/init -k CriticalAppMonitoring

-a exit,always -F arch=b64 -F path=/usr/bin/Xorg -k CriticalAppMonitoring

-a exit,always -F arch=b64 -F path=/usr/sbin/sshd -k CriticalAppMonitoring

-a exit,always -F arch=b64 -F path=/sbin/rsyslogd -k CriticalAppMonitoring

#  Ensure attribute changes are audited

-a exit,always -F arch=b64 -S chmod -S chown -S fchmod -S fchown -S setuid -S setreuid -S getxattr -S setxattr -k AttributeChanges

SIEM Deployment - Collecting Logs from Linux Servers

Most companies run their business critical systems on Linux servers, which are famous for their stability, performance, security among other capabilities. Collecting logs from Linux servers thus becomes an important step in realizing Log Management projects.

Log management problem and need for Linux and UNIX servers is thought and taken care of long before it is finally taken seriously by Microsoft; therefore configuration is more straightforward and works more stable in my experience.

There is however a list of items to be followed carefully in order to be sure that everything works fine. The list may be hard to keep in mind comparing to steps in Windows, so I list them down below.

1. Check auditd daemon configuration to see if auditing service works fine (/etc/audit/auditd.conf)
2. Check audit event dispatcher configuration (/etc/audisp/audispd.conf)
3. Configure audit event dispatcher syslog configuration (/etc/audisp/plugins.d/syslog.conf)
4.  Create audit rules by editing /etc/audit/audit.rules file (More detail below)
5. Configure the syslog daemon to redirect log messages to a collector server.
6. Restart the daemons to activate the configurations.

I took RedHat family of Linux systems (RHEL, CentOS, Fedora,etc.) for configuration example in this article and configuration steps and commands apply to almost all Linux distributions with small changes.

There is not much to say about first and second steps, as they are routine controls to see if daemons are enabled and fine tunings may be done if necessary.

At the third step, under the syslog.conf file we should configure the args parameter to say what facilities we want to send to the syslog. To be coherent with the below configuration I set it as below:

args = LOG_INFO LOG_LOCAL5 LOG_LOCAL4 LOG_LOCAL3

Then comes a very important step, configuring the audit.rules file which actually is your audit policy for the server. If you are up to this point, most probably your company should already have one and your audit.rules file should not be empty. But in case you started being interested with linux servers just for the sake of log management (like me), I would suggest you to first read and edit, and then copy the usr/share/doc/audit-x.y.z/stig.rules document as your audit.rules file. stig.rules file is a really well prepared document to guide you to write your own rules and it is very good for a starter honestly. In my case the configuration applied was like below:

[root@localhost etc]# vi /usr/share/doc/audit-2.3.7/stig.rules

[root@localhost etc]# cp /usr/share/doc/audit-2.3.7/stig.rules /etc/audit/audit.rules

cp: overwrite `/etc/audit/audit.rules'? y

I can also suggest you to add below lines in your audit.rules file as a best practice. (For 64bit systems in this example)

-a always,exit -F arch=b64 -S sethostname -S setdomainname -k HOSTNAME_CHANGED

-a always,exit -F arch=b64 -S kill -F a1=9 -k KILL9

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S settimeofday -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S adjtimex -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -F subj_type!=ntpd_t -S clock_settime –k SYSTEM_TIME_CHANGED

-w /etc/localtime -p wa -k SYSTEM_TIME_CHANGED

-a always,exit -F arch=b64 -S mount -k DEVICE_MOUNTED

-a always,exit -F dir=/boot -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/root -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/etc -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/bin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sbin -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/lib64 -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/usr -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/net -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/sys -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/cgroup -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/selinux -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/adm -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/lib -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/cron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/at -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F dir=/var/spool/anacron -F perm=wa -k SYSTEM_FILE_CHANGED

-a always,exit -F path=/var/log/messages -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/dmesg -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

-a always,exit -F path=/var/log/secure -F perm=wa -F subj_type!=syslogd_t -F subj_type!=logrotate_t -k LOG_ALTERED

In the fifth step, we should configure the syslog daemon. In Linux systems, rsyslog service is responsible from reading the events and writing them to specific log files. To decide which actions are going to be logged /etc/rsyslog.conf file should be edited with a text editor.

More specifically RULES section in rsyslog.conf file should be edited like below:

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                                                          /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none                           /var/log/messages

# The authpriv file has restricted access.

authpriv.*                                                                                    /var/log/secure

# Log all the mail messages in one place.

mail.*                                                                                          /var/log/maillog

# Log cron stuff

cron.*                                                                                         /var/log/cron

# Everybody gets emergency messages

*.emerg                                                     

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                                                        /var/log/spooler

# Save boot messages also to boot.log

local7.*                                                                                   /var/log/boot.log

#Log Management

syslog.info; auth.info; daemon.info;                               @@CollectorServerIP

authpriv.info; cron.info; kern.info

# System log information

local5.info, local4.info, local3.info                                 @@CollectorServerIP

In the above configuration @ sign symbolizes log sending over UDP 514 port and @@ symbolizes TCP 514 port. In order to not to lose any logs I configured it over tcp. I have been told by a colleague recently that this may add a significant load on systems where number of logs are important, but I still believe that tcp method should be given a chance before switching to udp, if it is deemed inevitable.

If you want to complicate things you may choose to send your logs you may send them encrypted but that configuration is not a part of this article.

As a final step, we restart the auditd and rsyslog services. At this point we must be able to see on the collector server log messages arriving to the syslog server software installed.