In Red Hat Enterprise Linux 3/4/5, the default system log tool is syslogd which is provided by package sysklogd, but since Red Hat Enterprise Linux 6, the rsyslogd became the default. rsyslog package is also provided since Red Hat Enterprise Linux 5.2.
Syslog is the most common method used for computer message logging. It is the foundation for most security auditing applications. Syslog stores all the messages from every device on the network in a local repository for use by a network management system to give the administrator a network-wide view from a single station.
Syslog’s architecture is based upon the concept of messages and facility codes. The model is a Client/Server with the clients (logging hosts) sending messages up to a max 1024 bytes to the syslog server using either TCP or UDP on port 514.
A message is labeled with an indicator as to which one of the software types generated the message. These are actually simple categories syslog defines to allow it to handle messages differently and specifically for each type.
They are identified by keywords: auth, authpriv, cron, daemon, FTP, lpr, kern, mail, news, syslog, user, uucp local0 … local7.
The selector field itself again consists of two parts, a facility and a priority, separated by a period (``.'').
The facility is one of the following keywords: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth),syslog, user, uucp and local0 through local7.
There is no way to define your own facilities but there are many predefined ones (up to 23 in all, depending on which syslog you use):
auth (Security events get logged with this)
authpriv (user access messages use this)
cron (for cron, at, and anacron, but not for the programs started by cron)
daemon (other daemon programs without a facility of their own)
kern (kernel messages)
lpr (print system)
mail
mark (used by syslogd to produce timestamps in log files)
news
syslog
user (for user programs)
uucp (obsolete form of networking)
local0 – local7 (any use; RH uses local7 for boot messages)
* (for all)
Syslog is the most common method used for computer message logging. It is the foundation for most security auditing applications. Syslog stores all the messages from every device on the network in a local repository for use by a network management system to give the administrator a network-wide view from a single station.
Syslog’s architecture is based upon the concept of messages and facility codes. The model is a Client/Server with the clients (logging hosts) sending messages up to a max 1024 bytes to the syslog server using either TCP or UDP on port 514.
A message is labeled with an indicator as to which one of the software types generated the message. These are actually simple categories syslog defines to allow it to handle messages differently and specifically for each type.
They are identified by keywords: auth, authpriv, cron, daemon, FTP, lpr, kern, mail, news, syslog, user, uucp local0 … local7.
The selector field itself again consists of two parts, a facility and a priority, separated by a period (``.'').
The facility is one of the following keywords: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth),syslog, user, uucp and local0 through local7.
There is no way to define your own facilities but there are many predefined ones (up to 23 in all, depending on which syslog you use):
auth (Security events get logged with this)
authpriv (user access messages use this)
cron (for cron, at, and anacron, but not for the programs started by cron)
daemon (other daemon programs without a facility of their own)
kern (kernel messages)
lpr (print system)
mark (used by syslogd to produce timestamps in log files)
news
syslog
user (for user programs)
uucp (obsolete form of networking)
local0 – local7 (any use; RH uses local7 for boot messages)
* (for all)
The priority is one of the following keywords, in ascending order: debug, info, notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg, panic (same as emerg). The keywords error, warn and panic are deprecated and should not be used anymore.
The priority defines the severity of the message.
The priority is one of the following eight levels, which are ranked in order from high to low priority:
emerg
alert
crit
err
warning
notice
info
debug (or “*”)
An asterisk (``*'') stands for all facilities or all priorities, depending on where it is used (before or after the period). The keyword none stands for no priority of the given facility.
You can specify multiple facilities with the same priority pattern in one statement using the comma (``,'') operator.
Multiple selectors may be specified for a single action using the semicolon (``;'') separator.
Remember that each selector in the selector field is capable to overwrite the preceding ones.
Using this behavior you can exclude some priorities from the pattern.
You may precede every priority with an equation sign (``='') to specify only this single priority and not any of the above. You may also (both is valid, too) precede the priority with an exclamation mark (``!'') to ignore all that priorities, either exact this one or this and any higher priority. If you use both extensions than the exclamation mark must occur before the equation sign, just use it intuitively.
You may prefix each entry with the minus ``-'' sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner.
Usually critical messages are also directed to ``root'' on that machine. You can specify a list of users that shall get the message by simply writing the login.
You may specify more than one user by separating them with commas (``,''). If they're logged in they get the message. Don't think a mail would be sent, that might be too late.
Emergency messages often go to all users currently online to notify them that something strange is happening with the system. To specify this wall feature use an asterisk (``*'').
When specifying a priority, that and all higher ones are selected too. A selector is one or more facilities (separated by commas), a dot, then the priority. More complex selectors are possible too; one such is shown below.) Some example selectors:
mail.* mail facility, any priority
mail.debug mail facility, debug or higher priority (same as *)
mail,news.* all messages from mail or news
auth.warning all security messages of warning or higher priority
*.info all messages from any facility except debug msgs
*.=info any facility, info msgs only (and not higher)
*.!err any facility, pri <= err only
*.!=alert any facility, any priority except alert
*.info;mail,news,authpriv.none
all msgs with info or higher priority except
mail, news, and authpriv
That last one is tricky. Using multiple selectors on a single line this way allows you to specify a general category first, then for the matching log messages you can specify exceptions. Always go from most general selector to most specific or your setup may not log what you think it should!.
Examp:
Here are some example,
# Store critical stuff in critical
#
*.=crit;kern.none /var/adm/critical
This will store all messages with the priority crit in the file /var/adm/critical, except for any kernel message.
# Kernel messages are first, stored in the kernel
# file, critical messages and higher ones also go
# to another host and to the console
#
kern.* /var/adm/kernel
kern.crit @finlandia
kern.crit /dev/console
kern.info;kern.!err /var/adm/kernel-info
The first rule direct any message that has the kernel facility to the file /var/adm/kernel.
The second statement directs all kernel messages of the priority crit and higher to the remote host finlandia.
This is useful, because if the host crashes and the disks get irreparable errors you might not be able to read the stored messages.
If they're on a remote host, too, you still can try to find out the reason for the crash.
The third rule directs these messages to the actual console, so the person who works on the machine will get them, too.
The fourth line tells the syslogd to save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info.
Everything from err and higher is excluded.
# The tcp wrapper loggs with mail.info, we display
# all the connections on tty12
#
mail.=info /dev/tty12
This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to /dev/tty12, the 12th console.
For example the tcpwrapper tcpd(8) uses this as it's default.
# Store all mail concerning stuff in a file
#
mail.*;mail.!=info /var/adm/mail
This pattern matches all messages that come with the mail facility, except for the info priority. These will be stored in the file /var/adm/mail.
# Log all mail.info and news.info messages to info
#
mail,news.=info /var/adm/info
This will extract all messages that come either with mail.info or with news.info and store them in the file /var/adm/info.
# Log info and notice messages to messages file
#
*.=info;*.=notice;\
mail.none /var/log/messages
This lets the syslogd log all messages that come with either the info or the notice facility into the file /var/log/messages,
except for all messages that use the mail facility.
# Log info messages to messages file
#
*.=info;\
mail,news.none /var/log/messages
This statement causes the syslogd to log all messages that come with the info priority to the file /var/log/messages. But any message coming either with the mail or the news facility will not be stored.
# Emergency messages will be displayed using wall
#
*.=emerg *
This rule tells the syslogd to write all emergency messages to all currently logged in users. This is the wall action.
# Messages of the priority alert will be directed
# to the operator
#
*.alert root,joey
This rule directs all messages with a priority of alert or higher to the terminals of the operator, i.e. of the users ``root'' and ``joey'' if they're logged in.
*.* @finlandia
This rule would redirect all messages to a remote host called finlandia. This is useful especially in a cluster of machines where all syslog messages will be stored on only one machine.
CONFIGURATION FILE SYNTAX DIFFERENCES
Syslogd uses a slightly different syntax for its configuration file than the original BSD sources. Originally all messages of a specific priority and above were forwarded to the log file. The modifiers ``='', ``!'' and ``-'' were added to make the syslogd more flexible and to use it in a more intuitive manner. The original BSD syslogd doesn't understand spaces as separators between the selector and the action field.
Syslog Severity Levels
The message severity levels are standard and they are listed in seven levels from debug at level 7 being the least severe up to emergency at level 0.
Security Level Severity Description
-------------- ---------- ---------------
0 Emergency System is unusable.
1 Alert Action must be taken immediately.
2 Critical Critical conditions.
3 Error Error conditions.
4 Warning Warning conditions.
5 Notice Normal but significant condition.
6 Informational Informational messages.
7 Debug Debug-level messages.
Message Format
A Syslog message takes the form: < PRI > HEADER MSG
Length 0 – 1024 bytes
PRI – stands for priority.
<PRI > one byte
Severity level 3 bits 0-7 Facility 5 bits
The Priority level is calculated by (value of the Facility x 8) + severity level
Example: a facility value of kernel message = 0 x 8 + severity level 1 = priority of <1>
Header
The header contains a timestamp, which is the time the message was generated. It also holds the IP address of the host.
Message
Tag: – The name of the process that generated the message
Content: – The message body
Configuring Syslog on Linux
Configuring logging on a Linux system is very important as logs allow administrators to troubleshoot issues both at the application and network layers and without logs (alerts, errors and notifications) they would be working blind.
In order to configure Syslog the /etc/syslog.conf or the later /etc/rsyslog.cong file may need to be configured.
To configure Syslog you need to deal with the relevant ‘.conf’ file for your syslog version. With rsyslog that’s rsyslog.conf and
usually located at ‘/etc/rsyslog.conf’. The config file is where you specify things like:
The format of your log messages. what file events from the different facilities (i.e. different parts of your system) should be logged to. what file events with different severities should be logged to.
where to log your data – e.g. you can log to files, but you can also log to databases or forward your logs to a remote server/logging service.
filters – i.e. remove events that you do not want logged.
It is in this file where the logging levels and preferences are set.
# Sample syslog.conf file that sorts messages by
# mail, kernel, cron and “other”
# send mail, cron, and kernel/firewall msgs to
# their respective log files
mail.* -/var/log/mail
kern.* -/var/log/kernel_n_firewall
cron.* -/var/log/cron
# save the rest in one file
*.*;mail.none;authpriv.none;cron.none /var/log/messages
This file consists of two columns. The first lists the facilities and severities of messages to expect and the second lists the files to which they should be logged. By default, RedHat/Fedora's /etc/rsyslog.conf file is configured to put most of the messages in the file /var/log/messages.
Here is a sample:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
In this case, all messages of severity "info" and above are logged, but none from the mail, cron or authentication facilities/subsystems
You can make this logging even more sensitive by replacing the line above with one that captures all messages from debug severity and above in the /var/log/messages file. This example may be more suitable for troubleshooting.
*.debug /var/log/messages
In this example, all debug severity messages; except auth, authpriv, news and mail; are logged to the /var/log/debug file in caching mode. Notice how you can spread the configuration syntax across several lines using the slash (\) symbol at the end of each line.
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
Here we see the /var/log/messages file configured in caching mode to receive only info, notice and warning messages except for the auth, authpriv, news and mail facilities.
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
You can even have certain types of messages sent to the screen of all logged in users. In this example messages of severity emergency and above triggers this type of notification. The file definition is simply replaced by an asterisk to make this occur.
*.emerg *
Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:
Files:
/var/log/maillog : Mail
/var/log/httpd/access_log : Apache access logs
Directories:
/var/log
/var/log/samba : Samba messages
/var/log/mrtg : MRTG messages
/var/log/httpd : Apache webserver messages
Note: In some older versions of Linux the /etc/rsyslog.conf file was very sensitive to spaces and would recognize only tabs.
The use of spaces in the file would cause unpredictable results. Check the formatting of your /etc/rsyslog.conf file to be safe.
Activating Changes to the syslog Configuration File
Changes to /etc/rsyslog.conf will not take effect until you restart syslog.
Managing the syslog daemon is easy to do, but the procedure differs between Linux distributions. Here are some things to keep in mind.
Firstly, different Linux distributions use different daemon management systems. Each system has its own set of commands to do similar operations.
The most commonly used daemon management systems are SysV and Systemd.
Secondly, the daemon name needs to be known. In this case the name of the daemon is rsyslog.
Armed with this information you can know how to: Start your daemons automatically on booting Stop, start and restart them later on during troubleshooting or when a configuration file change needs to be applied.
Common Linux log files names and usage
/var/log/messages : General message and system related stuff
/var/log/auth.log : Authenication logs
/var/log/kern.log : Kernel logs
/var/log/cron.log : Crond logs (cron job)
/var/log/maillog : Mail server logs
/var/log/qmail/ : Qmail log directory
/var/log/httpd/ : Apache access and error logs directory
/var/log/lighttpd/ : Lighttpd access and error logs directory
/var/log/boot.log : System boot log
/var/log/mysqld.log : MySQL database server log file
/var/log/secure or /var/log/auth.log : Authentication log
/var/log/utmp or /var/log/wtmp : Login records file
/var/log/yum.log : Yum command log file.
