After experiencing a variety of hardware failures recently, I finally completed my migration to syslog-ng with a central loghost paired with hourly log reporting using a wrapper script around logcheck. Once properly tuned, an ongoing process, mails generated and often of some interest.
To setup syslog-ng on machines on Debian Sarge, you need merely apt-get install syslog-ng and accept the removal of sysklogd and friends. Once that’s done, your regular log files will be written too as usual. syslog-ng as packaged for Debian effectively replaces the default sysklogd daemon without incident. To actually log to a loghost, you need only make a few additional changes.
On each client host that you wish to log to the loghost (generally in unision with local disk logging), you need to make two changes to your existing /etc/syslog-ng/syslog-ng.conf configuration.
First, add a new destination near the top of the section for destinations that points to your loghost. I prefer to use the IP as DNS may go down. Additionally, you may choose to use TCP instead of UDP for reliable message delivery. Due to an issue with TCP connection RST detection in 1.6.x I decided on UDP for my low traffic network instead.
destination remote_udp {
udp("192.168.55.5" port(514));
};
Next, you need to add the actual log target so data is actually logged. (Yes, all the semicolons and such are necessary.)
log{ source(s_all); destination(remote_udp); };
For the client, you’re done.
For your loghost, you will need some additional entries. The permissions and ownership are important if you want the default installation of Debian logcheck to be able to read your logfiles later.
options {
# logfile permissions
group(adm);
perm(0640);
# directory permissions at create time
dir_owner(root);
dir_group(adm);
dir_perm(0755);
# Provides $HOST as hostname, otherwise $HOST is IP address.
# syslog-ng blocks on DNS, so if you don't trust your DNS _disable_ this.
use_dns(yes);
}
source network {
udp(ip(0.0.0.0) port(514));
};
destination hosts {
file("/var/log/example.com/$HOST/$YEAR/$MONTH/$DAY/$FACILITY.$YEAR$MONTH$DAY"
owner(root) group(adm) dir_perm(0750) create_dirs(yes));
};
destination remote_udp {
udp("127.0.0.1" port(514));
};
# The network
log { source(network); destination(hosts); };
# Our loghost
log{ source(s_all); destination(remote_udp); };
Once you’ve got your loghost configured, you should start seeing some entries in your target directory.
sarah:~# echo "test" | logger sarah:~# echo "test" | logger -t test sarah:~# cat /var/log/example.com/sarah/2006/03/18/user.20060318 Mar 18 19:56:30 sarah logger: test Mar 18 19:57:17 sarah test: test sarah:~# ls /var/log/example.com/ faith nebula rebecca sarah
syslog-ng will ocassional give you a heartbeat regarding its activities, indicating how many, if any, log lines were lost.
syslog-ng[24288]: STATS: dropped 189
If you’re getting something other than zero, you may be logging to a destination that can’t keep up with your log volume or isn’t there. On a stock Debian Sarge install, the default syslog-ng configuration has an active destination to /dev/xconsole which will give you drops if you aren’t attached to it. Comment it out on every system for which you don’t use /dev/xconsole.
#log {
# source(s_all);
# filter(f_xconsole);
# destination(dp_xconsole);
#};
Once you have all those logs, you’ll probably want a method allowing you to watch them. Fortunately, you can simply install a copy of Debian’s logcheck.
sarah:~# apt-get install logcheck logcheck-database
logcheck will walk through each configuration file on your loghost with some encouragement, allowing you to receive timely emails on things it finds. By default, there are two different kinds of log entrieslogcheck will be looking for, security violations and system activity. The former is determined by keywords in /etc/logcheck/violations.d which you will come to love. The latter is whatever doesn’t match a security violation keyword and also isn’t explicitly ignored. So, you’ll be getting a lot of emails initially.
To encourage logcheck to look at your logfiles, assuming you used the expected tree under /var/log/example.com/..., you can download a copy of my logreports.pl script which has logcheck walk each configuration directory sequentially and email you with a report. You’ll want to replace the entry in /etc/cron.d/logcheck with a cron entry that runs logreports.pl instead. Also, make sure your modify the script to reflect your hostnames and logfile path.
$logbase = 'example.com'; my @hosts = qw/ foo bar hostname otherhost /;
Many messages that seem innocent will appear as security violations. Additionally, many system messages you really don’t care about will be reported to you. For best results, you must tune an exclude file for both security violations and system events so you receive only relevant alerts from logcheck.
The best description of how logcheck handles include and exclude pattern matches is included with the logcheck-database package.
sarah:~# zless /usr/share/doc/logcheck-database/README.logcheck-database.gz
Writing egrep regular expressions is all kinds of fun. By default, you will mostly be writing expressions that ignore either security violations or system events for which you have little interest in hearing about. Ideally, you will write tight patterns that match precisely what you want so you don’t accidently exclude critical events. For example, I don’t want to hear about restarts of syslog-ng during the log rotation window.
# syslog-ng tight restart window, for cron.daily and friends
^\w{3} [ 0-9]{2} 06:(2[0-9]|3[0-2]):[0-9]{2} [._[:alnum:]-]+ syslog-ng\[[0-9]+\]: SIGHUP received, restarting syslog-ng
Simply put, the above rule will match lines like the following, where the hostname, month, and day might vary only.
Mar 18 06:25:11 sarah syslog-ng[1164]: SIGHUP received, restarting syslog-ng
Most regular expression matches you write will by very similar in that the hostname, date, and time will all be wild cards, although you can get more creative as I did above and match a specific time window. Matching time windows can become arbitarily complicated as you need many regular expressions logically OR’d together. But you probably won’t need to worry about that starting out.
In either case, you’ll find it generally convenient to collect all your rules in two places, /etc/logcheck/ignore.d.server/logcheck-local and /etc/logcheck/violations.ignore.d/logcheck-local. The former will be used to ignore logfile entries that are reported by logcheck as being “System Events” while the latter will ignore logfile entries that are specifically flagged as being “Security Events”. If there is a log entry that’s being reported as a “Security Events” item and you want it to be demoted to a “System Events” item, I know of know obvious way to achieve that other than editing /etc/logcheck/violations.d/logcheck directory in such a way that the log entry in question isn’t flagged as being of a security nature.
In a default installation of logcheck you can safely choose to ignore cracking.*, ignore.d.paranoid, and ignore.d.workstation.
For testing logcheck as a whole, you’ll want to invoke it with testing and debugging options.
sarah:~# su -s /bin/bash \ -c "/usr/sbin/logcheck -tdo -l /var/log/internal.foo/sarah/2006/03/18/syslog.20060318" logcheck
For testing an individual rule, you can simply pass it to egrep yourself and see if it works. In this particular context, if a rule successfully matches, it will correctly exclude what is returned within the context of whether you include it in either /etc/logcheck/violations.ignore.d or /etc/logcheck/ignore.d.server. (The following is broken across multiple lines, but the actual regular expression must be on a single line.)
sarah:~# egrep -e \
'^\w{3} [ 0-9]{2} 06:(2[0-9]|3[0-2]):[0-9]{2} [._[:alnum:]-]+
syslog-ng\[[0-9]+\]: SIGHUP received, restarting syslog-ng' \
/var/log/internal.foo/sarah/2006/03/18/syslog.20060318 | wc -l
1
Sometimes you find things you weren’t even looking for. If you find you’re getting many instances of fetchmail complaining about CommonName mismatch it’s reasonably simple to resolve. In Debian Sarge the fetchmail excludes for the logcheck package matches the mismatch warning, so you will only see it when tailing the log or when logcheck is set to run in paranoid mode.
You’ll likely discover some of your own gems.
I have an exclude list for security events and for system events, if you want to jump start your exclude rules. Your network is doubtless different from mine, though.
You’ll also find excludes that Debian packagers feel are relevant for their packages under /etc/logcheck on each system you have. Not all packages ship with exclude files, but services generally do. Copy these files to your loghost so you don’t need to come up with them on your own.