2007 Feb 26 - Mon

Event Logs: Some Stuff To Try At Home

Somewhere in my collection of Monitoring Server Configs, I have some information on getting Cisco syslog stuff into a separate file.

For another customer site, I used Nagios coupled with Steve Shipway's Nagios EventLog agent for Windows to collect specific Windows Events and alarm on them. It was interesting and convoluted experience to get all this working. If there is interest, I'll post my process notes on how I got the whole thing integrated.

Today, or rather originally a week ago, an ISP requested that I forward some router syslog events to them so they could correlate their events with mine, or vice versa, my events with theirs. (ok, contrary to one of my recent articles, some ISP's do see the light of day in troubleshooting, although this same one hasn't quite grasped the IP SLA bonus yet).

Anyway, two routers, the endpoints on an mpls link, in two different regions behind two different firewalls are at issue. The knee jerk reaction is to add a second syslog entry in each router to forward to the ISP's syslog address. This will require, in addition to the second entry in each router, entries in each firewall.

That seemed silly. I thought: why not just forward the syslog entries from the server instead. Well, not so easy with the standard sylogd daemon in Debian.

Some searching lead to a number of interesting alternatives. The one slated for immediate testing is to try BalaBit's syslog-ng where I can forward based upon more refined rules such as host and message content. This is a simple Debian apt-get upgrade. DebianHelp offers some instrucitons for installation and use with php=-syslog-ng. Jeremy Mates's syslog-ng blog discusses some further syslog-ng configuration details. As a sidetrack, his blog also has some stuff for sendmail and other Linux Geek stuff. As a point of reference, one more syslog-ng site is cudeso.be.

Once I've got syslog-ng going, and the cisco log entries forwarded, I'm thinking about stopping event log watching with Nagios and Shipway's thing and instead trying Intersect Alliance's Snare Agent for windows sending syslog events to syslog-ng. I see they also know how to do stuff with Snort Logs, Apache Logs, and others.

Something I see a lot of words about but no real specifics is Splunk. They proclaim to be able to scan and correlate and do queries on logs from many servers. That could be an interesting tool, and free for systems with daily log files under 500M.