Sending email from syslog events
Historically, I have been monitoring my F5 devices using syslog and Zenoss. Syslog on the F5 devices forwards to my Zenoss system, which then parses the alerts by severity, source, and regex string comparison (node up/node down).
My Zenoss system died, so I started looking for other methods of performing the same tasks. Initially I looked at Nagios because we are using it to monitor our apps, sadly there doesnt appear to be native syslog monitoring, which means that solution is extra difficult.
I found the site, http://www.johnandcailin.com/blog/john/how-setup-real-time-email-notification-critical-syslog-events, that described how to make syslog email on its own. Below are the key steps needed to make this work
configuring syslog to write to a named-pipe
first, create a named-pipe for critical messages, for example:
# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessagesnext, configure syslog to log all critical messages written to the local0 facility to this pipe. add the following statement to your syslog.conf file.
local0.crit |/etc/syslog.pipes/criticalMessagessending out messages
the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i’ve included an example below, let’s call it /usr/bin/syslogMailer:
#!/bin/bash
# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
# example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#
alertRecipient="fireman@example.com" # the mail recipient for alerts
TMOUT=1 # don't wait > 1 second for input
# process each line of input and produce an alert email
while read line
do
# remove any repeated messages
echo ${line} | grep "message repeated" > /dev/null 2>&1
if test $? -eq 1
then
# send the alert
echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
fi
done
cron
# m h dom mon dow command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1
Next step is to have this working with systemd startup scripts.