This month's tip comes from Zenoss Community member Scott Haskell. This tip provides detailed documentation for implementing email acknowledgement with Postfix, Procmail and Python. Originally posted on his blog Technology, Food, Life, he updated the tip and posted it to the Zenoss : Email Acknowledgment - Postfix, Procmail and Python
Background
Event driven network monitoring systems rely on acknowledgment for multiple reasons. These reasons include:
- To let other Zenoss administrators and users know that someone is aware of the problem and taking action.
- To suppress escalation of alerts.
To acknowledge an alert, a responder typically:
- Clicks the acknowledge link in the body of the default Zenoss alert email.
- Logs into the Zenoss server, navigates to the alert on the device or in the event manager, and acknowledges the alert.
This process works well when you're in the office or your operations are business hours only. Typically, this is not the case for most users of Zenoss. Most users have critical systems that require 24x7 monitoring and alerting. Additionally, managers and executives require escalation of messages from the on-call personnel through management and sometimes to executive management.
The acknowledgment model of Zenoss, in its current state, is limited. An alert will never escalate, as long as it is acknowledged. In a 24x7 monitoring and alerting model, an alert can and will come during non-business hours. The first responder (on-call personnel) is usually mobile and does not always have immediate access to a computer. To compound this, many Zenoss servers live on private networks that require VPN authentication to access them. There are times where the on-call personnel can not get into the system quick enough to prevent escalation from occurring. We all know how upset managers get when the on-call grunt doesn't do her job properly.
One possible solution to this issue is email based event acknowledgment. The on-call responder replies to the alert email on her PDA. The alert email is routed back to the Zenoss server for processing with Procmail. The body of the email is piped into a python script, via Procmail. The device and event id are extracted and the alert is acknowledged through Zenoss API calls. An email is sent back to the responder reporting whether or not the acknowledgment was successful.
The biggest caveat, for this model, is that incoming mail needs to be routed to your Zenoss server. As such, this will require knowledge of mail relay administration or you'll need to rope in your mail relay administrator to help you out. Since every company's mail setup is unique (Spam filters, Exchange Servers, Postfix Servers, et. al) I will leave this as an exercise for the user of this how-to. The second caveat is that your mail relay becomes a single point of failure for delivery. If your mail relay is down or in schedule maintenance, you won't be able to acknowledge events.
** This will work with plain-text, base64 encoded and multipart email types (Iphone, Blackberry, Exchange). I have tested this on Zenoss Core 2.4.5 and 2.5.0
Installation
Postfix Installation
Install Postfix from RPM or source.
Procmail Installation
Install Procmail from RPM or source and configure per the README.
Configuration
Postfix Configuration
Edit your configuration file to use procmail. This is typically located at:
vi /etc/postfix/main.cf
Edit mailbox_command to reflect the path of your Procmail installation.
mailbox_command = /usr/bin/procmail
Restart Postfix
/etc/init.d/postfix restart
Procmail Configuration
We will be filtering email at the user level, not the system level. As the zenoss user:
vi ~/.procmailrc
chmod 600 ~/.procmailrc
Create a procmail directory to log to.
mkdir ~/procmail
Copy and paste these two sections into your .procmailrc. There can be space between them. The first section exports basic environment variables.
SHELL=/bin/bash
PMDIR=$HOME/procmail
LOGFILE=$PMDIR/pmlog
ZENHOME=/usr/local/zenoss
PYTHONPATH=$ZENHOME/lib/python
PATH=$ZENHOME/bin:$PATH
The next section is the Procmail formula. This says for any message that contains '[zenoss]' in the subject, forward the header and body (as STDIN) to /home/zenoss/zenemailack.py and redirect STDOUT and STDERR to $ZENHOME/log/zenemailack.log. This formula also locks and waits until the script has finished. You can relocate zenemailack.py anywhere on your system, just update the path in the formula accordingly.
:0
* ^Subject:.*\[zenoss\].*
{
:0 hbW:
| /home/zenoss/zenemailack.py >> $ZENHOME/log/zenemailack.log 2>&1
}
Zenoss UI Configuration
Under the 'settings' tab, from the 'settings' link on the main navigation menu (left navbar), change this value to reflect the valid email address to your zenoss user.
From Address For Emails zenoss@server.mydomain.com
* You must be able to receive emails at this email address on your Zenoss server.
Zenoss Server Configuration
Download the attached script below (zenemailack.py) and place it in /home/zenoss. You can install the file to any location of your choice. If you use a different location, make sure to update the Procmail formula to reflect the new path of your installation.
Change permissions to make the script executable.
chmod 755 ~/zenemailack.py
Edit zenemailack.py and change these variables to suit your environment:
# Update for outgoing email
MAIL = "/usr/sbin/sendmail -t "
from_addr = "zenoss@server.mydomain.com"
cc_address = ""
Testing
General
The first step is to make sure you can receive emails at zenoss@server.mydomain.com. If you can't receive a basic email, resolve this or you're out of luck.
Procmail
Once you can successfully receive emails, make sure that procmail is working. You can create a test formula in your .procmailrc to test that it's working correctly.
vi ~/.procmailrc
Append this to your .procmailrc and save it.
:0
* ^Subject:.*test
/dev/null
As zenoss user, test Procmail functionality by sending yourself an email with 'test' as the subject.
echo "foo" | mailx -s "test" zenoss
Check the procmail log for success.
tail ~/procmail/pmlog
From zenoss@localhost Wed Sep 3 16:11:26 2008
Subject: test
Folder: /dev/null 595
If you have no entry for this test in your Procmail log, Procmail is not working properly. Check your installation, double check that Postfix is configured to use Procmail and that you re-started Postfix after you configured it to use Procmail.
Re-edit ~/.procmailrc and remove the test formula you added above. Your .procmailrc should look like this after.
SHELL=/bin/bash
PMDIR=$HOME/procmail
LOGFILE=$PMDIR/pmlog
ZENHOME=/usr/local/zenoss
PYTHONPATH=$ZENHOME/lib/python
PATH=$ZENHOME/bin:$PATH
:0
* ^Subject:.*\[zenoss\].*
{
:0 hbW:
| /home/zenoss/zenemailack.py >> $ZENHOME/log/zenemailack.log 2>&1
}
Zenoss
Once you've verified that you can receive emails and Procmail works, test this with an actual event.
If you're impatient, you can inject an event into Zenoss. To do this, you can follow the admin guide. Just make sure your event matches an alertable condition.
Admin Guide - 7.1.8.1 – Creating Events Through the User Interface
If you're not impatient, just wait around until you get an alert. Once you receive the alert, simply reply. There is no need to type anything in your reply.
Make sure that your mail editor includes the body of the alert when you reply. I don't know of any that don't, by default, but I'm just throwing that in there.
You should receive a verification email back saying that the alert has been acknowledged or not. You’ll only receive a verification reply if the alert is in an unacknowledged state or something crapped out during the attempted acknowledgment. If you don’t get a reply email, the alert most likely cleared itself and has been moved to history, by Zenoss. Once an event has been acknowledged, any reply-to-all email threads will not generate email noise. They will, however, be logged.
For debugging or troubleshooting, you can view the log at:
$ZENHOME/log/zenemailack.log
Further configuration and debugging information may be found at the wiki entry: Email Acknowledgment - Postfix, Procmail and Python, which will have any updates to the instructions going forward. Thanks again to Scott Haskell for providing this tip!
