Jason's Web Thingy: Basic Procmail Stuff

Items of Interest

An exploration of procmail

Procmail has nearly been around since the dawn of time itself. Its archiac syntax is seemingly unintuitive at first. However, over time I've come to appreciate its quirks and its strengths. Therefore, tonight's exodus to Maude's is dedicated to you, procmail.

Procmail: Not the programming you expected

First, when you write procmail recipes, you're writing individual rules that look at things and perform actions. In fact, you can look at exactly one or more things and perform exactly one action, in each recipe. Second, it's important to note that you cannot easily have complex nested conditionals. In procmail, that kind of thing can quickly balloon out of control.

A typical procmail recipe consists of a flags header, one or more matching rules, and an action. It lives in ~/.procmailrc for a user and in /etc/procmailrc, optionally, for gobally defined rules. As an example, a basic recipe could look like the following:

:0 :
* ^Subject.*Meeting
meetings

The first portion is the flags section. The leading zero and colon have long since ceased to have a purpose, but is required for all rules. In the past other numbers were used. The second colon tells procmail to create a lock file before performing the action specified. This is very important to prevent multiple instances of procmail from trying to perform the action simultaneously. (You can omitt it in some circumstances, but more on that later.)

The second portion, denoted by the asterisk ('*'), is the matching rule. The rules are egrep compatible regular expressions. If you're familiar with Perl regular expressions, you'll find these are similar. If not, you ought to check out the egrep man page for details; You'll get tripped up quickly without a basic understanding of regular expression concepts. You can have additional matching rules, each on a new line and preceded by an asterisk. Above, the expression matches a line starting with the word 'subject', zero or more characters, and the word 'meeting'. You'll notice my capitalization here differs from the example. That's fine. Procmail ignores case unless you explicitly tell it otherwise.

The final line is the action. In this case, I specified the name of a file. Procmail will append any mails that match the specified matching rule to that file in Mbox format. You can do other neat things as well, like pipe the mail to a program or email it to another user.

Often, you'll want more than a single rule in your procmail file. Procmail works by comparing the headers of each email to each procmail rule, starting with the first recipe defined. When it finds a match, it will perform the recipe's specified action. If the action is considered a deliverable action, processing stops and the mail is considered delivered. A deliverable action is one that pipes an email to a program, writes it to a file, or emails it somewhere. A nondeliverable action is one in which the mail is sent to a filter. The result of a filter is usually an altered email returned from a program.

With that background, let's look at a real world example of a global procmail file I created. It contains things uncovered until now, like flags, logging, and variables. We'll take it one section at a time. This global file is owned by root, and procmail executes as the owner of its recipe file.

# Logging

LOGFILE=/var/log/procmail.log
VERBOSE=1
LOGABSTRACT=1

First, we set up logging. LOGFILE, VERBOSE, and LOGABSTRACT are all procmail internal variables you can set. The output is extremely detailed and should have you track down and problems with recipes not working as you antipicated. The procmail man page lists many additional internal variables you can set.

SPAMC="/usr/bin/spamc -f"
SPAMUSER="spamuser@example.com"

Variables! Procmail supports variables. Above I have defined a program to run, in this case SpamAssassin's spamc executable, and an email address. These are regular variables that have no special meaning.

:0
* ^X-Local
{ }

:0 wfE
* ^From .*\.example\.com
* ^To:.*\.example\.com
* ^Received: from.*example\.com.*by.*example\.com
| formail -f -b -a "X-Local: Yes"

Neither recipe above has a lock specified. This is okay, because neither rule writes to a file. Thusly, locking isn't needed. You can lock anyway, however, and only be penalized with a small performance penalty.

The first recipe above is an example of a null action recipe. The space in between the two curly braces is necessary, or an error will occur. It's essentially an empty action. Let's look at the next recipe that immediately follows it to see why.

The second recipe above is the first to use flags. Flags must follow the colon zero token and come before the colon for locking, if one is used. The specific flags above, in order from left to right, tell procmail to wait for the program to finish, treat the program as a filter, and only look at the recipe if the one immediately preceding it fails to match. (In retrospect, I could have combined both recipes, but the above works too.) The filter flag, specified by the letter f, tells procmail to consider the pipe as a filter and therefore this action is not a deliverable action. Instead, formail, part of the procmail suite of mail tools, adds the header I specify to the mail and then returns the mail to procmail. (Actually, only the header is looked at by default. You can specify capital letter B as a flag to have procmail look at the message body too.)

As a whole, the above set of recipes check to see if the email was sent within the example.com domain, and if so, flags the mail as internal. This is important later when extra checks are performed on mail, like passing it to SpamAssassin. Local mail is spared.

:0
* ! ^X-Local
{

So, if the mail is not local, we proceed. The above recipe's action is actually a block context, within which additional recipes live. The remaining recipes all live within this block, so at this point mail that is local gets passed to no additional recipes and is delivered (or subjected to the intended recipient's procmail file). If no recipes match a mail, the default delivery to /var/mail/$LOGNNAME occurs.

  :0
  * ^TO_(manager(s)?|nobody)
  * ! ^From.*example\.com
  {
    :0 wf
    *
    | formail -f -b -a "X-SpamChecked: Yes"

    :0
    *
    ! $SPAMUSER
  }

The above collection of recipes is much like the larger collection it lives within. The first recipe above initiates the block only for mails that are sent to the address managers@example.com and not sent from an address at example.com. If those conditions are satisfied, the next two recipes are processed. Both are examples of recipes with blank matching conditions. Both match on anything. The first of the two is another filter, nearly identical to the one seen earlier. The second has an action not yet seen before. An action prepended by an exclaimation point will deliver the mail to the specified address. In this instance, it's actually a variable that is expanded and used. This is a deliverable action and execution stops after the action occurs.

The result is undesirable mail to the managers@example.com address is forwarded to the $SPAMUSER's account for further action, if any.

In my case, the delivery occurs to an address on the same server. This puts an additional load on the server as each mail sent to managers@example.com that doesn't have a matching internal From: address is reinjected into the mail system. A better approach which I use now is the following:

  :0
  * ^TO_(manager(s)?|nobody)
  * ! ^From.*example\.com
  {
    :0 wf
    *
    | formail -f -b -a "X-SpamChecked: Yes"

    :0
    *
    | /usr/bin/procmail -d $SPAMUSER
  }

The action prepended by the exclaimation point has been replaced with a pipe. You'll notice this time there is no filter flag. We want this action to be deliverable, which it is by default without a filter flag. Now the mail is redirected to the $SPAMUSER for a delivery using a direct call to, you guessed it, procmail.

  :0 fw
  * ! ^X-SpamChecked
  | $SPAMC

  :0 wA
  * ^Subject:.*\*\*\*\*SPAM\*\*\*\*
  {
    :0 wf
    *
    | formail -f -b -a "X-SpamChecked: Yes"

    :0
    *
    ! $SPAMUSER
  }

Finally, we make the call out to SpamAssassin. First, we verify that the mail has not already seen SpamAssassin by checking against our internal flag. The filter flag is set here so the altered mail returned by SpamAssassin can be later delivered or discarded as necessary.

The next block is more interesting as it makes use of the 'A' flag. With that flag set, the next action, which is a block, is not executed unless the previous recipe's matching rule matched. In this case, the block only executes if we have not yet flagged this mail as being passed to SpamAssassin. Assuming the SpamChecked flag is set, the first recipe in the block is flagged to be a pipe and will execute for any mail it sees. Formail is once again called to add a header to the mail and pass it back to procmail. Finally, the mail is sent to the $SPAMUSER by way of the exclaimation point prefixed action.

} # end of local mail block

If the mail originally had the X-Local flag set, none of the above recipes would have been used and instead we would simply end up here at the end of the first block. In either case, processing of this global procmail file is now complete and user rules, if any, will not execute for any mails that make it to this point without reaching a deliverable recipe.

You can do far more complex things in procmail. If you'd like to look at some serious procmail, check out Catherine Hampton's SpamBouncer, which is designed to keep spam at bay. You can also check out the procmail mailing list archive.

Copyright and Revision Information

03-23-2003 - Initial draft
03-30-2003 - Draft completion