Quarantine spam email

Quarantine spam email

Quarantine and recover spam message for SpamAssassin, using procmail.

SpamAssassin if very powerfull for taging possible spam messages, but what should be done with such a message is left to the installer choice.

To answer the demand of a number of my users, I set-up a way to quaratine the spam messages, while offering them a way to automatically recover any message they consider could be a false positive (amont the first moth of testing, less than 1 of false positive was reported).

This is done through a four steps method, described bellow. The method has been written and tested for the purpose of CSIM system only, so you can use it but at your own risks.

This procedure will nicely interface with AMaViS quarantine system too, allowing users to recover any message that could have been tagged as containing a virus, but would still have some important information in the message body.

Step one, receive and quarantine

Quarantine is implemented by procmail, in a format very similar to the interfacing with procmail as described in SpamAssassin distribution.

:0fwE | /usr/local/bin/spamc -u $LOGNAME :0e { EXITCODE=$? } DATE=`date +%Y%m%d%H%M` PID=`echo $$` SPAMFILE=$DEFAULT-spam.$DATE.$PID :0: * ^X-Spam-Status: Yes $SPAMFILE

Example of procmailrc file to quarantine incoming spam

Incoming spam is ketp in one file per message, in /var/mail, the filename is of the format user-spam.date.pid. Where user is the user name of the recipient and date and pid are used to compose a unique file name.

There may be a better way to create the date and pid , but that was inspired from procmail documentation project.

Step two, generate a summary

Following Perl script is used to generate a summary of quarantined spam messages.

This script has been modified:

to correct a mistake in browsing the spam directory
to sort the repport by level of spam-iness

#!/usr/local/bin/perl chdir "/var/mail"; $interval=60*60; $today=time(); while (@user=getpwent) { # for HUMAN users, with group ID in the range [20-99] next if $user[3] <20 and $user [3]>=100; $user=$user[0]; opendir DIR, "."; undef $body; undef %spam; undef %from; undef %date; undef %subject; while ($file=readdir DIR) { # look for files with the name user-spam.date.pid next unless -f $file; next unless $file=~/^$user-spam\.\d+\.\d+$/; ($a, $b, $c, $d, $e, $f, $g, $h, $i, $mtime, $r)=stat $file; next unless $today-$mtime>$interval; # check that the file is at least 60 minutes old # this add a 60 minutes delay for a file # to be summarized but should remove every problem # of locking undef $subject; undef $date; undef $from; undef $spam; open IN, "/usr/local/bin/formail -bYczf -X From: -X Subject: -X Date: \ -X X-Spam-Status: < $file |"; # extract the From:, Date:, Spam-status: and Subject: of the # quarantined mail while (<IN>) { $subject=$_ if /^Subject: /; $date=$_ if /^Date: /; $from=$_ if /^From: /; $spam=$_ if /^X-Spam-Status: /; } # I keep the information in hash array for further sorting $subject{$file}=$subject if defined $subject; $date{$file}=$date if defined $date; $from{$file}=$from if defined $from; if ($spam=~/^X-Spam-Status: Yes, (score|hits)=(\d+(\.\d+)?) .*$/) { # extract the spam level $spam{$file}=$2; } # move the file out of the way system "/bin/mv -f $file /var/spamassassin/spam/$file"; } closedir DIR; undef $i; foreach $key (sort { $spam{$a} <=> $spam{$b} } (keys %spam)) { # generate the sorted report $i++; $body.=sprintf("%3.0d ", $i); $body.="File: $key\n"; $body.=" Spam Level: $spam{$key}\n" if defined $spam{$key}; $body.=" $date{$key}" if defined $date{$key}; $body.=" $from{$key}" if defined $from{$key}; $body.=" $subject{$key}" if defined $subject{$key}; $body.="\n"; } if (defined $body) { # add the begining of the report and send the email $body="You have received email(s) that is suspicious spam and was quarantined. Quarantined messages are kept for 30 days before they are automatically removed (http://www.cs.ait.ac.th/laboratory/email/spam.shtml#spam). If you wish to see any of the following message, reply to this email,including the lines with the File: information below. The word File: MUST be in your reply message, along with the filename. ".$body; open MESS, "|/usr/sbin/sendmail -t"; print MESS "From: quarantine\@cs.ait.ac.th\nTo: $user\@cs.ait.ac.th Reply-To: quarantine\@cs.ait.ac.th\nSubject: Quarantined messages\n\n"; print MESS $body; close MESS; } }

Perl script to prepare the summary and move the messages to the quarantine directory

It only applies to users with a groupid in the range [20-99] which for me corresponds to real users.

It will locate every file in /var/mail that have a name in the format user-spam.date.pid and extract the Subject:, From: and Date: headers from the message. This along with the filename consists of the summary of a quarantined message.

Once it has been summarized, the quarantined message is moved to /var/spamassassin/spam.

To avoid that SpamAssassin tries to proceed the summary email, I added the following rule at the begining of the procmail configuration file:

:0 : * ^From root * ^Subject: Quarantined messages * ^Reply-To: quarantine@cs.ait.ac.th $DEFAULT

Example of procmailrc shortcut SA

Every email sent by root, with the Subject: Quarantined messages and a Reply-To: quarantine@cs.ait.ac.th is delivered immediately in the users' $DEFAULT mailbox.

This script run by crontab, as often as you deam necessary. But it should not be so often as it would generate one summary per spam message. Else for the user it would defeat the purpose. I found out that once every 6 hours is fine for me.

The user will receive a summary that looks like:

Date: Mon, 18 Mar 2002 16:42:35 +0700 (ICT) From: quaratine@cs.ait.ac.th To: on@cs.ait.ac.th Reply-To: quarantine@cs.ait.ac.th Subject: Quarantined messages You have received email(s) that is suspicious spam and was quarantined. Quarantined messages are kept for 30 days before they are automatically removed. If you wish to see any of the following message, reply to this email,including the lines with the File: information below. The word File: MUST be in your reply message, along with the filename. 1 File: on-spam.200203181540.77885 Spam Level: 9.3 Date: Mon, 18 Mar 2002 15:40:00 +0700 (ICT) From: Mail Delivery Subsystem <MAILER-DAEMON> Subject: Postmaster notify: see transcript for details

Example of summary for quarantined spam

Step three, allow users to recover quarantined message

Users can recover any quarantined message, by sending an email to quarantine@cs.ait.ac.th where he mentions the identifier of the message you want to recover. Identifier of a quarantined message is a line of the form:
File: user-spam.date.pid.

To: quarantine@cs.ait.ac.th In-reply-to: <200203180942.g2I9gZM82106@mail.cs.ait.ac.th> (quaratine@cs.ait.ac.th) Subject: Re: Quarantined messages References: <200203180942.g2I9gZM82106@mail.cs.ait.ac.th> --text follows this line-- > 1 File: on-spam.200203181540.77885 > Spam Level: 9.3 > Date: Mon, 18 Mar 2002 15:40:00 +0700 (ICT) > From: Mail Delivery Subsystem <MAILER-DAEMON> > Subject: Postmaster notify: see transcript for details

Example of email send for recovering a quarantined message

Such email are procced by a special receipt in procmail:

:0 *^TOquarantine | /usr/local/sbin/recover-spam :0 : * ^From quarantine@cs.ait.ac.th $DEFAULT

Example of procmailrc call the recovery script

The first receipt calls the Perl script /usr/local/sbin/recover-spam for every email sent to qurantine. The second receipt makes sure that an email recovered by the following procedure is pushed without further checking in SA.The Perl script is as follow:

#!/usr/local/bin/perl while (<STDIN>) { # skip the header, until a blank line, keep only the From: $from=$1 if /^From\s+([\w\.\-]+)/; last if /^$/; } while (<STDIN>) { next if /File: information/; next if /File: MUST/; next unless /File: ([\w-\.]*)/; # in the message body, locate every line with # "File:" followed by a valid file name of the form # user-spam.date.pid $file=$1; $user=$file; $user=~s/-spam\.\d+\.\d+//; next unless $user eq $from; # check that the user mentionned in the filename # is the same as the user that sent the email # don't allow one to recover messages from others $file="/var/spamassassin/spam/$file"; next unless -f $file; system "/usr/bin/procmail -f quarantine\@cs.ait.ac.th -Y -d $user<$file"; # use procmail to push the file in the users' mailbox unlink $file; # remove the file as it has been delivered

Perl script to recover quarantined message

The script skips the headers, it only keeps the username of the sender of the recover message, for control purpose later on.

Then in the body of the message it will locate every line of the form
File: user-spam.date.pid.
and will push the corresponding file in the users' mailbox, only if the user is the same as the sender of the recover message.

Once a quarantined message has been recovered, the file is removed.

Step four, expire quarantined messages

Old quarantined messages are expired when they are more than 30 days old. The following Perl script will do the job:

#!/usr/local/bin/perl $interval=31*24*60*60; # 2,678,400 seconds $today=time(); chdir "/var/spamassassin/spam"; opendir DIR, "."; while ($file=readdir DIR) { next unless -f $file; next unless $file=~/-spam\.\d+\.\d+$/; ($a, $b, $c, $d, $e, $f, $g, $h, $i, $mtime, $r)=stat $file; next unless $today-$mtime>$interval; unlink $file; # remove any file older that $interval seconds }

A perl script to expire old quarantined spam

This is run by cron, as often as you deam necessary.

Complete synopsis of procmail configuration file

To sumamrize, the procmail configuration file is as follow:

# recover quarantine messages :0 *^TOquarantine | /usr/local/sbin/recover-spam # email pushed by the quarantine recovcery are delivered immediately :0 : * ^From quarantine@cs.ait.ac.th $DEFAULT # do not check the summary messages :0 : * ^From root * ^Subject: Quarantined messages * ^Reply-To: quarantine@cs.ait.ac.th $DEFAULT # check one message with smapc/spamd :0fwE | /usr/local/bin/spamc -u $LOGNAME :0e { EXITCODE=$? } DATE=`date +%Y%m%d%H%M` PID=`echo $$` SPAMFILE=$DEFAULT-spam.$DATE.$PID # quarantine in /var/mail/user-spam.date.pid :0: * ^X-Spam-Status: Yes $SPAMFILE