Настройка spamassassin, postgrey, clamav

EFFECTIVE TRAINING

You need to train with both spam and ham mails. One type of mail
alone will not have any effect.

Note that if your mail folders contain things like forwarded spam,
discussions of spam-catching rules, etc., this will cause trouble. You
should avoid scanning those messages if possible. (An easy way to do this
is to move them aside, into a folder which is not scanned.)

If the messages you are learning from have already been filtered through
SpamAssassin, the learner will compensate for this. In effect, it learns what
each message would look like if you had run "spamassassin -d" over it in
advance.

Another thing to be aware of, is that typically you should aim to train
with at least 1000 messages of spam, and 1000 ham messages, if
possible. More is better, but anything over about 5000 messages does not
improve accuracy significantly in our tests.

Be careful that you train from the same source — for example, if you train
on old spam, but new ham mail, then the classifier will think that
a mail with an old date stamp is likely to be spam.

It’s also worth noting that training with a very small quantity of
ham, will produce atrocious results. You should aim to train with at
least the same amount (or more if possible!) of ham data than spam.

On an on-going basis, it is best to keep training the filter to make
sure it has fresh data to work from. There are various ways to do
this:

1. Supervised learning

This means keeping a copy of all or most of your mail, separated into spam
and ham piles, and periodically re-training using those. It produces
the best results, but requires more work from you, the user.

(An easy way to do this, by the way, is to create a new folder for
‘deleted’ messages, and instead of deleting them from other folders,
simply move them in there instead. Then keep all spam in a separate
folder and never delete it. As long as you remember to move misclassified
mails into the correct folder set, it is easy enough to keep up to date.)

2. Unsupervised learning from Bayesian classification

Another way to train is to chain the results of the Bayesian classifier
back into the training, so it reinforces its own decisions. This is only
safe if you then retrain it based on any errors you discover.

SpamAssassin does not support this method, due to experimental results
which strongly indicate that it does not work well, and since Bayes is
only one part of the resulting score presented to the user (while Bayes
may have made the wrong decision about a mail, it may have been overridden
by another system).

3. Unsupervised learning from SpamAssassin rules

Also called ‘auto-learning’ in SpamAssassin. Based on statistical
analysis of the SpamAssassin success rates, we can automatically train the
Bayesian database with a certain degree of confidence that our training
data is accurate.

It should be supplemented with some supervised training in addition, if
possible.

This is the default, but can be turned off by setting the SpamAssassin
configuration parameter "bayes_auto_learn" to 0.

4. Mistake-based training

This means training on a small number of mails, then only training on
messages that SpamAssassin classifies incorrectly. This works, but it
takes longer to get it right than a full training session would.

Quota

See Dovecot Quota. Enable overall quota plugin in /etc/dovecot/conf.d/10-mail.conf:

mail_plugins = $mail_plugins quota

Dovecot Quota Backend

Dovecot quota plugin can relay on several backends: dirsize means calculate the actual disk space used every time (very slow!!), maildir means voluntary quota via maildirsize file, etc. We will instead use fs, the quota enforced by the Linux filesystem. In /etc/dovecot/conf.d/90-quota.conf set:

plugin {
  quota = fs:User quota
}

Dovecot Quota service for Postfix

We enable a service that will be checked by Postfix when accepting mail via SMTP. Into /etc/dovecot/conf.d/90-quota.conf add the following:

service quota-status {
  executable = quota-status -p postfix
  inet_listener {
    port = 12340
  }
  client_limit = 1
}

In the same configuration file, set also the responses returned by the service. Notice that if you set a grace quota space of 10%, you must set the filesystem soft quota limit 10% smaller than the hard one; or even more, to be safe:

plugin {
  quota_grace = 10%%
  quota_status_success = DUNNO
  quota_status_nouser = DUNNO
  quota_status_overquota = "554 5.2.2 Quota exceeded (mailbox for user is full)"
}

After reloading dovecot.service, you will have a daemon listening on port 12340/TCP. To check if the service is working, you can telnet to the TCP port and paste some strings of text, the relevant lines are recipient and size in bytes (see more on smtpd-policy language in this example):

request=smtpd_access_policy
protocol_state=RCPT
protocol_name=SMTP
sender=foo@bar.tld
recipient=user@domain.org
client_address=1.2.3.4
client_name=another.domain.tld
size=123000

action=554 5.2.2 Quota exceeded (mailbox for user is full)

In Postfix configuration /etc/postfix/main.cf add a restriction on smtpd (the order of restrictions matter):

smtpd_recipient_restrictions =
    ...
    check_policy_service inet:127.0.0.1:12340,
    ...

Quota and dovecot-lda

dovecot-lda, on local deliveries, will chek user quota too. Debian default is to include global mail_plugins, so no need to add quota into /etc/dovecot/conf.d/15-lda.conf:

protocol lda {
  mail_plugins = $mail_plugins
}

Dovecot Quota for IMAP

To enable quota support into the IMAP server, add to /etc/dovecot/conf.d/20-imap.conf:

protocol imap {
  mail_plugins = $mail_plugins imap_quota
}

Verify that QUOTA is displayed among CAPABILITY(ies), using telnet to port 143/TCP:

telnet localhost 143
a1 LOGIN username SuperSecret
a1 OK  Logged in
a2 LOGOUT

Checking Email Header and Body with SpamAssassin

SpamAssassin ships with many spam detection rules in /usr/share/spamassassin/ directory. Allow me to explain some of the rules.

In the /usr/share/spamassassin/20_head_tests.cf file, you can find the following two lines.

header MISSING_HEADERS       eval:check_for_missing_to_header()
describe MISSING_HEADERS     Missing To: header
header __HAS_DATE            exists:Date
meta MISSING_DATE            !__HAS_DATE
describe MISSING_DATE        Missing Date: header
header __HAS_FROM            exists:From
meta MISSING_FROM            !__HAS_FROM
describe MISSING_FROM        Missing From: header

You might want to use the Cron job shipped with SpamAssassin to automatically update SpamAssassin’s rules on a daily basis. If so, open the file and change to .

[править] Общие сведения

Статья предназначена для системных администраторов или опытных технических специалистов, и дает представления о возможности реализации и совмещения встроенной системы фильтрации спама хостинга 1Gb.ru со своей собственной, используя при этом только ресурсы хостинга.

  • управление через web-интерфейс;
  • простота управления;
  • белые/черные списки;
  • настраиваемые простые контекстные фильтры.

Минусы:

  • Отсутсвие «весов» при оценке заголовков, содержимого письма;
  • Отсутсвие байесовой фильтрации.

Поэтому под эффективной фильтрацией мы будем понимать не хороший спам-фильтр на стороне сервера, а их совокупность или комплекс мер применяемых для отсеивания нежелательной почты.

ManageSieve Daemon

Install the dovecot-managesieved Debian package, verify that the daemon is responding on port 4190/TCP. The protocol is enabled by the included snippet /usr/share/dovecot/protocols.d/managesieved.protocol.

Test the ManageSieve Daemon

See ManageSieve Troubleshooting. It is possible to test a ManageSieve daemon talking the proper language over an interactive TCP/IP session. The language is specified by RFC5804.

To authenticate ourself to the server, we’ll need a base64 encoded string containing \000login\000password:

echo -en "\000niccolo@mail.domain.org\000MySecret" | base64
AG5pY2NvbG9AbWFpbC5kb21haW4ub3JnAE15U2VjcmV0
telnet localhost 4190

In this session we will authenticate, check if there is space to store a script, create a script and activate it:

CAPABILITY
AUTHENTICATE "PLAIN" "AG5pY2NvbG9AbWFpbC5kb21haW4ub3JnAE15U2VjcmV0"

HAVESPACE "myscript" 200
PUTSCRIPT "myscript" {109+}
require ;
if header :contains "Subject" "newsletter"
{
    redirect :copy "other@domain.org";
}

LISTSCRIPTS
SETACTIVE "myscript"

LOGOUT

To calculate the size of the script (109 bytes in our case), use cr+lf as newline separator.

Connect Roundcube to the ManageSieve Daemon

Install the Debian package roundcube-plugins, which cointains the managesieve plugin. Edit the /etc/roundcube/plugins/managesieve/config.inc.php file, and set:

$config=array();
$config'managesieve_port' = 4190;
$config'managesieve_host' = 'localhost';
$config'managesieve_default' = '/etc/dovecot/sieve/global';

Enable the plugin adding it to the array in /etc/roundcube/config.inc.php:

$config'plugins' = array(
    'managesieve'
);

After configuring the two files, login into the Roundcube web interface, click on Settings, you should have a new tab labeled Filters on the left (after Preferences, Folders, etc.)

Roundcube, ManageSieve, sieve_extprograms and Include

It seems that Roundcube with ManageSieve plugin, cannot handle scripts which use the sieve_extprograms plugin or the include directive. The filters are not properly shown into the web interface, and they gets corrupted when the plugin edit/rewrite the script.

Dovecot IMAP and POP3

Install the Debian packages:

  • dovecot-imapd

  • dovecot-pop3d

Once installed, the basic services IMAP2 on port TCP/143 and POP3 on port TCP/110 are already working, eventually with the Dovecot Authentication on users as seen above.

Enable SSL, imaps and pop3s

See SSL Dovecot Configuration.

We need to enable SSL certificates for encryption, and enable IMAPS on port TCP/993 and POP3S on port TCP/995. We will use the same Let’s Encrypt certificates obtained for the web server. So we set into /etc/dovecot/conf.d/10-ssl.conf:

ssl = yes
ssl_cert = </etc/letsencrypt/live/smtp.domain.org/fullchain.pem
ssl_key = </etc/letsencrypt/live/smtp.domain.org/privkey.pem

Dovecot will read the SSL files with root privileges, so no particular settings should be required.

Once restarted the dovecot.service, we will find the services listening on the well known ports, and we can check the SSL certificates.

GETTING STARTED

First a high-level overview:

Build a significant sample of both ham and spam.

I suggest several thousand of each, placed in SPAM and HAM directories or
mailboxes. Yes, you MUST hand-sort this — otherwise the results won’t be much
better than SpamAssassin on its own. Verify the spamminess/haminess of EVERY
message. You’re urged to avoid using a publicly available corpus (sample) —
this must be taken from YOUR mail server, if it is to be statistically useful.
Otherwise, the results may be pretty skewed.

Use this tool to teach SpamAssassin about these samples, like so:
        sa-learn --spam /path/to/spam/folder
        sa-learn --ham /path/to/ham/folder
        ...

Let SpamAssassin proceed, learning stuff. When it finds ham and spam
it will add the «interesting tokens» to the database.

If you need SpamAssassin to forget about specific messages, use the —forget option.

This can be applied to either ham or spam that has run through the
sa-learn processes. It’s a bit of a hammer, really, lowering the
weighting of the specific tokens in that message (only if that message has
been processed before).

Learning from single messages uses a command like this:
        sa-learn --ham --no-sync mailmessage

This is handy for binding to a key in your mail user agent. It’s very fast, as
all the time-consuming stuff is deferred until you run with the "--sync"
option.

Autolearning is enabled by default

Learning from user actions

There used to be a nice Dovecot plugin called “Antispam” but unfortunately it became deprecated. Bummer. The currently recommended way is to use the “IMAPSieve” plugin instead. There is nothing to install – it comes with the Dovecot packages. We just need to configure it.

First order of business is enabling the IMAPSieve plugin for the IMAP protocol/service in Dovecot. Edit the /etc/dovecot/conf.d/20-imap.conf file and look for the line reading “mail_plugins”. Turn it into:

And anywhere within the { … } section add these lines:

Note: if you come from previous versions of this guide, your mail directories may still use dots (“.”) instead of paths (“/”) to compose the folder structure. A clear sign is that you find directories like “INBOX.Junk” instead of just “Junk” in mail directories within /var/vmail. In that case replace the word “Junk” in the section above by “INBOX.Junk”.

Next up let’s create the Sieve scripts that we told Dovecot about. Create a new directory /etc/dovecot/sieve to put our new files in:

Then create the file /etc/dovecot/sieve/learn-spam.sieve and let it contain:

Let’s do the same for /etc/dovecot/sieve/learn-ham.sieve

These two scripts need to be compiled – that is turning them into machine-readable code:

This creates two new files “learn-ham.svbin” and “learn-spam.svbin” that look like gibberish inside but are now in a format that Dovecot’s Sieve plugin can understand.

Let’s fix the permissions of these files, too, while we are at it:

And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is /etc/dovecot/sieve/rspamd-learn-spam.sh which contains:

The second script teaches ham and is called /etc/dovecot/sieve/rspamd-learn-ham.sh. Make it contain:

These two shell scripts must be executable so go fix that:

I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process works:

  1. a user moves a spam email into their “Junk” folder
  2. Dovecot realizes that this triggers the Sieve rule “imapsieve_mailbox1” so it calls the Sieve script /etc/dovecot/sieve/learn-spam.sieve (in fact the *.svbin version of the script)
  3. Sieve will take the email and send (“pipe”) it to the executable rspamd-learn-spam.sh shell script
  4. the script in turn runs the email through the “/usr/bin/rspamc learn_spam” command

I am sure you are eager to try it out. Yes, it’s about time. However to see that it actually works I suggest you edit the /etc/dovecot/conf.d/10-logging.conf file and set “mail_debug=yes”. That will add a lot more detail to the /var/log/mail.log file but on a busy server may also lead to headaches.

Restart Dovecot…

…and watch the /var/log/mail.log file…

Look for errors and warnings. If at the end you see that Dovecot called the “rspamd-learn-spam.sh” script then you should be fine.

That’s it. Nifty, isn’t it?

Step 3: Configure Postfix

We are going to configure Postfix to handle the SMTP connections and send the messages for each user introduced in the MySQL Database.

First we need to create a copy of the default file, in case you want to revert to the default configuration.

Open the main.cf file to modify it:

First we need to comment the TLS Parameters and append other parameters. In this tutorial, we are using the Free SSL certificates and the paths that are suggested in the tutorial (link), but you could modify depending your personal configurations.

Then we are going to append the following parameters below the TLS settings that we have changed in the previous step:

We need to comment the default settings and replace it with . This change allows your VPS to use the virtual domains inside the MySQL table.

Verify that myhostname parameter is set with your FQDN.

Append the following line for local mail delivery to all virtual domains listed inside the MySQL table.

Finally, we need to add these three parameters to tell Postfix to configure the virtual domains, users and aliases.

Note: Compare these changes with this file to detect mistakes or errors:

We are going to create the final three files that we append in the main.cf file to tell Postfix how to connect with MySQL.

First we need to create the file. It’s necessary to change the values depending your personal configuration.

Then we need to restart Postfix.

We need to ensure that Postfix finds your domain, so we need to test it with the following command. If it is successful, it should returns 1:

Then we need to create the mysql-virtual-mailbox-maps.cf file.

We need to restart Postfix again.

Finally, we are going to create the last file to configure the connection between Postfix and MySQL.

Restart Postfix

We need to uncomment these lines and append other parameters:

In some cases, we need to restart Postfix to ensure port 587 is open.

Note: You can use this tool to scan your domain ports and verify that port 25 and 587 are open (http://mxtoolbox.com/SuperTool.aspx)

Set Message Maximum Size

By default, SpamAssassin does not check messages with attachments larger than 500KB, as indicated by the following line in the file.

spamc: skipped message, greater than max message size (512000 bytes)

The default is set to 512000 (bytes). A high value could increase server load, but I think the default size is a little bit small. To increase the max-size, edit file and add the following lines at the end.

#Spamc options
OPTIONS="${OPTIONS} -- --max-size=5120000"

The empty option tells spamass-milter to pass all remaining options to spamc, which understands the option. I increased the size to 5000KB. Save and close the file. Then restart spamass-milter.

sudo systemctl restart spamass-milter
Добавить комментарий

Ваш адрес email не будет опубликован. Обязательные поля помечены *

Adblock
detector