Screaming Penguin - Source Does Matter

Setting up a mailserver on Linux starts out very easy and then gets exponentially more complicated as you say "I want to do X."

Setting up a good, one, nay a great one, nay an AWESOME one, is possible but is also somewhat tricky. Ok so now "awesome" needs defined in this context. Mailserver needs to support virtual hosts (easy), mailserver needs to do secure mail (pretty easy), mailserver needs to do SMTP_AUTH (not so easy), mailserver needs to filter spam and get the hell rid of it and optionally mailserver needs to be able to scan for viruses (another not so easy).

Now that I have all this working like a charm I thought I would share a bit about it in case anyone else finds it useful. Note that I have planned for a few weeks to do a full out "HowTO" on the matter but havent had time to do so and dont forsee the time for that in the near future, so this is the quick version of what I did with links to helper material (and maybe the step by step detailed version will be available someday, maybe not).

I started down this path a few months ago, just researching and testing, etc. What mailserver to use? What anti-spam products are available and what to use? What security stuff? I ended up using Sendmail for the main SMTP server because thats what I have always used and it has always kicked ass. I heard Qmail and Postfix are good too, but I am sold on Sendmail. I have also always used UW for imap and pop servers (this article not really about that side of it, but just for the record, lots of people like to complain about the performance of UW imap/pop, but thats because they dont know how to configure it and then blame the product, its great stuff when you know what you are doing) so I stayed with those as well. Then after much research on spam stuff I decided to use
SpamAssassin and in order to use that with Sendmail I needed a milter (mail filter) and I chose MimeDefang for that. Note that MimeDefang also fit the bill as to scanning for viruses, nice.

I wont get into to much detail about why I chose these specific products but suffice to say it was mainly for their great reputation and interoperability with Sendmail. SpamAssassin and MimeDefang are totally incredible open source products that outdo what many commercial thousands of dollar products do by a mile. Great stuff.

For the security side I researched SMTP_AUTH and decided it was a great idea. I had never used it before but basically it lets users authenticate to an SMTP server (which is not normally the case) and then relay based on that authentication. That means all the silly relay rules based on domain name and crap can go away, authenticate and you can relay. In order to use SMTP_AUTH with Sendmail you need the Cyrus SASL layer. SASL stands for Simple Authentication and Security Layer, it connects to external mechanisms to validate the authentication (it can use password, shadow, pam, kerberos, ldap, etc). In addition with the security stuff I wanted to encrypt the SMTP sessions using SSL. Sendmail does this via STARTTLS, which stands for Start Transport Layer Security

SO I had decided I needed Sendmail with STARTTLS and SMTP_AUTH (which needs SASL) and MimeDefang/SpamAssassin. It was admittedly a jumble of stuff but it wasnt really all that difficult to pull it all together (it took a few trys mind you, but its doable).

Ultra High Level How To:

• Prerequisites: OpenSSL, Perl, compiler stuff such as GCC (most all Linux machines have this stuff)
• Get source code for all these products and put in /usr/src
• Install SASL first and MAKE SURE IF WORKS by using the test server and client. Install it normally, see the install stuff that comes with it, and make sure to disable-gssappi and disable-kerberos if you dont have/want those. Then make sure you have the libraries you intended to have in /usr/lib/sasl (login, plain, digest, etc, whatever method you want it to be able to use, must be there). Then test using the sample server and client, this was THE MOST HELPFUL thing for me and its not mentioned at all in any of the related howtos (see samples/readme stuff for info on how to test). Whatever methods you can authenticate with using the test server and client are what you can later use with Sendmail (LOGIN, PLAIN, DIGEST-MD5, CRAM-MD5). If one is not available on the test server and client it AINT gonna work with Sendmail. Dont even bother moving on to anything else until SASL does exactly what you expect it to do.
• Compile Sendmail with custom config files for the first time adding STARTTLS support. This is the SSL stuff. It does require you to make a certificate but thats pretty easy especially if you are using RedHat. Just follow the HowTo, very straightforward. Be familiar with how to compile Sendmail and how to use the installer or manual M4 to generate the configuration files. Its really not very hard and you will need to do it several more times to add more options.
• Compile Sendmail again with custom config files by MERGING your existing config files that already have the STARTTLS stuff in them. This time for support with SASL using these instructions on the Sendmail SMTP_AUTH pages. Make sure Sendmail works and responds with the AUTH options you expect before moving on. Mega Thanks to Claus Aßmann.
• Setup MimeDefang per the MimeDefang HowTo. Install all the required perl modules and SpamAssassin as it states you should. It again requires you to compile Sendmail with new options, do so with the new and the previous STARTTLS/SASL/STMP_AUTH options all merged. When you are done you will have config files for MimeDefang and SpamAssassin in several places. You can change those later to suit your filtering/anti-spam/virus needs. Make sure Sendmail and MimeDefang/SpamAssasin work properly. Start and stop, use the SpamAssassin tests, etc. Note that if you just want to filter SPAM you DONT need to install a virus scanner. Also note that you DONT need spamc and spamd, dont even worry about it, these are in the SpamAssassin docs but are not of use when using with MimeDefang, MimeDefang will spawn processes as needed to handle things and doesnt need the spamd stuff. Mega thanks to Mickey Hill.
• Rock on with your kick ass Sendmail SMTP server that now supports STARTTLS/SMTP_AUTH and has MimeDefang/SpamAssassin milter spam and virus utilties.

Note that once you are setup there is a list of clients that support SMTP_AUTH and its pretty extensive. Outlook supports it, Netscape/Mozilla supports it, etc. One caveat is that Outlook uses the LOGIN auth mech which is not valid per the RFC, its there basically just for Outlook (whats new) and Netscape/Mozilla uses the PLAIN mech. The most useful setup then is to use LOGIN and PLAIN as your available auth mechanisms (this goes back to how you setup SASL). HOWEVER THIS IS, UH, AS IT STATES, PLAIN TEXT so dont use it without STARTTLS or some other method to encrypt the channel (unless you are aware of the vulnerability and dont give a crap, note that most POP and IMAP servers are not encrypting the channel either and in most cases its the same username/password across the wire). Sendmail CAN BE CONFIGURED to not even show the AUTH mechanism unless the channel is encrypted (see the linked "JoreyBump" article for details).

All in all its a GREAT setup that I now use for supporting several hundred users. The flexibility of SpamAssassin allows me to NOT drop any mail but simply add a header to it if it is deemed spam so that clients can then do what they want with it (drop it or put it in a different folder, etc). Having the SPAM blocking stuff and a reliable way to allow relay even for "roaming" users makes a, let me say it again, GREAT, mail server. Thanks to the open source community for providing such incredible products.

One more note, if you don't want to go through all the setup, you can BUY this stuff all ready to go from Roaring Penguin. Checkout CanIT. It comes with GUI menus, stats reporting, support, etc. I of course advocate doing it yourself, so that you will understand what all is going on, but this is a good alternative if you dont agree.   sial.org: smtpauth-starttls talk