Building a Scalable Mail System? 109
clusteredMail asks: "I work for a small ISP that up until now has survived with single servers for most critical roles, including the mail server. We are planning to introduce multiple mail servers (primarily for email collection via POP3 and IMAP) and want to put in place the most scalable, resistant to failure system that we can manage. Everything is currently running on one or another flavour of Linux. In my mind, the ultimate scenario would be to have some sort of distributed/clustered file system between the multiple machines, so that any user could log onto any server, and the loss of a single server would not cause downtime for any group of users. Has anyone in the Slashdot community had to put together a system like this using Linux and Open Source Software? If so, how did you fare and what were the major stumbling blocks?"
"So far, the plan is to split up the mail accounts between multiple servers and use some sort of connection proxy to sort out which account logs into which server but this seems like a rough approach. The disadvantage to this setup: if one server fails all the users who have accounts on that machine will be in the dark, email-wise."
Check out Perdition (Score:5, Informative)
You mean, like Cyrus? (Score:5, Informative)
It's not for the faint of heart, but only takes a couple of "a-ha!" moments to go from lost to competent. Good luck!
Use NFS (Score:3, Informative)
Scaling can be done easily by adding more NFS boxes and managing the directory structure with links or whatnot.
Re:You mean, like Cyrus? (Score:4, Informative)
However if you used the Murder as your frontend for clients, and applied fairly standard high availability tactics to the individual backends you could achieve clustering. Make each backend server a redundant load balanced virtual server, then make the Murder know about the mailbox locations on the virutal systems.
I'm sure it could be done, but its definitely not something that Cyrus does out of the box.
In practice the multiple servers w/ proxy has been good enough for CMU. With good hardware for the backend servers, and good RAID arrays, hardware failures are rare.
For starters... (Score:5, Informative)
Layer 4/7 switching (Score:2, Informative)
Using this the only thing your servers need in common is backend storage that you can easily mount off NFS etc.
My solution (Score:5, Informative)
Data stores were maildirs on NetApps
SMTP servers running Postfix
IMAP servers running Courier IMAP
Logins via NIS
IMAP and SMTP failover by means of load balancers
The SMTP and IMAP servers get NIS-distributed automounter tables, so everyone's homedir is available everywhere. The load balancers distribute the load out to the SMTP and IMAP servers, and work around any that fail. Mail comes into the SMTP servers, and Postfix delivers to maildirs in the users' homedirs. Any SMTP server can deliver to any user. Users log in with IMAP on the Courier IMAP servers. Again, all homedirs are everywhere, so it doesn't matter which server they hit.
Adding capacity at any point is easy - you just add more servers of the appropriate type when you need more. IMAP and SMTP are fully redundant. Load balancers usually only operate in failover pairs, but you can add more A records in DNS for more LB pairs if you need it.
The one sticky point is the data stores on the NFS servers. Adding capacity is easy (just add more servers). but there's no easy way to make this fully redundant. See notes for more.
So there you have it. That'll scale to a pretty large system, and it's simple to implement. It's not THE MOST scalable system, but if you have to ask, this is probably sufficient for your needs.
Notes:
You must use maildirs, not mbox. Maildirs perform very well even on NFS, because there can be multiple simultaneous readers and writers. mbox requires locking.
With NetApp, or Red Hat Cluster Server, or any other cluster NFS server, you can make the head end redundant, so your disk shelf becomes the last single point of failure. If you run RAID 1+0, you can have all the disks mirrored across two shelves, so at least the hardware is completely redundant. However, there are still rare, but possible failure modes. STONITH is, ultimately, a problem that has no perfect solution. (Look it up if you're not familiar with STONITH.)
NetApp makes very reliable NFS servers. Even in single head configurations my uptime experience has been incredibly good. Dual head is even better. But they're god awful expensive. There are other ones you can buy at all different price points. Clustered file systems like Coda sound really sexy, but they're still half baked. Lustre http://www.lustre.org/ [lustre.org] might work well, but it wasn't available when I last did this, so I can't say. Choose what's appropriate to your needs and budget.
I used NIS. These days LDAP is more fashionable. Make your LDAP server redundant of course.
You need redundant networks. In the simplest case, put half of each type of servers (IMAP, SMTP, LB, NFS) on two different switches.
I never bothered with POP, but you can get POP servers for maildirs, too.
Configure your load balancers to balance per session - IE, if a user creates multiple IMAP connections, they all go to the same server. This helps keep down the number of NFS mounts, LDAP requests, etc.
Software opinions: I like Postfix and Courier. They're simple, robust, flexible enough for most situations, and perform very well. Cyrus also has a good following in the large-scale arena, but does things different. Qmail's non-OSS license prevents people from releasing versions that strip out djb's quirky way of doing things, which is why I left it for Postfix (and never looked back). Sendmail doesn't suck as much as it used to, but I haven't really seen why I SHOULD use it these days either. Any of these can be made to work, though, so use whatever you're comfortable with.
Tip for any email system: outright reject (IE, don't accept at all, don't send to someone's spam folder) as much spam as you can. If 90% of your mail is spam, and you reject the 90% most-likely-spam (delivering the other 10% more questionable stuff to a spam folder), you've just increased your mail performance and disk space by > 5x.
Good luck!
A typical way to set this up. (Score:3, Informative)
High availability redundant NFS servers for storing the mailbox data and user information.
One or more machines mounting this file-system for handling POP, IMAP, and SMTP from accounts and mailfolders off the NFS server.
Webmail can be tricky because you need to make sure that either users always hit the same machine for webmail during a session, or session information is shared among the cluster. LVS systems can handle either of these scenarios, so it's not a problem, just something you have to be aware of.
LVS systems up front, again running High Availability which do load-balancing and automatic removal of failed servers. These are the machines that have the IPs which your customers contact, and then get spread across the real machines in the middle layer above.
This sort of solution works really well, and we have deployed it for customers of ours with good results. You can get started for only $5k to $10k worth of hardware and if you're building this from scratch it will probably only take you around 100 hours. If you have experience with this sort of setup it can take as little as 10 to 20.
If $5k to $10k for hardware is out of your budget, you probably shouldn't be looking at this sort of solution. Individual stand-alone servers or even a single pointy box, possibly with high availability, is probably where you want to be in that case.
linux-ha.org is the place to go for High Availability software on Linux.
Sean
A Real-world Big Design (Score:5, Informative)
Before I feed you the design, let me tell you a *crucial* concept that you must carry with you at all times.
EMAIL SYSTEMS ARE PROTOCOL SPEAKERS BETWEEN USER DIRECTORIES AND STORAGE.
Read that and inwardly digest it before you even start to design your system.
For the design, first, I'm going to proselytize a particular piece of software.
DOVECOT IS THE FREE POP/IMAP SERVER OF THE FUTURE. It leaves the Cyrus codebase rotting in the slime. It already kicks Courier's butt in performance and ease of deployment. It's beautifully coded; it has the most elegant authentication architecture; it's exceptionally fast. It isn't complete yet but it's featureful and stable enough that I have successfully deployed 1.0-betas into production. http://www.dovecot.org/ [dovecot.org] for the last IMAP server you'll ever need.
Here is the design:
1 x OpenLDAP 2.3 master server
2 x OpenLDAP 2.3 read-only replicas
2 x world-facing mail servers running Postfix 2.3
4 x mail scanning servers running amavisd-new 2.3.3, ClamAV, SpamAssassin, Sophos SAVI and Sophos PMX-ENGINE. LMTP in from the mail front-ends; ESMTP out to the mail storage.
2 x mail storage front-ends running Postfix 2.3 and Dovecot IMAP/POP3 1.0-beta. These servers also run mysql for amavisd-new quarantine and squirrelmail user options. Actual storage is over NFS to the NetApps. Using Dovecot's Sieve-based delivery agent for server-side filtering.
2 x Squirrelmail webmail servers. We have our own skin, and our own sqm plugins as the user interface to our various system options - which are all in LDAP. We have integrated MailZu into sqm as a quarantine view/release interface.
2 x NetApp FAS3020c heads w/4TB NFS storage allocated to mail.
Everything is load-balanced using foundry hardware LBs. It's very high-throughput and very reliable. It's also easy to monitor (we're using Nagios).
Base OS is Debian Sarge with applicable backports. I'd prefer FreeBSD but this happens to be a Debian shop, and I wasn't out to change their world, just their mail system.
Probably the most borderline item is mysql's performance as a quarantine DB; however much RAM and index/query tuning we throw at it, I'm yet to be satisfied with InnoDB's performance on this 100GB+ INSERT-heavy database.
If I could change one thing about it, it'd be to use the extremely pretty and surprisingly good value @mail (a commercial choice) rather than SquirrelMail. I'd also consider Fedora Directory Server over OpenLDAP, but it wasn't looking ready for this design at the time.
I have to say there is some bad advice in this thread; now for the hatchet:
Cyrus: difficult to configure, doesn't support shared storage, horribly ugly codebase, and has some nasty-ass failure modes.
Qmail: stale, poorly integrated MTA software from the bitchiest developer in town.
Sendmail: doesn't scale. Even the developers think so, which is why Sendmail X is a rip-off of postfix.
Communigate Pro: if I don't get to futz with the source for integration and value-add, I'm not interested.
GFS/GPFS: you don't need the complexity or interesting failure modes of shared-block-storage filesystems. Stay away.
Linux NFS: isn't reliable enough. We've had problems with data corruption to Linux NFS, both kernel and userland. Right now the only NFS server implementations I trust are NetApp's and Solaris's. No doubt the Linux one can/will improve, or already has, but trust is a hard thing to build
Re:For starters... (Score:4, Informative)
You shouldn't state this as an absolute, because it's not. You also need to give reasons WHY to use maildir.
An example exception case: We had an application where thousands of very small emails needed to be delivered to a single mailbox every minute. They all get picked up every minute by POP, and all messages are deleted every cycle. mbox is *vastly* better in this scenario, because you don't have to create all the files, move them around a few times, stat large dirs every time POP runs, etc. With mbox, all the delivery threads become sequential, so you cut down seek overhead, and the POP read becomes a single large file read, which is far faster. You also cut way down on metadata updates, and caching works better.
mbox shines in this scenario, and it's not that uncommon. Many customer service apps work like this.
In the situation of handling many users's email in a scalable system Maildir is usually better (NFS-safe, concurrent delivery, efficient individual message deletion, etc), but you've not even considered the other range of things available. MH and database backends come to mind. Each has their good and bad points.
Re:I wrote my SMTP/POP servers to handle this prob (Score:3, Informative)
We use open source software throughout our system and contribute back most of our changes (where they actually have some utility outside our little world, 50 line perl programs that just query out database for status information need not apply - and we wouldn't want to inflict our web framework on the world. It certainly doesn't need another web framework with a steep learning curve and funky special cases!)
Right as I type this (or at least when I stop typing and get my arse back to working on what I'm being paid to do) we're setting up a replicated environment with pairs of Cyrus servers:
* Dual Xeon with hyperthreading
* 8Gb RAM
* 12 SATA drives configured as 4 arrays:
[73Gb RAID 1] [73Gb RAID 1] [1.2Tb RAID5] [1.2Tb RAID5]
Each array is then split in half, with the first partition holding an active Cyrus partition and the second half holding a replicated set from its "pair" server. This spreads IO evenly.
The small RAID1 sets are faster disks and they hold the metadata partitions which get most of the IO.
We don't have these things in production yet (still huge IBM monster machines with 6Gb memory and terabytes of attached SATA storage. They're much more reliable but don't have replication, so it's a tradeoff that makes an array failure much more painful to recover from).
Every so often I do wish I had the time to build a full IMAP server from scratch with a modern indexed database engine (boo hiss in the direction of Berkeley DB) and the capability to store multiple copies of files. I've already done something like that with our virtual filesystem that provides DAV and FTP access to files on the servers, as well as websites viewing parts of your filespace - it's shiny and I can reboot any server in the VFS backend set without worrying about impacting production.
Mail:Toaster (Score:3, Informative)
http://www.tnpi.biz/internet/mail/toaster/ [tnpi.biz]
it's qmail/imap based and scales quite well in my experience.