Slashdot Log In
Ask Slashdot: Building a Large Email Service
Posted by
Cliff
on Thu Jul 29, 1999 04:52 PM
from the built-it-and-they-will-come dept.
from the built-it-and-they-will-come dept.
Rewd asks:
"I'm looking at implementing a large scale email server (cluster) to handle POP3 and IMAP4 for about 25000 people, including a lot of attachments. I'd like to go for an Open Source solution, but a lot of people around here want to go for Microsoft Exchange on NT.
Has anyone here successfully built anything like this? Can you recommend any combinations and components which are particularly
efficent, capable, secure and reliable?"
This discussion has been archived.
No new comments can be posted.
Ask Slashdot: Building a Large Email Service
|
Log In/Create an Account
| Top
| 484 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Cyrus probably a good bet. (Score:3)
The Cyrus server at CMU is probably your best bet. You'll find it at at this link [cmu.edu].
It's worth noting that this project is currently supporting all of CMU's e-mail needs. It's also my understanding that it forms the basis for Netscape's Message Server and Post.Office. This should satisfy any concerns about it's scalability. It has lots of handy features like kerberos authentication, a database style message repository, support for ACAP, etc.
Alternatively try QMail [qmail.org]. Personally, while I think it provides better SMTP performance than Sendmail, I'd rather use the Cyrus IMAP server than the UW one (the only one supported by QMail). You could go with using a combo of sendmail|postfix + Cyrus for incomming mail (i.e. what your MX records point to) and QMail for outgoing mail. It depends on your performance needs
Exchange Server is NOTORIOUS for being both difficult and expensive when you need it to scale to a large number of users, although I understand it's improved substancially since the 4.x days when it was just impossible.
Run, do not walk, away from Exchange. (Score:5)
MS would like people to believe that Exchange is an enterprise-level communications tool, when it fact it is a buchered and bloated decendant of a mediocre 1992 X.400 email system from Data Connection Limited (check out http://www.datcon.co.uk/press/messserv.h tm [datcon.co.uk]) Don't believe the version number; Exchange is in its second major release (4.x really is 1.x, 5.x = 2.x, etc) and still has significant stability problems. [slashdot.org]
In my experience, Exchange can support 300 users per server happily on commonly acceptable x86 corporate server hardware (say, a 2 processor PII with 512mb ram). It seems that (in my limited experience, lest MS lawyers take this to be a declaration of fact, which it is not) once you've reached this level, doubling the ram and adding more cpu's has only a minimal effect, which means that you really have to add more servers to add capacity.
Let's do the math. 25,000 users at 500 users per server (to be quite generous) means that you're going to need a Windows NT server farm of about 50 systems just to do email. Again, being generous bargain hunters, let's say you can buy one of these servers for $10kUS. That means you're out $500,000 just for hardware. In my experience, you can support 500 POP users easily on a SPARC 2 or IPX, which can be had these days for about $500 decked out (including a 17" monitor). You could support the same (probably many more) on a $500 x86 box running any of the free *nixes. Assume you blow $500 on disk storage for these boxen just to level the starting line, bringing the total cost to $1000 per. That's still only $50,000.
One less zero usually gets the accountants' attention on an expenditure like this.
But let's talk about administrative support. IMHO you're going to need 1:1 admin per NT server at that usage level, given that remote admin of NT is difficult, and 500 users per server is going to prompt more than the occasional pretty blue interface. (Nevermind the security team you're going to need for a major NT installation.) Say a cheap NT admin costs $50kUS including benefits & overhead. You're looking at an HR budget of $2,500,000us. On the other hand, say you splurge and spend $150kUS per *nix admin. If they couldn't handle 10 little boxen apiece, I'll eat the electrons this was posted with. That's an HR budget of $750,000us.
That's 1/10th the hardware expense and 1/3 the maintenance expense of using Exchange. And that's (a) making some wild assumptions that benefit the Exchange argument, and (b) assumes that you're running *nix on shit hardware. Spend 5 times as much on hardware for new, supported stuff (say $250,000us, which would buy you a couple of well-outfitted Sparc 4500s, or 10 really gorgeous systems from VA Research [varesearch.com]). Your downtime will become next to nothing, you'll still have spent only half of what you would have for NT and Exchange, and your ongoing yearly administrative cost will be 1/3 of the other option. The *nix administrative savings alone will pay for the *nix hardware in a few months.
Oh yeah. I forgot the expense of 50 copies of Windows NT, 50 copies of Exchange Server, and 25,000 client licenses... (*erk*!!)
Notes from Linux '99 (Score:3)
The open source solution was much more cost effective and has proved fairly stable.
Unfortunately the proceedings from the event are not yet online, however I'll try and forward you a copy (or post a link to this thread) as it may prove useful to you.
--
Re:No recommendation... (Score:5)
A sysadmin at, ahem, a "large jeans manufacturer" was put in charge of Exchange on hundreds of NT servers. He dutifully logged and reported dozens of bugs, system outages, etc., to MS support, as the thing crashed and burned like the Hindenburg II. After a few months of this, Microsoft decided to act on the problems. The solution was simple: they sent a letter to his boss saying he was a troublemaker.
Where are "Exchange horror stories" online? cost? (Score:3)
Knowing MS Exchange is a "Bad Thing", and I'd like to save the company money where possible, I decided to search the web for a collection of "horror stories and MS Exchange"... to my surprise I couldn't find ANYTHING!
Now I've seen articles here and there (InfoWorld, news.com etc.) about Exchange bugs, but I would have thought SOMEONE had collected URL's and posted them. Nothing. I'd have to do a lot of research to get this info, and given my workload it would be an unwise distraction.
The second thing I'd like to know, is how much does MS Exchange COST? I know the price varies, and larger companies get breaks if they "cozy" up to MS, but that doesn't help me much. Say a company has 50-150 employees... what does that translate into just for the software licensing?
Re:No recommendation... (Score:5)
Listen to this advice, it's obviously born on the hard back of experience, just as much as me reiterating this same line: do not use exchange.
For example:
This is only a start, but I'm sure other people have many of their own reasons as well...
I remember our migration of a mere 750 (users) with extreme horror. We had to manually create each user.
You can create mailboxes in exchange via a config file with the mailbox import tool, although I figured it out by looking at files it created and not via any documentation. With exchange 5.5 I'm pretty sure you can create mailboxes with ldap (although this is far from documented last I looked).
As to solutions, I haven't used any open source email solutions with more than ~5000 users, for which sendmail and the UW pop3d and imapd worked well for the users that I had (many were very light on email). I'd be really neat to integrate an MTA and an IMAP server with ldap to support IMAP referrals and smart mail redirection. I know some of this is done as sendmail has LDAP patches and example rules for this, but I'm not so sure about IMAP side.
Why Reinvent the Wheel? (Score:4)
Exchange => Pain (Score:3)
Check out Cyrus, from Carnegie Mellon, which is gratis (but not free).
Or maybe you'd like to spend some money. Then there are lots of companies, like Mirapoint [mirapoint.com], who I work for.
Large Scale free email (Score:5)
I know of three potential semi-free solutions.
Carnegie Mellon Cyrus (go to the FTP site and download the latest version. Don't rely on the way out of date web page to link to it.) IMAP server.
University of Washington's imapd. This seems to be under more active development, and supports a nice range of features, mailbox formats, and security mechanisms. However, it uses the passwd file (although you might be able to get around this using PAM) and it doesn't natively support quotas. (although you can do this at the OS level.
Darthmouth's Blitzmail Server: This has been ported to linux, and is *wonderfully* scalable across multiple machines. It inlcudes its own directory services too. The only problem is that it doesn't support Imap (although some work has started on that front), and the only database it supports as a backend is oracle. I would love it if someone hacked it to use mysql of postgresql with IMAP support, but that's a tall order. The client is also under-featured.
All of these have their drawbacks though. You might wish to go with a commercial IMAP/POP server on linux. There are a few good ones that exist. You definitely don't want to go with exchange. A lot of people go that route because they are forced to. My experience with exchange 5.5 was so bad that I would not recommend it to anyone.
-OT
Well, over here at Cisco... (Score:5)
No recommendation... (Score:3)
If you are going to setup 25,000 users, do not, repeat NOT, use Exchange. I remember our migration of a mere 750 with extreme horror. We had to manually create each user.
Of course I was simply a lowly programmer working under the direction of our totally incompetent network admin--maybe there was an easier way and she missed that topic in the training the week before.
What you really need is a requirements analysis. Exchange is a totally different thing than, say, Sendmail. Analyzing what you need will tell you which to go with. For instance, do you need public folders, scheduling, etc? If so, maybe use Exchange. Do you need configurability, speed and Internet email? Then you want not-Exchange.
---
Put Hemos through English 101!
Sendmail @ Netcom (Score:3)
We noted that directory lookups got worse in a distinct knee -- i.e., we had no problems for a long time and then we hit a magic number and things went all to hell. I do not know offhand how well linux or Solaris deals with directory lookups, but you could test easily enough.
The thing you didn't tell us was what the volume would be like; the number of users matters for the mail spool but the number of email messages matters for the CPU usage... I suspect that you won't need a very heavy box, though. Email is cheaper than you might think.
Oh. Run a DNS server on the mail hub, to avoid a lot of lengthy DNS queries on some other poor machine. Flush the cache daily.
That's me: here are the details (Score:4)
Re:No recommendation... (Score:3)
I use and maintain an Exchange server (well, three) and the main server consumes 10 gigs of a harddrive and all of a 333 MHz Pentium. This is for about 200 users and most are not that active.
Besides the hardware overhead there are other negatives to Exchange. Namely, it does not route internet traffic well, it has poor error reporting, and it "clusters" badly. I'll take each point one by one.
My company has affiliates in small offices around the world and they have neither the on-site resources or talent to maintain an e-mail server so these offices use our Exchange server as POP3 and SMTP. This creates an open relay and all attempts to close the relay have met with stiff opposition -- users complaining they now have to use a password, cannot remember what domain they are on, and general users resisting change. At the moment, Exchange has no true "Back Office" solution for this problem and I would have to personally configure all of our affiliate offices if I want to completely secure routing.
The error reporting come down to this -- either you log all of the messages passing through Exchange or none of them. I wanted to log the messages that caused errors for obvious reasons and after about 4 days noticed the drives filling up with archives all all the messages, not just those messages generating errors. Microsoft admits this is a problem but there is still no fix, at least not in SP2.
And finally, "clustering". I'm not talking about true clustering but instead about using multiple Exchange servers to distribute the load somewhat. We have two e-mail domains and wanted to start putting people on the second domain to balance the load. Each server runs fine on its own but for some reason they hate talking to each other. The replication services keep stopping (pausing, really) and site connector is more frustrating than helpful.
I have not had many problems with our Exchange server otherwise. It runs forever and reliably. It has the longest uptime of any of our NT machines, only needing a reboot every month or two. However, I'd think long and hard before accepting a job caring 25,000 user's e-mail if the server were NT. Anything over about 1000 users you should look elsewhere if you can.
Exchange on NT for 25000 people??? PLEASE, NO!! (Score:3)
First of all, suggesting to implement an NT solution for an organization of that size is already tempting your job security, but to actually do it?
Assuming standard users and needs for this system, I can only recommend using a Lotus Notes/Domino system. If you've got the cash, there is simply no better solution out there, or even close.
Run Domino (the server end of Notes) on several UNIX servers. Solaris (SPARC and x86), AIX, and HP/UX are all supported, with a Linux port (Caldera 2.3 (currently in beta) and Red Hat 6.0 will be supported, as well as SuSE 6.1 and Pacific HiTech) on it's way Q4 99 per DevCon.
Notes has got all you'll ever need, and R5 simply blows away anything M$ has out there. You've got to pay for seats with Notes, but to tell you the truth, Exchange is free, and you get less than you pay for.
Plus, your users can run the Notes client on any Win32 they think is prettiest (please tell me you'll use NT and not 9x on the client end).
Look at this [weightlessdog.com] for a guy in your situation who had to deal with Exchange.
Some other really good links are here [sandia.gov], here [computerworld.com], and here:
http://www.notes.net/50beta.nsf/7d6a87824e2f097
(problem with the last one, copy it and cut out the space that is stuck between the zero's, the href tag keeps putting it in! It is a great article though : )
(TIP: Show the guys with the money those links so they know why you should use a Domino/Notes solution.)
Outlook doesn't scale, look at other solutions (Score:4)
This is a large client trying to implement a server farm of 20+ NT machines, each server supporting 600-800 users, and combining the whole lot into a coherent whole. Fortunately I only have to fix their poor network designs. The team of administrators now numbers more than 50, most are MCSEs, none less than 5 years experience with Micro~1.oft products. They are tearing their hair out on a daily basis. Complaints number in the hundreds every day, and thats just the users who haven't given up completely.
My advice is to start looking at the larger commercial products, possibly Netscape's server. Get a reputable vendor to support it.
If you look at open source systems, start with OpenBSD and NetBSD.
Divide your system up between the MTA doing delivery/reception of the messages, and the MTA serving the users. Its ok if email to the outside world goes down for short periods of time, its almost expected. But if users cant get to their mailbox 100% of the time, you will look bad.
You also need to look at managing more than 32000 or 65000 users in the future, remember that various *nixes have either 15 or 16 bit UID fields. You should make sure user accounts/authentication/logins are separate from any UID system on any machine type. This means getting some kind of medium sized DB, and tying it into your auth and login schemes. Others have done it, its not that hard (look at AOL with 10million+ user accounts)
the AC