Saint Aardvark's Journal: Again with the mail server, why not?

So Monday afternoon the new mail server started to slow down. I'd been getting worried over the weekend about the fact that the MRTG graphs were showing big timeouts, and was thinking that if I had to do it over again I'd not use vinum so much: the disk access was just really slow. (True story: during the install I'd noticed this, and tried untarring a file (600kb or so). It took 17 seconds, nearly all disk access. Tried it on my box at home for comparison (450MHz Celeron, random cheap disk drives) and it took 6 seconds. Hrm...) On Monday it got to the point where it was refusing connections...arghh. So I swapped in the old co-front-end server (no filtering) that I'd taken off to let the new one handle the load, stopped Sendmail, tried not to panic and thought about what to do. Talked it over w/co-workers/people who know better, and came to the conclusion that the box really did need to be rebuilt.

(ObDisclaimer: I really, really wish I was certain I know what I'm talking about here.)

Vinum was set up like this: three separate partions on each of four drives made up three separate vinum devices (one for /tmp, one for /var, and one for /usr). I've been told since that you can split up one big vinum device into diff. partitions/slices, so that was a waste. As well, the four drives were all on two controllers (did I mention they're IDE?). Plus, in the vinum conf. file I just put everything in order: /dev/ad0s1, /dev/ad1s1, /dev/ad2s1, /dev/ad3s1. Again, I've been told since that that means vinum will end up writing to them in that order, which slows down writes even more.

Decided that we didn't really need to worry about keeping the spam we caught in the event of a disk failure, but the mail queue would be a nice thing to have (even though, at any given moment, it's 99% mutli-thousand returns to spammer mail servers that aren't accepting any mail...gotta do something about that). So I set up /usr, / and /var to live on the onboard Promise mirror array; 40GB total. One drive each went on each of the two vanilla IDE controllers left; one is for /var/procmail, and one for /var/procmail/spamassassin. It seems like such a waste to have 40GB for each of those directories (!), but I can't do much about that now.

Stayed late on Monday to set it up. Took about four hours, what with figuring out why I wasn't able to mount partitions via NFS (whoops-a-daisy). If I were to do it again, I could probably do it in two if everything went right. Put it back on the front end about quarter to ten, went home and slept, then took the other, older mail server off the front end the next day. Disk access is much better now, and hopefully it'll stay that way.

God it was fun, though. This is exactly the sort of thing I want to do, and while the mistakes are painful the learning experience is great. (Part of the reason I'm putting all this up here is so that it might be useful to someone else, a la the excellent FreeBSD Diary page. The other part, of course, is sheer ego. And for prospective employers to find...shudder. :-))

So to learn: more about vinum. More about Sendmail: got the O'Reilly book at work; made it through all the LHS/RHS chapters only to get to where it said "But of course no one uses that. Now we'll talk about .mc files." Arghh! Plus, started reading about m4 and nearly threw the book across the room in disgust. I'm sure there is a Very Good Reason for yet another scripting language, but I'll be damned if I can figure out what it might be at this point in my young life. (Given the age of Sendmail, I'm guessing it was easier to write one rather than wait for perl/python/Turing machines to be invented.) God, m4 is hairy. I know, I know, plug away at it, but still.

Back on topic: more about NFS. I figured out how to install a kernel from a read-only NFS-mounted partition, but I've yet to figure out how to install ports from a read-only NFS-mounted /usr/ports. (Li'l help?)

Other news: bought a 486 (SX!) laptop on eBay for $21 US. As it turns out, should be easy enough to install Linux or FreeBSD on here, which'll mean a cool li'l email/diagnostic toy^Hol (you know, for all those times I debugging raw Ethernet frames onsite). Should be in next week or so.