I'd go with a fanless 350MHz Pentium II, running Windows 98SE, with 256MB ECC RAM(some DOS apps flip out if you have that much extended memory, though), on an Intel BX-chipset motherboard(Asus P2B comes to mind). Go with a name-brand power supply like Antec, non-PFC(you do want to use it with a cheap UPS without worries over compatibility, right?), with a power rating of 500W or more(not that you need it, it will probably last longer if you don't push it as close to its limits). Install two large, quiet chassis fans from different manufacturers. I'd use a 5400 RPM laptop drive(will probably need a $5-$10 mount/connector adapter), or even 7200 RPM, 20-80GB or so, with a 5-year warranty. Too large and the BIOS will have problems, and you'd probably not use it anyway.
If you want something newer(and you may, as I'm not sure what the lifespan on older slot 1 motherboards are...), you may want to look into a fanless socket 370 VIA C3, running on an Intel BX2 motherboard(only because it supports ECC, you can see other chipsets that nominally support it here: http://www.anime.net/~goemon/linux-ecc/
Oh, and don't get a cheapo $10 case. They'll warp and bend, and could conceivably short out your motherboard. Not to mention slice open your supple flesh.